godotengine
/
godot
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
godot/main/main_builders.py

139 lines
4.7 KiB
Python
Raw Normal View History

Running builder (content generator) functions in subprocesses on Windows - Refactored all builder (make_*) functions into separate Python modules along to the build tree - Introduced utility function to wrap all invocations on Windows, but does not change it elsewhere - Introduced stub to use the builders module as a stand alone script and invoke a selected function There is a problem with file handles related to writing generated content (*.gen.h and *.gen.cpp) on Windows, which randomly causes a SHARING VIOLATION error to the compiler resulting in flaky builds. Running all such content generators in a new subprocess instead of directly inside the build script works around the issue. Yes, I tried the multiprocessing module. It did not work due to conflict with SCons on cPickle. Suggested workaround did not fully work either. Using the run_in_subprocess wrapper on osx and x11 platforms as well for consistency. In case of running a cross-compilation on Windows they would still be used, but likely it will not happen in practice. What counts is that the build itself is running on which platform, not the target platform. Some generated files are written directly in an SConstruct or SCsub file, before the parallel build starts. They don't need to be written in a subprocess, apparently, so I left them as is.
2018-03-17 22:23:55 +00:00
"""Functions used to generate source files during build time
All such functions are invoked in a subprocess on Windows to prevent build flakiness.
"""
from platform_methods import subprocess_main
from compat import byte_to_str
from collections import OrderedDict
def make_splash(target, source, env):
src = source[0]
dst = target[0]
with open(src, "rb") as f:
buf = f.read()
with open(dst, "w") as g:
g.write("/* THIS FILE IS GENERATED DO NOT EDIT */\n")
g.write("#ifndef BOOT_SPLASH_H\n")
g.write("#define BOOT_SPLASH_H\n")
SCons: Format buildsystem files with psf/black Configured for a max line length of 120 characters. psf/black is very opinionated and purposely doesn't leave much room for configuration. The output is mostly OK so that should be fine for us, but some things worth noting: - Manually wrapped strings will be reflowed, so by using a line length of 120 for the sake of preserving readability for our long command calls, it also means that some manually wrapped strings are back on the same line and should be manually merged again. - Code generators using string concatenation extensively look awful, since black puts each operand on a single line. We need to refactor these generators to use more pythonic string formatting, for which many options are available (`%`, `format` or f-strings). - CI checks and a pre-commit hook will be added to ensure that future buildsystem changes are well-formatted. (cherry picked from commit cd4e46ee65dab6baa6a143bf3b3f64244be36712)
2020-03-30 06:28:32 +00:00
g.write("static const Color boot_splash_bg_color = Color(0.14, 0.14, 0.14);\n")
Running builder (content generator) functions in subprocesses on Windows - Refactored all builder (make_*) functions into separate Python modules along to the build tree - Introduced utility function to wrap all invocations on Windows, but does not change it elsewhere - Introduced stub to use the builders module as a stand alone script and invoke a selected function There is a problem with file handles related to writing generated content (*.gen.h and *.gen.cpp) on Windows, which randomly causes a SHARING VIOLATION error to the compiler resulting in flaky builds. Running all such content generators in a new subprocess instead of directly inside the build script works around the issue. Yes, I tried the multiprocessing module. It did not work due to conflict with SCons on cPickle. Suggested workaround did not fully work either. Using the run_in_subprocess wrapper on osx and x11 platforms as well for consistency. In case of running a cross-compilation on Windows they would still be used, but likely it will not happen in practice. What counts is that the build itself is running on which platform, not the target platform. Some generated files are written directly in an SConstruct or SCsub file, before the parallel build starts. They don't need to be written in a subprocess, apparently, so I left them as is.
2018-03-17 22:23:55 +00:00
g.write("static const unsigned char boot_splash_png[] = {\n")
for i in range(len(buf)):
g.write(byte_to_str(buf[i]) + ",\n")
g.write("};\n")
g.write("#endif")
def make_splash_editor(target, source, env):
src = source[0]
dst = target[0]
with open(src, "rb") as f:
buf = f.read()
with open(dst, "w") as g:
g.write("/* THIS FILE IS GENERATED DO NOT EDIT */\n")
g.write("#ifndef BOOT_SPLASH_EDITOR_H\n")
g.write("#define BOOT_SPLASH_EDITOR_H\n")
SCons: Format buildsystem files with psf/black Configured for a max line length of 120 characters. psf/black is very opinionated and purposely doesn't leave much room for configuration. The output is mostly OK so that should be fine for us, but some things worth noting: - Manually wrapped strings will be reflowed, so by using a line length of 120 for the sake of preserving readability for our long command calls, it also means that some manually wrapped strings are back on the same line and should be manually merged again. - Code generators using string concatenation extensively look awful, since black puts each operand on a single line. We need to refactor these generators to use more pythonic string formatting, for which many options are available (`%`, `format` or f-strings). - CI checks and a pre-commit hook will be added to ensure that future buildsystem changes are well-formatted. (cherry picked from commit cd4e46ee65dab6baa6a143bf3b3f64244be36712)
2020-03-30 06:28:32 +00:00
g.write("static const Color boot_splash_editor_bg_color = Color(0.14, 0.14, 0.14);\n")
Running builder (content generator) functions in subprocesses on Windows - Refactored all builder (make_*) functions into separate Python modules along to the build tree - Introduced utility function to wrap all invocations on Windows, but does not change it elsewhere - Introduced stub to use the builders module as a stand alone script and invoke a selected function There is a problem with file handles related to writing generated content (*.gen.h and *.gen.cpp) on Windows, which randomly causes a SHARING VIOLATION error to the compiler resulting in flaky builds. Running all such content generators in a new subprocess instead of directly inside the build script works around the issue. Yes, I tried the multiprocessing module. It did not work due to conflict with SCons on cPickle. Suggested workaround did not fully work either. Using the run_in_subprocess wrapper on osx and x11 platforms as well for consistency. In case of running a cross-compilation on Windows they would still be used, but likely it will not happen in practice. What counts is that the build itself is running on which platform, not the target platform. Some generated files are written directly in an SConstruct or SCsub file, before the parallel build starts. They don't need to be written in a subprocess, apparently, so I left them as is.
2018-03-17 22:23:55 +00:00
g.write("static const unsigned char boot_splash_editor_png[] = {\n")
for i in range(len(buf)):
g.write(byte_to_str(buf[i]) + ",\n")
g.write("};\n")
g.write("#endif")
def make_app_icon(target, source, env):
src = source[0]
dst = target[0]
with open(src, "rb") as f:
buf = f.read()
with open(dst, "w") as g:
g.write("/* THIS FILE IS GENERATED DO NOT EDIT */\n")
g.write("#ifndef APP_ICON_H\n")
g.write("#define APP_ICON_H\n")
g.write("static const unsigned char app_icon_png[] = {\n")
for i in range(len(buf)):
g.write(byte_to_str(buf[i]) + ",\n")
g.write("};\n")
g.write("#endif")
def make_default_controller_mappings(target, source, env):
dst = target[0]
g = open(dst, "w")
g.write("/* THIS FILE IS GENERATED DO NOT EDIT */\n")
SCons: Format buildsystem files with psf/black Configured for a max line length of 120 characters. psf/black is very opinionated and purposely doesn't leave much room for configuration. The output is mostly OK so that should be fine for us, but some things worth noting: - Manually wrapped strings will be reflowed, so by using a line length of 120 for the sake of preserving readability for our long command calls, it also means that some manually wrapped strings are back on the same line and should be manually merged again. - Code generators using string concatenation extensively look awful, since black puts each operand on a single line. We need to refactor these generators to use more pythonic string formatting, for which many options are available (`%`, `format` or f-strings). - CI checks and a pre-commit hook will be added to ensure that future buildsystem changes are well-formatted. (cherry picked from commit cd4e46ee65dab6baa6a143bf3b3f64244be36712)
2020-03-30 06:28:32 +00:00
g.write('#include "core/typedefs.h"\n')
g.write('#include "main/default_controller_mappings.h"\n')
Running builder (content generator) functions in subprocesses on Windows - Refactored all builder (make_*) functions into separate Python modules along to the build tree - Introduced utility function to wrap all invocations on Windows, but does not change it elsewhere - Introduced stub to use the builders module as a stand alone script and invoke a selected function There is a problem with file handles related to writing generated content (*.gen.h and *.gen.cpp) on Windows, which randomly causes a SHARING VIOLATION error to the compiler resulting in flaky builds. Running all such content generators in a new subprocess instead of directly inside the build script works around the issue. Yes, I tried the multiprocessing module. It did not work due to conflict with SCons on cPickle. Suggested workaround did not fully work either. Using the run_in_subprocess wrapper on osx and x11 platforms as well for consistency. In case of running a cross-compilation on Windows they would still be used, but likely it will not happen in practice. What counts is that the build itself is running on which platform, not the target platform. Some generated files are written directly in an SConstruct or SCsub file, before the parallel build starts. They don't need to be written in a subprocess, apparently, so I left them as is.
2018-03-17 22:23:55 +00:00
# ensure mappings have a consistent order
platform_mappings = OrderedDict()
for src_path in source:
with open(src_path, "r") as f:
# read mapping file and skip header
mapping_file_lines = f.readlines()[2:]
current_platform = None
for line in mapping_file_lines:
if not line:
continue
line = line.strip()
if len(line) == 0:
continue
if line[0] == "#":
current_platform = line[1:].strip()
if current_platform not in platform_mappings:
platform_mappings[current_platform] = {}
elif current_platform:
line_parts = line.split(",")
guid = line_parts[0]
if guid in platform_mappings[current_platform]:
SCons: Format buildsystem files with psf/black Configured for a max line length of 120 characters. psf/black is very opinionated and purposely doesn't leave much room for configuration. The output is mostly OK so that should be fine for us, but some things worth noting: - Manually wrapped strings will be reflowed, so by using a line length of 120 for the sake of preserving readability for our long command calls, it also means that some manually wrapped strings are back on the same line and should be manually merged again. - Code generators using string concatenation extensively look awful, since black puts each operand on a single line. We need to refactor these generators to use more pythonic string formatting, for which many options are available (`%`, `format` or f-strings). - CI checks and a pre-commit hook will be added to ensure that future buildsystem changes are well-formatted. (cherry picked from commit cd4e46ee65dab6baa6a143bf3b3f64244be36712)
2020-03-30 06:28:32 +00:00
g.write(
"// WARNING - DATABASE {} OVERWROTE PRIOR MAPPING: {} {}\n".format(
src_path, current_platform, platform_mappings[current_platform][guid]
)
)
Running builder (content generator) functions in subprocesses on Windows - Refactored all builder (make_*) functions into separate Python modules along to the build tree - Introduced utility function to wrap all invocations on Windows, but does not change it elsewhere - Introduced stub to use the builders module as a stand alone script and invoke a selected function There is a problem with file handles related to writing generated content (*.gen.h and *.gen.cpp) on Windows, which randomly causes a SHARING VIOLATION error to the compiler resulting in flaky builds. Running all such content generators in a new subprocess instead of directly inside the build script works around the issue. Yes, I tried the multiprocessing module. It did not work due to conflict with SCons on cPickle. Suggested workaround did not fully work either. Using the run_in_subprocess wrapper on osx and x11 platforms as well for consistency. In case of running a cross-compilation on Windows they would still be used, but likely it will not happen in practice. What counts is that the build itself is running on which platform, not the target platform. Some generated files are written directly in an SConstruct or SCsub file, before the parallel build starts. They don't need to be written in a subprocess, apparently, so I left them as is.
2018-03-17 22:23:55 +00:00
valid_mapping = True
for input_map in line_parts[2:]:
if "+" in input_map or "-" in input_map or "~" in input_map:
SCons: Format buildsystem files with psf/black Configured for a max line length of 120 characters. psf/black is very opinionated and purposely doesn't leave much room for configuration. The output is mostly OK so that should be fine for us, but some things worth noting: - Manually wrapped strings will be reflowed, so by using a line length of 120 for the sake of preserving readability for our long command calls, it also means that some manually wrapped strings are back on the same line and should be manually merged again. - Code generators using string concatenation extensively look awful, since black puts each operand on a single line. We need to refactor these generators to use more pythonic string formatting, for which many options are available (`%`, `format` or f-strings). - CI checks and a pre-commit hook will be added to ensure that future buildsystem changes are well-formatted. (cherry picked from commit cd4e46ee65dab6baa6a143bf3b3f64244be36712)
2020-03-30 06:28:32 +00:00
g.write(
"// WARNING - DISCARDED UNSUPPORTED MAPPING TYPE FROM DATABASE {}: {} {}\n".format(
src_path, current_platform, line
)
)
Running builder (content generator) functions in subprocesses on Windows - Refactored all builder (make_*) functions into separate Python modules along to the build tree - Introduced utility function to wrap all invocations on Windows, but does not change it elsewhere - Introduced stub to use the builders module as a stand alone script and invoke a selected function There is a problem with file handles related to writing generated content (*.gen.h and *.gen.cpp) on Windows, which randomly causes a SHARING VIOLATION error to the compiler resulting in flaky builds. Running all such content generators in a new subprocess instead of directly inside the build script works around the issue. Yes, I tried the multiprocessing module. It did not work due to conflict with SCons on cPickle. Suggested workaround did not fully work either. Using the run_in_subprocess wrapper on osx and x11 platforms as well for consistency. In case of running a cross-compilation on Windows they would still be used, but likely it will not happen in practice. What counts is that the build itself is running on which platform, not the target platform. Some generated files are written directly in an SConstruct or SCsub file, before the parallel build starts. They don't need to be written in a subprocess, apparently, so I left them as is.
2018-03-17 22:23:55 +00:00
valid_mapping = False
break
if valid_mapping:
platform_mappings[current_platform][guid] = line
platform_variables = {
"Linux": "#if X11_ENABLED",
"Windows": "#ifdef WINDOWS_ENABLED",
"Mac OS X": "#ifdef OSX_ENABLED",
"Android": "#if defined(__ANDROID__)",
"iOS": "#ifdef IPHONE_ENABLED",
"Javascript": "#ifdef JAVASCRIPT_ENABLED",
"UWP": "#ifdef UWP_ENABLED",
}
g.write("const char* DefaultControllerMappings::mappings[] = {\n")
for platform, mappings in platform_mappings.items():
variable = platform_variables[platform]
g.write("{}\n".format(variable))
for mapping in mappings.values():
SCons: Format buildsystem files with psf/black Configured for a max line length of 120 characters. psf/black is very opinionated and purposely doesn't leave much room for configuration. The output is mostly OK so that should be fine for us, but some things worth noting: - Manually wrapped strings will be reflowed, so by using a line length of 120 for the sake of preserving readability for our long command calls, it also means that some manually wrapped strings are back on the same line and should be manually merged again. - Code generators using string concatenation extensively look awful, since black puts each operand on a single line. We need to refactor these generators to use more pythonic string formatting, for which many options are available (`%`, `format` or f-strings). - CI checks and a pre-commit hook will be added to ensure that future buildsystem changes are well-formatted. (cherry picked from commit cd4e46ee65dab6baa6a143bf3b3f64244be36712)
2020-03-30 06:28:32 +00:00
g.write('\t"{}",\n'.format(mapping))
Running builder (content generator) functions in subprocesses on Windows - Refactored all builder (make_*) functions into separate Python modules along to the build tree - Introduced utility function to wrap all invocations on Windows, but does not change it elsewhere - Introduced stub to use the builders module as a stand alone script and invoke a selected function There is a problem with file handles related to writing generated content (*.gen.h and *.gen.cpp) on Windows, which randomly causes a SHARING VIOLATION error to the compiler resulting in flaky builds. Running all such content generators in a new subprocess instead of directly inside the build script works around the issue. Yes, I tried the multiprocessing module. It did not work due to conflict with SCons on cPickle. Suggested workaround did not fully work either. Using the run_in_subprocess wrapper on osx and x11 platforms as well for consistency. In case of running a cross-compilation on Windows they would still be used, but likely it will not happen in practice. What counts is that the build itself is running on which platform, not the target platform. Some generated files are written directly in an SConstruct or SCsub file, before the parallel build starts. They don't need to be written in a subprocess, apparently, so I left them as is.
2018-03-17 22:23:55 +00:00
g.write("#endif\n")
g.write("\tNULL\n};\n")
g.close()
SCons: Format buildsystem files with psf/black Configured for a max line length of 120 characters. psf/black is very opinionated and purposely doesn't leave much room for configuration. The output is mostly OK so that should be fine for us, but some things worth noting: - Manually wrapped strings will be reflowed, so by using a line length of 120 for the sake of preserving readability for our long command calls, it also means that some manually wrapped strings are back on the same line and should be manually merged again. - Code generators using string concatenation extensively look awful, since black puts each operand on a single line. We need to refactor these generators to use more pythonic string formatting, for which many options are available (`%`, `format` or f-strings). - CI checks and a pre-commit hook will be added to ensure that future buildsystem changes are well-formatted. (cherry picked from commit cd4e46ee65dab6baa6a143bf3b3f64244be36712)
2020-03-30 06:28:32 +00:00
if __name__ == "__main__":
Running builder (content generator) functions in subprocesses on Windows - Refactored all builder (make_*) functions into separate Python modules along to the build tree - Introduced utility function to wrap all invocations on Windows, but does not change it elsewhere - Introduced stub to use the builders module as a stand alone script and invoke a selected function There is a problem with file handles related to writing generated content (*.gen.h and *.gen.cpp) on Windows, which randomly causes a SHARING VIOLATION error to the compiler resulting in flaky builds. Running all such content generators in a new subprocess instead of directly inside the build script works around the issue. Yes, I tried the multiprocessing module. It did not work due to conflict with SCons on cPickle. Suggested workaround did not fully work either. Using the run_in_subprocess wrapper on osx and x11 platforms as well for consistency. In case of running a cross-compilation on Windows they would still be used, but likely it will not happen in practice. What counts is that the build itself is running on which platform, not the target platform. Some generated files are written directly in an SConstruct or SCsub file, before the parallel build starts. They don't need to be written in a subprocess, apparently, so I left them as is.
2018-03-17 22:23:55 +00:00
subprocess_main(globals())