Implements https://github.com/godotengine/godot-proposals/issues/3371.
New `target` presets
====================
The `tools` option is removed and `target` changes to use three new presets,
which match the builds users are familiar with. These targets control the
default optimization level and enable editor-specific and debugging code:
- `editor`: Replaces `tools=yes target=release_debug`.
* Defines: `TOOLS_ENABLED`, `DEBUG_ENABLED`, `-O2`/`/O2`
- `template_debug`: Replaces `tools=no target=release_debug`.
* Defines: `DEBUG_ENABLED`, `-O2`/`/O2`
- `template_release`: Replaces `tools=no target=release`.
* Defines: `-O3`/`/O2`
New `dev_build` option
======================
The previous `target=debug` is now replaced by a separate `dev_build=yes`
option, which can be used in combination with either of the three targets,
and changes the following:
- `dev_build`: Defines `DEV_ENABLED`, disables optimization (`-O0`/`/0d`),
enables generating debug symbols, does not define `NDEBUG` so `assert()`
works in thirdparty libraries, adds a `.dev` suffix to the binary name.
Note: Unlike previously, `dev_build` defaults to off so that users who
compile Godot from source get an optimized and small build by default.
Engine contributors should now set `dev_build=yes` in their build scripts or
IDE configuration manually.
Changed binary names
====================
The name of generated binaries and object files are changed too, to follow
this format:
`godot.<platform>.<target>[.dev][.double].<arch>[.<extra_suffix>][.<ext>]`
For example:
- `godot.linuxbsd.editor.dev.arm64`
- `godot.windows.template_release.double.x86_64.mono.exe`
Be sure to update your links/scripts/IDE config accordingly.
More flexible `optimize` and `debug_symbols` options
====================================================
The optimization level and whether to generate debug symbols can be further
specified with the `optimize` and `debug_symbols` options. So the default
values listed above for the various `target` and `dev_build` combinations
are indicative and can be replaced when compiling, e.g.:
`scons p=linuxbsd target=template_debug dev_build=yes optimize=debug`
will make a "debug" export template with dev-only code enabled, `-Og`
optimization level for GCC/Clang, and debug symbols. Perfect for debugging
complex crashes at runtime in an exported project.
This makes it possible to retrieve all relevant versioning info used to
generate `core/version_generated.gen.h` in the buildsystem.
Notably it makes the custom logic parsing the `GODOT_VERSION_STATUS`
environment variable to override status easy to reuse.
We want to replace libnethost as it gives us issues with some compilers.
Our implementation tries to mimic libnethost's hostfxr_resolver search
logic. We try to use the same function names for easier comparing in
case we need to update this in the future.
We're targeting .NET 5 for now to make development easier while
.NET 6 is not yet released.
TEMPORARY REGRESSIONS
---------------------
Assembly unloading is not implemented yet. As such, many Godot
resources are leaked at exit. This will be re-implemented later
together with assembly hot-reloading.
Modules can now call:
env.module_add_dependencies(name: str, deps: list, optional: bool)
To add required or optional dependencies during the "can_build" step.
Required dependencies will be checked and the module will be not be
enabled when they are missing, printing a warning to notify the user.
* Changed to use the same stages as extensions.
* Makes the initialization more coherent, helping solve problems due to lack of stages.
* Makes it easier to port between module and extension.
* removed the DRIVER initialization level (no longer needed).
All compilation messages are now written in blue to ensure that
compiler errors and warnings stand out more. Messages were also
slightly shortened to make them easier to fit on a single line on
narrow terminals.
This makes it possible to change the branch of the documentation that
URLs are pointing to without having to modify all class reference
files.
In the XML class reference, the `$DOCS_URL` placeholder should be used,
and will be replaced automatically in the editor and when generating
the RST class reference.
The documentation branch string is set in `version.py`.
Co-authored-by: Hugo Locurcio <hugo.locurcio@hugo.pro>
Mapping and other "abstract base classes" were moved after python 3.3 from collections to collections.abc
Python 3.3 is long gone and a newer version of python won't support this code.
Whenever we change the name (or remove) generated cpp files with the `.gen.cpp`
extension, users run into build issues when switching between branches (i.e.
switching before and after the name change/removal). This is because we glob
`*.cpp` so if a now-obsolete file from a previous build is present, we'll
include it too, potentially leading to bugs or compilation failure (due to
missing headers or invalid code).
So globbing patterns in `add_source_files` will now skip files ending with
`.gen.cpp`, which should instead be passed explicitly where they're used.
Add support for specifying defines for each configuration
Add support for specifying extra cli args for each configuration
Add support for specifying extra includes for each configuration
This could cause spurious errors on CI when trying to prune the cache,
as for some reason it tries to remove files/paths which do not exist.
That points at a bug in the `cache_progress` logic but at least this
workaround should prevent CI failures.
Emscripten is LLVM-based so we want to follow the same logic. But we can't just
put it as a match in `methods.using_clang()` as that would mess with the
compiler version detection logic used to restrict old GCC and Clang releases.
`VERSION_STATUS` is part of what constitutes the reference version for a given
Godot build, and is part of the version check for compatible export templates.
For dev snapshots (alpha, beta, RCs), we usually set the `VERSION_STATUS` to
a specific build number (e.g. `beta2`), but this change doesn't end up
committed to the Git repository as we don't want to keep changing `version.py`
for testing builds.
So this new environment override will be what can be used in official builds
and by users making custom builds for specific snapshots.
* This PR adds the ability to disable classes when building.
* For now it's only possible to do this via command like:
`scons disable_classes=RayCast2D,Area3D`
* Eventually, a proper UI will be implemented to create a build config file to do this at large scale, as well as detect what is used in the project.
Test sources and build parameter were not supplied to the visual studio project. This resulted in a build that was not able to be test using the --test command. Adding build parameter ensures we can test, and supplying the sources ensures we have all files to write new tests and edit existing ones.
The `dev=yes` and `production=yes` options work as aliases to set a number of
options, while still aiming to allow overriding specific options if the user
wishes so. (E.g. `production=yes use_lto=no` should work to enable production
defaults *but* disable LTO.)
That wasn't working as `ARGUMENTS.get()` returns a string and not a boolean as
expected by `BoolVariable`, and this wasn't flagged as a bug... So added a
helper method using SCons' `BoolVariable._text2bool` to do the conversion
manually.
This adds `custom_modules_recursive` which allows to detect and collect
all nested C++ modules which may reside in any directory specified by
`custom_modules` option.
The detection logic is made to be more strict because `SCSub` may be
used for organizing hierarchical builds within a module itself, so the
existence of `register_types.h` and `config.py` is checked as well
(these are all required for a C++ module to be compiled by Godot).
For performance reasons, built-in modules are not checked recursively,
and there's no benefit of doing so in the first place.
It's now possible to specify a directory path pointing to a *single*
module, as it may contain nested modules which are detected recursively.
First, compile the engine normally with:
```
scons custom_modules="path/to/your/modules" vsproj=yes
```
Then run the Visual Studio project. You can now rebuild the engine if
you need to make changes to custom modules directly within IDE.
A new `env.Run` method is added which allows to control the verbosity
of builders output automatically depending on whether the "verbose"
option is set. It also allows to optionally run any SCons commands in a
subprocess using the existing `run_in_subprocess` method, unifying
the interface. `Action` objects wrap all builder functions to include a
short build message associated with any action.
Notably, this removes quite verbose output generated by `make_doc_header`
and `make_editor_icons_action` builders.
Until https://github.com/psf/black/pull/1328 makes it in a stable release,
we have to use the latest from Git.
Apply new style fixes done by latest black.
A new `methods.dump(env)` is added to dump the construction environment
used by SCons to build Godot to a `.scons_env.json`. The file can be used
for debugging purposes and any external tool.
This is still a bit hacky and eventually we should rework the way we handle
optional dependencies (especially with regard to builtin/system libs), but
it's a simple first step.
Fixes#39219.
The insertion order for dictionaries is only a language feature for
Python 3.6/3.7+ implementations, and not prior to that.
This ensures that the engine won't be rebuilt if the order of detected
modules changes in any way, as the `OrderedDict` should guarantee
inerstion order.
The existence of `SCsub` is checked instead. This file is required for
all modules, and prevents the build system to leave modules without
`config.py` undetected, leading to silently ignoring the module during
compilation.
This patch adds ability to include external, user-defined C++ modules
to be compiled as part of Godot via `custom_modules` build option
which can be passed to `scons`.
```
scons platform=x11 tools=yes custom_modules="../project/modules"
```
Features:
- detects all available modules under `custom_modules` directory the
same way as it does for built-in modules (not recursive);
- works with both relative and absolute paths on the filesystem;
- multiple search paths can be specified as a comma-separated list.
Module custom documentation and editor icons collection and generation
process is adapted to work with absolute paths needed by such modules.
Also fixed doctool bug mixing absolute and relative paths respectively.
Implementation details:
- `env.module_list` is a dictionary now, which holds both module name as
key and either a relative or absolute path to a module as a value.
- `methods.detect_modules` is run twice: once for built-in modules, and
second for external modules, all combined later.
- `methods.detect_modules` was not doing what it says on the tin. It is
split into `detect_modules` which collects a list of available modules
and `write_modules` which generates `register_types` sources for each.
- whether a module is built-in or external is distinguished by relative
or absolute paths respectively. `custom_modules` scons converter
ensures that the path is absolute even if relative path is supplied,
including expanding user paths and symbolic links.
- treats the parent directory as if it was Godot's base directory, so
that there's no need to change include paths in cases where custom
modules are included as dependencies in other modules.
Some required changes are made:
- locally imported SCons-specific packages within the method;
- `global` variables converted to `nonlocal` (used in nested functions).
Configured for a max line length of 120 characters.
psf/black is very opinionated and purposely doesn't leave much room for
configuration. The output is mostly OK so that should be fine for us,
but some things worth noting:
- Manually wrapped strings will be reflowed, so by using a line length
of 120 for the sake of preserving readability for our long command
calls, it also means that some manually wrapped strings are back on
the same line and should be manually merged again.
- Code generators using string concatenation extensively look awful,
since black puts each operand on a single line. We need to refactor
these generators to use more pythonic string formatting, for which
many options are available (`%`, `format` or f-strings).
- CI checks and a pre-commit hook will be added to ensure that future
buildsystem changes are well-formatted.
Scons' `Environment.subst()` does that, and was already used in the
other place where we query an env variable (`env["LINK"]` in x11 code).
Fixes `3.2` iOS build after cherry-pick of #36559 (previously it only
ran for GCC code, not iOS's Clang), and the same issue would likely
affect `master` if iOS builds were enabled right now.
We already had `MODULE_*_ENABLED` defines but only in the modules
environment, and a few custom `*_ENABLED` defines in the main env
when we needed the information in core.
Now this is defined in a single header which can be included in the
files that need this information.
Unify pack file version and magic to avoid hardcoded literals.
`version.py` now always includes `patch` even for the first release in
a new stable branch (e.g. 3.2). The public name stays without the patch
number, but `Engine.get_version_info()` already included `patch == 0`,
and we can remove some extra handling of undefined `VERSION_PATCH` this
way.
Co-authored-by: Rémi Verschelde <rverschelde@gmail.com>
The new 'split_libmodules=yes' option is useful to work around linker
command line size limitations when linking a huge number of objects.
We're currently over 64k chars when linking libmodules.a on Windows
with MinGW, which triggers issues as seen in #30892.
Even on Linux, we can also reach linker command line size limitations
by adding more custom modules.
We force this option to True for MinGW on Windows, which fixes#30892.
Additional changes to lib splitting:
- Fix linking of the split module libs with interdependent symbols,
hacking our way into LINKCOM and SHLINKCOM to set the `--start-group`
and `--end-group` flags.
- Fix Python 3 compatibility in `methods.split_lib()`.
- Drop seemingly obsolete condition for 'msys' on 'posix'.
- Drop the unnecessary 'split_drivers' as the drivers lib is no longer
too big since we moved all thirdparty builds to modules.
Co-authored-by: Hein-Pieter van Braam-Stewart <hp@tmm.cx>
When checking mingw-w64 version, at least on debian, the regex being used returned 86 because the name of the binary in debian starts with x86_64-w64 so we use the dumpversion option that gcc has. This fixes not compiling because gcc versions < 7 don't have some checks like shadow-local
Also added support for SCons project-absolute paths (starting with #) and
warning about duplicates in add_source_files(), and fixed
default_controller_mappings.gen.cpp being included twice after first build
due to *.cpp globbing.
Part of #30270.
It's the recommended way to set those, and is more portable
(automatically prepends -D for GCC/Clang and /D for MSVC).
We still use CPPFLAGS for some pre-processor flags which are not
defines.
Also include website URL and make it configurable via version.py
together with the rest of the engine branding.
Add mention to MIT license in --help output.
Many contributors (me included) did not fully understand what CCFLAGS,
CXXFLAGS and CPPFLAGS refer to exactly, and were thus not using them
in the way they are intended to be.
As per the SCons manual: https://www.scons.org/doc/HTML/scons-user/apa.html
- CCFLAGS: General options that are passed to the C and C++ compilers.
- CFLAGS: General options that are passed to the C compiler (C only;
not C++).
- CXXFLAGS: General options that are passed to the C++ compiler. By
default, this includes the value of $CCFLAGS, so that setting
$CCFLAGS affects both C and C++ compilation.
- CPPFLAGS: User-specified C preprocessor options. These will be
included in any command that uses the C preprocessor, including not
just compilation of C and C++ source files [...], but also [...]
Fortran [...] and [...] assembly language source file[s].
TL;DR: Compiler options go to CCFLAGS, unless they must be restricted
to either C (CFLAGS) or C++ (CXXFLAGS). Preprocessor defines go to
CPPFLAGS.
If it needs to be hardcoded (for the sake of reproducible builds),
it should be together with the other hardcoded version info.
And yeah, two months in, let's move to 2019.
so that godot package builds reproducibly
in spite of indeterministic filesystem readdir order
and http://bugs.python.org/issue30461
See https://reproducible-builds.org/ for why this is good.
Sort font input file list, so that builtin_fonts.gen.h
is created in a reproducible way
Sort list of platforms, so that editor/register_exporters.gen.cpp
is created in a reproducible way
Sort list of source files, so that .a files and resulting godot binaries
are created in a reproducible way
This value becomes part of get_version_info output,
but if it is changing every year without any other change,
it cannot be a useful indicator of anything.
Using a constant value, makes the package build reproducible.
See https://reproducible-builds.org/ for why this is good.
Previously the compiler would use system headers located at
/System/Library/Frameworks, which could result in compilation failures
due to the headers not always being up-to-date in regards to the
latest installed macOS SDK headers that come with Xcode.
Fix the issue by passing the SDK path via the -isysroot option to the
compiler and linker invocations.
If no custom SDK path is given, the build system queries the SDK path
via xcrun --show-sdk-path, which returns something similar to
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/
/Developer/SDKs/MacOSX.sdk/
Querying via xcrun is now also done for iphone (and simulator)
platforms as well.
Here is an example of a compilation failure message due to outdated
headers:
platform/osx/os_osx.mm:1421:41: error: use of undeclared identifier 'NSAppKitVersionNumber10_12'; did you mean 'NSAppKitVersionNumber'?
if (floor(NSAppKitVersionNumber) >= NSAppKitVersionNumber10_12) {
^~~~~~~~~~~~~~~~~~~~~~~~~~
NSAppKitVersionNumber
/System/Library/Frameworks/AppKit.framework/Headers/NSApplication.h:26:28: note: 'NSAppKitVersionNumber' declared here
- Refactored all builder (make_*) functions into separate Python modules along to the build tree
- Introduced utility function to wrap all invocations on Windows, but does not change it elsewhere
- Introduced stub to use the builders module as a stand alone script and invoke a selected function
There is a problem with file handles related to writing generated content (*.gen.h and *.gen.cpp)
on Windows, which randomly causes a SHARING VIOLATION error to the compiler resulting in flaky
builds. Running all such content generators in a new subprocess instead of directly inside the
build script works around the issue.
Yes, I tried the multiprocessing module. It did not work due to conflict with SCons on cPickle.
Suggested workaround did not fully work either.
Using the run_in_subprocess wrapper on osx and x11 platforms as well for consistency. In case of
running a cross-compilation on Windows they would still be used, but likely it will not happen
in practice. What counts is that the build itself is running on which platform, not the target
platform.
Some generated files are written directly in an SConstruct or SCsub file, before the parallel build starts. They don't need to be written in a subprocess, apparently, so I left them as is.
Rémi Verschelde
Björn Feber
nikitalita
Ignacio Roldán Etcheverry
Fredia Huya-Kouadio
SaracenOne
Aaron Franke
Pedro J. Estébanez
Fabio Alessandrelli
Nathan Franke
bruvzg
reduz
Rémi Verschelde
Mario Liebisch
Hugo Locurcio
Hristo Stamenov
Edward Auttonberry
jfons
MmAaXx500
Resul Çelik
Anshul7sp1
Marcel Admiraal
Andrii Doroshenko (Xrayez)
Bartłomiej T. Listwon
Tomasz Chabora
unknown
Nickolai Korshunov
Juan Linietsky
Joost Heitbrink
santouits
hbina085
Hendrikto
marxin
George Marques
Naoto Kondo
lupoDharkael
K. S. Ernest (iFire) Lee
Bernhard M. Wiedemann
Alexandru Croitor
dragmz
Viktor Ferenczi