Commit Graph

95 Commits

Author SHA1 Message Date
Pedro J. Estébanez 74b9c38d58 ResourceLoader: Add thread-aware resource changed mechanism 2024-09-06 08:57:09 +02:00
Pedro J. Estébanez c450f4d667 ResourceLoader: Simplify handling of unregistered tasks 2024-09-05 09:54:03 +02:00
Pedro J. Estébanez f4d76853b9 WorkerThreadPool (plus friends): Overhaul unlock allowance zones
This fixes a rare but possible deadlock, maybe due to undefined behavior. The new implementation is safer, at the cost of some added boilerplate.
2024-08-21 12:22:52 +02:00
Pedro J. Estébanez df23858488 ResourceLoader: Fix edge cases in the management of user tokens
1. Make handling of user tokens atomic:
   Loads started with the external-facing API used to perform a two-step setup of the user token. Between both, the mutex was unlocked without its reference count having been increased. A non-user-initiated load could therefore destroy the load task when it unreferenced the token.
   Those stages now happen atomically so in the one hand, the described race condition can't happen so the load task life insurance doesn't have a gap anymore and, on the other hand, the ugliness that the call to load could return `ERR_BUSY` if happening while other thread was between both steps is gone.
   The code has been refactored so the user token concerns are still outside the inner load start function, which is agnostic to that for a cleaner implementation.
2. Clear ambiguity between load operations running on `WorkerThreadPool`:
   The two cases are: single-loaded thread directly started by a user pool task and a load started by the system as part of a multi-threaded load.
   Since ensuring all the code dealing with this distinction would make it very complex, and error-prone, a different measure is applied instead: just take one of the cases out of the dicotomy. We now ensure every load happening on a pool thread has been initiated by the system.
   The way of achieving that is that a single-threaded user-started load initiated from a pool thread, is run as another task.
2024-08-21 12:22:52 +02:00
Pedro J. Estébanez 5c970db2e4 ResourceLoader: Optimize remap check by deferring until a non-mutex zone 2024-08-21 12:19:43 +02:00
Pedro J. Estébanez 28619e26cf ResourceLoader: Enhance deadlock prevention
Benefits:
- Simpler code. The main load function is renamed so it's apparent that it's not just a thread entry point anymore.
- Cache and thread modes of the original task are honored. A beautiful consequence of this is that, unlike formerly, re-issued loads can use the resource cache, which makes this mechanism much more performant.
- The newly added getter for caller task id in WorkerThreadPool allows to remove the custom tracking of that in ResourceLoader.
- The check to replace a cached resource and the replacement itself happen atomically. That fixes deadlock prevention leading to multiple resource instances of the same one on disk. As a side effect, it also makes the regular check for replace load mode more robust.
2024-08-21 12:19:43 +02:00
Pedro J. Estébanez bd0959ebdd ResourceLoader: Properly push & pop TLS state on recursive load tasks 2024-08-21 12:19:43 +02:00
Rémi Verschelde 5ca419e32c
Merge pull request from gamelessone/fix-cond-var
Fix use-after-free of `ConditionVariable` in `ResourceLoader`
2024-08-21 11:14:22 +02:00
Pedro J. Estébanez e9407d4877 ResourceLoader: Revert workaround resource loading crashes due to buggy TLS
This reverts commit 41c0785636.
2024-08-19 12:22:31 +02:00
Aleksey Vasenev 2ff6594928 Fix use condition_variable after free 2024-07-30 13:59:20 +03:00
Pedro J. Estébanez 5b5cdf2414 Fixup recent changes to threading concerns
ResourceLoader:
- Fix invalid tokens being returned.
- Remove no longer written `ThreadLoadTask::dependent_path` and the code reading from it.
- Clear deadlock hazard by keeping the mutex unlocked during userland polling.

WorkerThreadPool:
- Include thread call queue override in the thread state reset set, which allows to simplify the code that handled that (imperfectly) in the ResourceLoader.
- Handle the mutex type correctly on entering an allowance zone.

CommandQueueMT:
- Handle the additional possibility of command buffer reallocation that mutex unlock allowance introduces.
2024-07-16 11:03:02 +02:00
Pedro J. Estébanez ec61c5064c ResourceLoader: Support polling and get-before-complete on the main thread 2024-06-28 11:25:10 +02:00
Pedro J. Estébanez c1391489e3 GDScript: Enhance handling of cyclic dependencies 2024-06-26 17:44:32 +02:00
Pedro J. Estébanez f61c63e3a1 Avoid editor error reporting using resource loader thread's call queues 2024-05-31 09:42:37 +02:00
A Thousand Ships 4ed62665a2
Replace `.bind(...).call_deferred()` with `.call_deferred(...)` 2024-05-27 13:29:57 +02:00
Pedro J. Estébanez 5e144022e7 Enhance cache modes in resource loading
- Unify documentation, hoping to clear misconcepctions about about propagation of the cache mode across dependant loads.
- Clarify in docs that `CACHE_MODE_REPLACE` now also works on the main resource (from ).
- Add two recursive modes, counterparts of `CACHE_MODE_REPLACE` and `CACHE_MODE_IGNORE`, since it seems some need them (see , ).
- Let resources, even loaded with one of the ignore-cache modes, get a path, which is useful for tools.
2024-02-26 14:59:04 +01:00
Muller-Castro a8bc9f3e78 Add const lvalue ref to core/* container parameters 2024-02-14 11:20:36 -03:00
vittorioromeo 71e0082ab0 Extract 'SafeBinaryMutex' to separate header
This change simply extracts 'SafeBinaryMutex' from 'mutex.h' to
    'safe_binary_mutex.h', in an effort to reduce the compilation
    speed impact of including `mutex.h`.
2024-02-04 16:59:54 +01:00
Pedro J. Estébanez f5ca58d32f Avoid regressing in progress reporting in resource load 2024-01-05 19:33:58 +01:00
Saracen 0b0a6109b6 Make notify_dependency_error only defer calls if called from secondary threads. 2023-09-26 03:36:11 +01:00
Yuri Sizov d8ff69d53c Extract ScriptInstance to simplify includes
This allows to include script_instance.h directly in the
generated gdvirtual.gen.inc, and remove excessive includes
from the codebase.

This should also allow Resource to use GDVIRTUAL macros,
which wasn't possible previously due to a circular dependency.
2023-09-06 22:54:38 +02:00
Pedro J. Estébanez 41c0785636 Workaround resource loading crashes due to buggy TLS 2023-07-03 13:54:55 +02:00
Pedro J. Estébanez 45d0b38076 Enhance thread safety of loaders and importers 2023-05-12 17:44:48 +02:00
Pedro J. Estébanez 7537a0521f Simplify ResourceLoader error callbacks 2023-05-12 17:44:48 +02:00
Pedro J. Estébanez 045401c64e Use WorkerThreadPool for threaded resource loading 2023-05-10 18:53:41 +02:00
Pedro J. Estébanez b6647a5808 Avoid sync issues in materials with scheduled shader updates 2023-05-10 18:53:41 +02:00
Pedro J. Estébanez 8983b20ccd Avoid interaction issues between resource loading threads 2023-05-10 18:53:41 +02:00
Pedro J. Estébanez 618bb173ba Fix race condition in resource loader when a load task is reused 2023-02-20 21:20:05 +01:00
Raul Santos 9e9eac4676
Use enum instead of int in virtual methods return type 2023-01-31 19:06:49 +01:00
Rémi Verschelde ebd0b40f6e
Merge pull request from reduz/support-script-class-name-in-efs
Support script global resource name in EditorFileSystem
2023-01-21 16:54:23 +01:00
Juan Linietsky dddd8d43f6 Support script global resource name in EditorFileSystem
* Works for binary and text files.
* Makes EditorQuickOpen work with custom resources again.
* Information is cached and easily accessible.

Properly fixes . Supersedes  and supersedes 

**WARNING**: This required breaking backwards binary compatibility (.res and .scn files). Files saved after this PR is merged will no longer open in any earlier versions of Godot.
2023-01-21 14:19:27 +01:00
kobewi c0083e431b Cleanup unused engine code v2 2023-01-19 13:02:18 +01:00
Rémi Verschelde d95794ec8a
One Copyright Update to rule them all
As many open source projects have started doing it, we're removing the
current year from the copyright notice, so that we don't need to bump
it every year.

It seems like only the first year of publication is technically
relevant for copyright notices, and even that seems to be something
that many companies stopped listing altogether (in a version controlled
codebase, the commits are a much better source of date of publication
than a hardcoded copyright statement).

We also now list Godot Engine contributors first as we're collectively
the current maintainers of the project, and we clarify that the
"exclusive" copyright of the co-founders covers the timespan before
opensourcing (their further contributions are included as part of Godot
Engine contributors).

Also fixed "cf." Frenchism - it's meant as "refer to / see".
2023-01-05 13:25:55 +01:00
Adam Scott 4a16b8630e Fix `ResourceLoader::thread_load_tasks` crash 2022-12-07 16:59:10 -05:00
Gilles Roudière 8aa904f56c Exposes ResourceFormatLoader.recognize_path to scripting 2022-10-11 12:35:27 +02:00
reduz 6236a688b7 Implement Feature Build Profiles
This PR is a continuation of  (which was implemented exactly a year ago!)

* Add a visual interface to select which classes should not be built into Godot (well, they are built if something else uses them, but if not used the optimizer will remove them out).
* Add a detection system to scan the project and figure out the actual classes used.
* Added the ability for SCons to load build profiles.

Obligatory Screen:

A simple test with a couple of nodes in the scene resulted in a 25% reduction for the final binary size

TODO:

* Script languages need to implement used class detection (left for another PR).
* Options to disable servers or server functionalities (like 2D or 3D physics, navigation, etc). Are missing, that should also greatly aid in reducing binary size.
* Options to disable some modules would be desired.
* More options to disable drivers (OpenGL, Vulkan, etc) would be desired.

In general this PR is a starting point for more contributors to improve and enhance this functionality.
2022-07-22 10:53:23 +02:00
reduz 45af29da80 Add a new HashSet template
* Intended to replace RBSet in most cases.
* Optimized for iteration speed
2022-05-20 22:40:38 +02:00
reduz 746dddc067 Replace most uses of Map by HashMap
* Map is unnecessary and inefficient in almost every case.
* Replaced by the new HashMap.
* Renamed Map to RBMap and Set to RBSet for cases that still make sense
  (order matters) but use is discouraged.

There were very few cases where replacing by HashMap was undesired because
keeping the key order was intended.
I tried to keep those (as RBMap) as much as possible, but might have missed
some. Review appreciated!
2022-05-16 10:37:48 +02:00
reduz 0a57f964a3 Implement missing Node & Resource placeholders
Implemented by request of @neikeq to advance in the GDExtension version of Mono.

* If a Resource type is missing upon load, it will be remembered together with its data (Unless manually overriden).
* If a Node type is missing upon load, it will be also be remembered together with its data (unless deleted).

This feature makes working with GDExtension much easier, as it ensures that missing types no longer cause data loss.
2022-05-03 17:08:09 +02:00
Hugo Locurcio 180e5d3028
Remove `RES` and `REF` typedefs in favor of spelled out `Ref<>`
These typedefs don't save much typing compared to the full `Ref<Resource>`
and `Ref<RefCounted>`, yet they sometimes introduce confusion among
new contributors.
2022-05-03 01:43:50 +02:00
Rémi Verschelde fe52458154
Update copyright statements to 2022
Happy new year to the wonderful Godot community!
2022-01-03 21:27:34 +01:00
reduz 3682978aee Replace BIND_VMETHOD by new GDVIRTUAL syntax
* New syntax is type safe.
* New syntax allows for type safe virtuals in native extensions.
* New syntax permits extremely fast calling.

Note: Everything was replaced where possible except for `_gui_input` `_input` and `_unhandled_input`.
These will require API rework on a separate PR as they work different than the rest of the functions.

Added a new method flag METHOD_FLAG_OBJECT_CORE, used internally. Allows to not dump the core virtuals like `_notification` to the json API, since each language will implement those as it is best fits.
2021-08-22 08:23:58 -03:00
luz paz 3564c16cb8
Fix various typos with codespell
Found via `codespell -q 3 -S ./thirdparty,*.po,./DONORS.md -L ackward,ang,ans,ba,beng,cas,childs,childrens,dof,doubleclick,fave,findn,hist,inout,leapyear,lod,nd,numer,ois,ony,paket,seeked,sinc,switchs,te,uint`
2021-07-25 11:21:51 +02:00
reduz 32b43cfeb3 Implement Resource UIDs
* Most resource types now have unique identifiers.
* Applies to text, binary and imported resources.
* File formats reference both by text and UID (when available). UID always has priority.
* Resource UIDs are 64 bits for better compatibility with the engine.
* Can be represented and used textually, example `uuid://dapwmgsmnl28u`.
* A special binary cache file is used and exported, containing the mappings.

Example of how it looks:

```GDScript
[gd_scene load_steps=2 format=3 uid="uid://dw86wq31afig2"]

[ext_resource type="PackedScene" uid="uid://bt36ojelx8q6c" path="res://subscene.scn" id="1_t56hs"]
```

GDScript, shaders and other special resource files can't currently provide UIDs, but this should be doable with special keywords on the files.
This will be reserved for future PRs.
2021-07-24 09:16:52 -03:00
Pedro J. Estébanez 04688b92ff Rename Reference to RefCounted 2021-06-11 18:48:42 +02:00
Rémi Verschelde 0d1d719178 doc: Sync classref with current source
And fix various bogus bindings following previous PRs.
2021-02-19 14:39:14 +01:00
reduz f8d03b98e7 Improve resource load cache
-Added a new method in Resource: reset_state , used for reloading the same resource from disk
-Added a new cache mode "replace" in ResourceLoader, which reuses existing loaded sub-resources but resets their data from disk (or replaces them if they chaged type)
-Because the correct sub-resource paths are always loaded now, this fixes bugs with subresource folding or subresource ordering when saving.
2021-02-11 15:44:28 -03:00
Rémi Verschelde b5334d14f7
Update copyright statements to 2021
Happy new year to the wonderful Godot community!

2020 has been a tough year for most of us personally, but a good year for
Godot development nonetheless with a huge amount of work done towards Godot
4.0 and great improvements backported to the long-lived 3.2 branch.

We've had close to 400 contributors to engine code this year, authoring near
7,000 commit! (And that's only for the `master` branch and for the engine code,
there's a lot more when counting docs, demos and other first-party repos.)

Here's to a great year 2021 for all Godot users 🎆
2021-01-01 20:19:21 +01:00
reduz 127458ed17 Reorganized core/ directory, it was too fatty already
-Removed FuncRef, since Callable makes it obsolete
-Removed int_types.h as its obsolete in c++11+
-Changed color names code
2020-11-07 20:17:12 -03:00
Rémi Verschelde 0ee0fa42e6 Style: Enforce braces around if blocks and loops
Using clang-tidy's `readability-braces-around-statements`.
https://clang.llvm.org/extra/clang-tidy/checks/readability-braces-around-statements.html
2020-05-14 21:57:34 +02:00