* Remove bke, ed and wm prefixes
* Add prefixes like: geom, object, blend, lib.
* Shorten some category names
* A few log level changes to improve --log-level info output
Pull Request: https://projects.blender.org/blender/blender/pulls/140244
The root cause of the crash was that currently the depsgraph does not support
being rebuilt twice without being evaluated in the meantime. While not ideal to
rebuild the depsgraph twice, it's really something I'd expect to work without
crashes/leaks.
The double-rebuild when switching the scene was introduced by b6e1afb6e1
which tagged the depsgraph relations indirectly. Tagging relations at that place
should be valid though. The same bug can easily be reproduced by explicitly
writing code that rebuilds the depsgraph twice as shown in
https://projects.blender.org/blender/blender/issues/139079#issuecomment-1615029.
So far, we've found two places that need to be fixed to properly support
rebuilding the depsgraph before it has been evaluated:
* `update_invalid_cow_pointers`: `previously_visible_components_mask` was used
to check if the id is new and therefore not expanded yet. However, this check
is wrong in the case when the depsgraph was not evaluated yet. Instead, check
whether the ID is expanded directly. IDs which don't use copy-on-eval are
still handled properly due to another existing check.
* `DepsgraphNodeBuilder::begin_build`: This just discarded
allocated-but-not-expanded IDs without freeing them. Now they are freed when
their ownership is not transferred to `IDInfo`.
See
https://projects.blender.org/blender/blender/issues/139079#issuecomment-1615029
for more details.
Pull Request: https://projects.blender.org/blender/blender/pulls/141199
This commit cleanly splits IDProperties storage for its two very different
usages:
* "User-defined" data, also known as "custom properties". Mostly exposed
in UI and as 'dictionary' in Python scripting, these are unique to each data
(each object can have its own set of custom properties).
* "System-defined" data, mainly as backend-storage for runtime RNA
structures and properties. While these are not necessarily present in the
storage, they are registered for a data type, and therefore always available
to all data of that type through RNA access.
See #123232 for rationales, designs and alternative investigated solutions.
## User-facing Changes
When using Blender, the only noticeable change is that python-defined RNA
data are not listed anymore in the Custom Properties panels (e.g. Cycles
data).
From a Python API perspective, the main changes are:
* Runtime RNA structs defined by sub-classing `PropertyGroup` and
registering them are no more accessible through the 'dict' syntax.
* They remain accessible through a dedicated 'bl_system_properties_get()`
callback, but its usages are only expected to be for testing and
debugging.
* The result of this call will be `None` by default when there has been
nothing written into it yet, unless its optional `do_create` parameter
is set to `True`.
* Some types (like `Node`, `UIList`, etc.) cannot store raw IDProperties
anymore (using the 'dict' syntax).
## Technical Details
* Adds System idprops to some data types (IDs, ViewLayer...).
* Moves some other containers (e.g operator properties, or some UI types like
UILists) to only have system-defined properties.
* For a few specific types (like `PropertyGroup`), the same storage is used,
but exposed in RNA as both user and system properties.
* Adds RNA API accessor callback to system idprops.
* Adds a function `bl_system_properties_get()`, which wraps system-defined
idprops and gives 'dict-like' access to them. Currently mainly used by some
unittests.
* System IDProps are always ignored by RNA diffing code (and therefore
liboverride processing), as their value is already exposed through RNA
properties, and should always be processed through these RNA properties.
* Modifies bpy rna binding to use these new system idprops for property
accesses, and keeps using user-defined idprops for 'dict-type' accesses.
* Handles versioning by copying 'user idprops' (existing ones) into new
'system idprops'.
### IDProperties Split
These types keep their historic IDProperty storage for user properties,
and get a new `system_id_properties` storage for system properties:
`ID`, `ViewLayers`, `Bone`, `EditBone`, `BoneCollection`, `PoseBone`, `Strip`
These types can both be extended with registrable RNA properties, and
expose Custom Properties in the UI.
### IDProperties become System Ones
These types keep a single IDProperties storage (their DNA struct does not
change), but it is now exclusively used for system-defined properties.
`OperatorProperty`, `View3DShading`, `UIList`, `AddonPreferences`,
`KeyConfigPreferences`, `GizmoProperties`, `GizmoGroupProperties`,
`Node`, `NodeSocket`, `NodeTreeInterfaceSocket`, `TimelineMarker`,
`AssetMetaData``
Since user properties were never available in the UI for them, they lose
their 'dict-like' IDProperties access in the Python API.
### Single Storage, Exposed as Both in API
These types still use a single IDProperty storage, but expose it both as
user properties and as system ones through RNA API.
* `PropertyGroup`: They need both access APIs since they are both
used for raw IDProperty groups (the 'dict-like' access), and
internally to access data of runtime-defined RNA structs.
* `IDPropertyWrapPtr`: Done mainly to follow `PropertyGroup`.
* `NodesModifier`: cannot become purely system idprops currently, as
there is no other access than the 'raw ID properties' paths to their
values. This can be changed once #132129 is finally implemented.
Pull Request: https://projects.blender.org/blender/blender/pulls/135807
When creating depsgraph relationships for drivers, avoid creating the
'unshare' depsgraph node for drivers on custom properties and on
shapekey `value` properties.
This should fix a significant part of the performance regression
mentioned in #140706.
Pull Request: https://projects.blender.org/blender/blender/pulls/140724
Replace Listbase with a VectorList for duplist generation.
Performing one heap allocation for each dupli becomes very costly on
scenes with many instances.
Using a VectorList can make playback up to 15% faster.
Most scenes will have lower performance improvements,
but none of the ones I tested were slower.
VectorLists also have the advantage of being able to query the size of
the duplilist.
Pull Request: https://projects.blender.org/blender/blender/pulls/138947
This is part of the short term roadmap goal of simplifying the
compositor workflow
(see https://projects.blender.org/blender/blender/issues/134214).
The problem is that many users don't know how to get started with
compositing in Blender, even when they have used Blender for other
areas, e.g. modeling.
Note: although the solution makes compositor node trees reusable
accross blend files, this is a nice side effect and not the main goal
of the PR.
This PR implements a "New" button that creates a new compositing node
tree, and manages trees as IDs. This has following advantages:
- Consistent with other node editors and other parts of Blender,
therefore making it easier to getting started with compositing if users
are familiar with shading or geometry nodes
- Give users the ability to reuse the compositing node tree by linking
or appending it.
Note: The parameter "Use Nodes" is still present in this PR, but will
be removed (in a backward compatible way) in a follow up PR.
Pull Request: https://projects.blender.org/blender/blender/pulls/135223
Not only depend on the camera position but also on the other camera parameters.
This is important because otherwise Geometry Nodes won't be updated when e.g.
the focal length changes which is important when implementing e.g. camera
culling.
This already works when depending on a specific camera directly, but not when
depending on the active camera.
Pull Request: https://projects.blender.org/blender/blender/pulls/140046
Previously, the modifier name was used to identify it in a compute context or
viewer path. Using `ModifierData.persistent_uid` (which was only introduced
later) has two main benefits: * It is stable even when the modifier name
changes. * It's cheaper and easier to work with since it's just an integer
instead of a string.
Note: Pinned viewer nodes will need to be re-pinned after the change.
Pull Request: https://projects.blender.org/blender/blender/pulls/138864
Like other runtime structs, it doesn't make sense to write this
data to files. And moving it out of DNA to an allocated C++ struct
means we can use other C++ features in it, like the new Mutex
type which I switched to in this commit.
Pull Request: https://projects.blender.org/blender/blender/pulls/138551
This patch adds a new `BLI_mutex.hh` header which adds `blender::Mutex` as alias
for either `tbb::mutex` or `std::mutex` depending on whether TBB is enabled.
Description copied from the patch:
```
/**
* blender::Mutex should be used as the default mutex in Blender. It implements a subset of the API
* of std::mutex but has overall better guaranteed properties. It can be used with RAII helpers
* like std::lock_guard. However, it is not compatible with e.g. std::condition_variable. So one
* still has to use std::mutex for that case.
*
* The mutex provided by TBB has these properties:
* - It's as fast as a spin-lock in the non-contended case, i.e. when no other thread is trying to
* lock the mutex at the same time.
* - In the contended case, it spins a couple of times but then blocks to avoid draining system
* resources by spinning for a long time.
* - It's only 1 byte large, compared to e.g. 40 bytes when using the std::mutex of GCC. This makes
* it more feasible to have many smaller mutexes which can improve scalability of algorithms
* compared to using fewer larger mutexes. Also it just reduces "memory slop" across Blender.
* - It is *not* a fair mutex, i.e. it's not guaranteed that a thread will ever be able to lock the
* mutex when there are always more than one threads that try to lock it. In the majority of
* cases, using a fair mutex just causes extra overhead without any benefit. std::mutex is not
* guaranteed to be fair either.
*/
```
The performance benchmark suggests that the impact is negilible in almost
all cases. The only benchmarks that show interesting behavior are the once
testing foreach zones in Geometry Nodes. These tests are explicitly testing
overhead, which I still have to reduce over time. So it's not unexpected that
changing the mutex has an impact there. What's interesting is that on macos the
performance improves a lot while on linux it gets worse. Since that overhead
should eventually be removed almost entirely, I don't really consider that
blocking.
Links:
* Documentation of different mutex flavors in TBB:
https://www.intel.com/content/www/us/en/docs/onetbb/developer-guide-api-reference/2021-12/mutex-flavors.html
* Older implementation of a similar mutex by me:
https://archive.blender.org/developer/differential/0016/0016711/index.html
* Interesting read regarding how a mutex can be this small:
https://webkit.org/blog/6161/locking-in-webkit/
Pull Request: https://projects.blender.org/blender/blender/pulls/138370
Previously to handle unevaluated objects in line art "load all"
iteration, a `include_objects` variable is added in depsgraph
iteration settings, and this was only processed by object iterator
but not for any of the dupli objects. Now `make_duplis_collection`
will also handle `include_objects`
The only case where this border case can lead to crash is that a
line art grease pencil object is inside one of the dupli collection,
which isn't a valid use in the first place. But this fix makes it
more robust.
Pull Request: https://projects.blender.org/blender/blender/pulls/137323
Checking whether relations already existed was a bottleneck when building
collection -> object hierarchy relations. The duplicate checking would be
unnecessary since an object can only be in a collection once, but this function
to build relations can be called multiple times for the same collection.
To avoid the need to check for duplicates, only run the relation creation
once for each collection. The `BuilderMap` gives a simple way to do that.
Note that because a collection might be built separately as a non-layer
collection, there may still be duplicate hierarchy relations.
Building a Depsgraph for 60k simple objects became 1.86x faster, from 2.39
to 1.29 seconds.
Pull Request: https://projects.blender.org/blender/blender/pulls/137737
* Remove `DEG_get_evaluated_object` in favor of `DEG_get_evaluated`.
* Remove `DEG_is_original_object` in favor of `DEG_is_original`.
* Remove `DEG_is_evaluated_object` in favor of `DEG_is_evaluated`.
Pull Request: https://projects.blender.org/blender/blender/pulls/138317
Add a new node to the dependency graph, to act as a single entry point
before drivers are evaluated in parallel.
The node will take all the driven RNA properties, and write their
current value to the property again. This ensures that any implicitly
shared data is copied to ensure writability. Subsequent concurrent
writes by the driver evaluation will then be safe, as the
thread-unsafe part has already been performed.
Fixes: #132423
Pull Request: https://projects.blender.org/blender/blender/pulls/135802
Ref: #132179
Renames:
- `Editing.act_seq` -> `Editing.act_strip`
- `SequenceModifierData` -> `StripModifierData`
- Its member `mask_sequence` is now `mask_strip`.
- `MetaStack.parseq` -> `MetaStack.parent_strip`
- Remaining function names/parameters that were not dealt with in #132748
- Various references to `seq` or `sequence` throughout code and docs when
referring to a strip
Also moves `_get` to the end of the renamed function names where
applicable for standardization (unless "by" or "from" are used).
There should be no changes to current behavior.
Pull Request: https://projects.blender.org/blender/blender/pulls/138077
Allows to log human-readable list of recalc bits that are set.
The usage is pretty simple:
```
printf("%s: %s %s\n", __func__, id->name, DEG_stringify_recalc_flags(id->recalc).c_str());
```
Pull Request: https://projects.blender.org/blender/blender/pulls/137715
The goal here is to avoid having to cast to and from `ID` when getting the
evaluated or original ID using the depsgraph API, which is often verbose and not
type safe. To solve this, there are now `DEG_get_original` and
`DEG_get_evaluated` methods which are templated on the type and use a new
`is_ID_v` static type check to make sure it's only used with valid types.
This allows removing quite some verbosity on all the call sites. I also removed
`DEG_get_original_object`, because that does not have to be a special case
anymore.
Pull Request: https://projects.blender.org/blender/blender/pulls/137629
Use `blender::Stack` instead of `BLI_Stack`. Theoretically this can
improve performance because it makes use of inlining to remove function
call overhead and give the compiler more information. I observed a build
time improvement of about 4%, from 1156 to 1113 ms with 30k cube objects.
Notably, now `deg_graph_flush_visibility_flags` doesn't call any
non-inlined functions.
Pull Request: https://projects.blender.org/blender/blender/pulls/137355
I observed allocation becoming a bottleneck when building the depsgraph
with scenes with many simple data-blocks. One of the main culprits was
the struct that encodes relations between nodes in the graph.
Instead of allocating each `Relation` with a separate allocation call,
combine them into a `LinearAllocator`. That is must faster because it
allocates in large chunks and just bumps an offset on each allocation.
In a test file with 30 thousand cube objects, I observe a 1.18x
improvement in depsgraph evaluation time, from 1370 to 1163 ms.
The depsgraph isn't completely re-allocated when it's rebuilt, so the
allocator added in this PR has to be cleared manually. In the future,
it can be used for other structs or potentially strings.
Pull Request: https://projects.blender.org/blender/blender/pulls/137303
Adding "writeback" callbacks to the depsgraph should be temporary, since these
callbacks can bind pointers that become invalid. In this case: the evaluated
nodes modifier bound by a bake node callback. These callbacks are cleared after
being added in the `deg_flush_updates_and_refresh` function, but there are other
cases where depsgraph updates are executed which don't support writeback
callbacks (`object_force_modifier_update_for_bind` run by the smooth modifier
"bind" function).
To prevent dangling invalid pointers in outdated callbacks, disallow adding
callbacks in any case other than the `deg_flush_updates_and_refresh` function.
Since callbacks can be added from any depsgraph update, the safest way to
prevent adding them is in the depsgraph itself. If the `use_writeback_callbacks`
flag is not set, any callback is simply discarded.
Pull Request: https://projects.blender.org/blender/blender/pulls/137083
When the collection is disabled and was only referenced by a node tree, it was
missing from the depsgraph. The Geometry Nodes modifier tried to add a relation
to it, but it couldn't because the collection was not in the depsgraph in the first place.
Now, the collection is always added to the depsgraph when it's referenced by a
node tree that's in the depsgraph too. This makes the existing code that adds the
relation from the collection to the Geometry Nodes modifier work.
Pull Request: https://projects.blender.org/blender/blender/pulls/136328
This implements bundles and closures which are described in more detail in this
blog post: https://code.blender.org/2024/11/geometry-nodes-workshop-october-2024/
tl;dr:
* Bundles are containers that allow storing multiple socket values in a single
value. Each value in the bundle is identified by a name. Bundles can be
nested.
* Closures are functions that are created with the Closure Zone and can be
evaluated with the Evaluate Closure node.
To use the patch, the `Bundle and Closure Nodes` experimental feature has to be
enabled. This is necessary, because these features are not fully done yet and
still need iterations to improve the workflow before they can be officially
released. These iterations are easier to do in `main` than in a separate branch
though. That's because this patch is quite large and somewhat prone to merge
conflicts. Also other work we want to do, depends on this.
This adds the following new nodes:
* Combine Bundle: can pack multiple values into one.
* Separate Bundle: extracts values from a bundle.
* Closure Zone: outputs a closure zone for use in the `Evaluate Closure` node.
* Evaluate Closure: evaluates the passed in closure.
Things that will be added soon after this lands:
* Fields in bundles and closures. The way this is done changes with #134811, so
I rather implement this once both are in `main`.
* UI features for keeping sockets in sync (right now there are warnings only).
One bigger issue is the limited support for lazyness. For example, all inputs of
a Combine Bundle node will be evaluated, even if they are not all needed. The
same is true for all captured values of a closure. This is a deeper limitation
that needs to be resolved at some point. This will likely be done after an
initial version of this patch is done.
Pull Request: https://projects.blender.org/blender/blender/pulls/128340
While profiling a scene with many objects, I noticed some unexpected
functions taking a significant time of depsgraph creation/evaluation.
String length calculation and equality comparison was taking longer than
it should, and some simple methods were appearing in profiles that should
be inlined instead.
There are more places where this sort of change would be helpful, this
commit just changes places using `OperationIDKey` and `ComponentIDKey`.
Pull Request: https://projects.blender.org/blender/blender/pulls/136795
This replaces the deprecated DrawData mechanism by the
usage of the update timestamp `last_update`.
The compositor keeps the `last_update` value of the cached ID
and compares it with the value on the ID at the time of evaluation.
Rel #134690
Pull Request: https://projects.blender.org/blender/blender/pulls/134878
This was cause by the legacy undo recreating the depsgraph for
every update.
Depsgraph is recreated for each undo step which make the update
counter stay at 1. The fix is to protect for this case using a
global atomic counter.
This PR creates 2 namespaces for VSE code:
- `blender::seq` for sequencer core code
- `blender::ed::vse` for editor code
These names are chosen to not be in conflict with each other.
No namespace was used for RNA.
Finally, file `BKE_sequencer_offscreen.h` was moved from BKE to sequencer.
Pull Request: https://projects.blender.org/blender/blender/pulls/135500
The general idea is to keep the 'old', C-style MEM_callocN signature, and slowly
replace most of its usages with the new, C++-style type-safer template version.
* `MEM_cnew<T>` allocation version is renamed to `MEM_callocN<T>`.
* `MEM_cnew_array<T>` allocation version is renamed to `MEM_calloc_arrayN<T>`.
* `MEM_cnew<T>` duplicate version is renamed to `MEM_dupallocN<T>`.
Similar templates type-safe version of `MEM_mallocN` will be added soon
as well.
Following discussions in !134452.
NOTE: For now static type checking in `MEM_callocN` and related are slightly
different for Windows MSVC. This compiler seems to consider structs using the
`DNA_DEFINE_CXX_METHODS` macro as non-trivial (likely because their default
copy constructors are deleted). So using checks on trivially
constructible/destructible instead on this compiler/system.
Pull Request: https://projects.blender.org/blender/blender/pulls/134771
This patch refactors static cache invalidation of images by tracking an
update count. Images now store a runtime update count that is updated
every time the image is tagged for update. Cached images store a copy of
the update count at the moment they were cached, and are invalidated if
if it changed.
Compared to #134878, this is simpler and more robust, since update IDs
are isolated to images only and not to the DEG update count. Though this
only supports images specifically because they are not covered by the
copy-on-evaluation system, which means #134878 will cause multiple
depsgraph to fight over images.
Pull Request: https://projects.blender.org/blender/blender/pulls/134905
Grease Pencil ID data was being evaluated in the
`BKE_grease_pencil_data_update` function, but this function
is called from the evaluated objects.
The fix adds a new `BKE_grease_pencil_eval_geometry` function
thats called as a callback from `build_object_data_geometry_datablock`.
Resolves#134980, #135004.
Pull Request: https://projects.blender.org/blender/blender/pulls/135051
The main goal of these changes are to improve static (i.e. build-time)
checks on whether a given data can be allocated and freed with `malloc`
and `free` (C-style), or requires proper C++-style construction and
destruction (`new` and `delete`).
* Add new `MEM_malloc_arrayN_aligned` API.
* Make `MEM_freeN` a template function in C++, which does static assert on
type triviality.
* Add `MEM_SAFE_DELETE`, similar to `MEM_SAFE_FREE` but calling
`MEM_delete`.
The changes to `MEM_freeN` was painful and useful, as it allowed to fix a bunch
of invalid calls in existing codebase already.
It also highlighted a fair amount of places where it is called to free incomplete
type pointers, which is likely a sign of badly designed code (there should
rather be an API to destroy and free these data then, if the data type is not fully
publicly exposed). For now, these are 'worked around' by explicitly casting the
freed pointers to `void *` in these cases - which also makes them easy to search for.
Some of these will be addressed separately (see blender/blender!134765).
Finally, MSVC seems to consider structs defining new/delete operators (e.g. by
using the `MEM_CXX_CLASS_ALLOC_FUNCS` macro) as non-trivial. This does not
seem to follow the definition of type triviality, so for now static type checking in
`MEM_freeN` has been disabled for Windows. We'll likely have to do the same
with type-safe `MEM_[cm]allocN` API being worked on in blender/blender!134771
Based on ideas from Brecht in blender/blender!134452
Pull Request: https://projects.blender.org/blender/blender/pulls/134463
`DEGCustomDataMeshMasks` makes `IDInfo` struct non-trivial, so it should
not be allocated or freed with C-style allocation.
Instead of allocating this data, now use its only persistent user
(the `DepsgraphNodeBuilder::id_info_hash_` map) as owner, move the rest
of the code releasing COW IDs to a new destructor for `IDInfo`, and let
standard C++ destruction do the job.
NOTE: This change seems to give some performances improvements in
affected code (e.g. about 4% improvement in affected area of
`DepsgraphNodeBuilder::begin_build`). However, this does not affect global
performances in a measurable way.
Related to work on improving our memory allocation code in !134463.
Pull Request: https://projects.blender.org/blender/blender/pulls/134563
This fixes the (unreported) issue where solo'ing an NLA track would only
play back animation when _any_ of the tracks were unmuted.
Some code considered a track to be muted when its `NLATRACK_MUTED` flag
was set regardless of the `NLATRACK_SOLO` flag, whereas other code did
consider the `NLATRACK_SOLO` flag.
Now all the code is consistent with what actual animation evaluation is
doing.
Pull Request: https://projects.blender.org/blender/blender/pulls/134500
