Relinking code would weirdly enough allow clearing of extra/fake user
status on IDs used by affected ID, which would be utterly wrong.
Fairly unclear why this was working OK in reported case before rBa71a513def20,
could not spot any obvious reason just from reading code...
Also, in `libblock_remap_data_update_tags`, only transfer fake user
status if `new_id` is not NULL (otherwise that would have removed that
falg from `old_id`, without actually transferring it to anything).
While this only had minor potential effect, both code incrementing
usercount of newly remapped IDs were wrong.
Original one would by-pass any 'ensured user' handling, newer one would
systematically make the ID directly linked...
`id_us_plus_no_lib` is to be used here.
We better handle NULL object pointers before doing layer collections
resync, otherwise said resync process has to deal with those NULL
pointers. By the look of it this mistake has been there since the origin
of the remapping/relinking code.
Also for safety (and optimization), do not perform layer collection
resync from `libblock_remap_data_postprocess_object_update` when
`libblock_remap_data_postprocess_collection_update` is called
immediately afterwards.
Also added same 'skip on NULL collection object pointer' check to
`layer_collection_local_sync` as the one in
`layer_collection_objects_sync`, since it's fairly hard to always
guaranty there is no such NULL pointer when calling that code.
Caused by rB43bc494892c3, moving this 'new id' relink to generic
remapping code added the over-head of proper, generic post-processing,
compared to the special-cases previous code was only designed to handle.
Fortunately with recent 'multi-remapping' work we can easily rewrite
that new id relink code to use the multi-remapping approach too.
No behavioral change is expected from this commit, besides the improved
performances (essentially restored to what they where before
rB43bc494892c3).
Similar to other changes to ID remapping, gives huge speedups in some
cases, like certain types of liboverride creation.
Case from {T96092} goes from 1725 seconds (almost 30 minutes) to 45
seconds to generate the liboverride, on my machine.
Reviewed By: jbakker
Maniphest Tasks: T96092
Differential Revision: https://developer.blender.org/D14240
This commit renames enums related the "Curve" object type and ID type
to add `_LEGACY` to the end. The idea is to make our aspirations clearer
in the code and to avoid ambiguities between `CURVE` and `CURVES`.
Ref T95355
To summarize for the record, the plans are:
- In the short/medium term, replace the `Curve` object data type with
`Curves`
- In the longer term (no immediate plans), use a proper data block for
3D text and surfaces.
Differential Revision: https://developer.blender.org/D14114
This patch increases the performance when remapping data.
{D13615} introduced a mechanism to remap multiple items in a single go.
This patch uses the same mechanism when remapping data inside ID datablocks.
Benchmark results when loading the village scene of sprite fright on AMD Ryzen 7 3800X 8-Core Processor
Before this patch 115 seconds
When patch applied less than 43 seconds
There is still some room for improvement by porting relink code.
Reviewed By: mont29
Maniphest Tasks: T95279
Differential Revision: https://developer.blender.org/D14043
Use a shorter/simpler license convention, stops the header taking so
much space.
Follow the SPDX license specification: https://spdx.org/licenses
- C/C++/objc/objc++
- Python
- Shell Scripts
- CMake, GNUmakefile
While most of the source tree has been included
- `./extern/` was left out.
- `./intern/cycles` & `./intern/atomic` are also excluded because they
use different header conventions.
doc/license/SPDX-license-identifiers.txt has been added to list SPDX all
used identifiers.
See P2788 for the script that automated these edits.
Reviewed By: brecht, mont29, sergey
Ref D14069
Based on discussions from T95355 and T94193, the plan is to use
the name "Curves" to describe the data-block container for multiple
curves. Eventually this will replace the existing "Curve" data-block.
However, it will be a while before the curve data-block can be replaced
so in order to distinguish the two curve types in the UI, "Hair Curves"
will be used, but eventually changed back to "Curves".
This patch renames "hair-related" files, functions, types, and variable
names to this convention. A deep rename is preferred to keep code
consistent and to avoid any "hair" terminology from leaking, since the
new data-block is meant for all curve types, not just hair use cases.
The downside of this naming is that the difference between "Curve"
and "Curves" has become important. That was considered during
design discussons and deemed acceptable, especially given the
non-permanent nature of the somewhat common conflict.
Some points of interest:
- All DNA compatibility is lost, just like rBf59767ff9729.
- I renamed `ID_HA` to `ID_CV` so there is no complete mismatch.
- `hair_curves` is used where necessary to distinguish from the
existing "curves" plural.
- I didn't rename any of the cycles/rendering code function names,
since that is also used by the old hair particle system.
Differential Revision: https://developer.blender.org/D14007
Part of T91671.
Not much else to say, this is mainly a massive deletion of code.
Note that a few cleanups possible after this proxy removal were kept out
of this commit to try to reduce a bit its size.
Reviewed By: sergey, brecht
Maniphest Tasks: T91671
Differential Revision: https://developer.blender.org/D13995
During sprite fright loading of complex scenes would spend a long time in remapping ID's
The remapping process is done on a per ID instance that resulted in a very time consuming
process that goes over every possible ID reference to find out if it needs to be updated.
If there are N of references to ID blocks and there are M ID blocks that needed to be remapped
it would take N*M checks. These checks are scattered around the place and memory.
Each reference would only be updated at most once, but most of the time no update is needed at all.
Idea: By grouping the changes together will reduce the number of checks resulting in improved performance.
This would only require N checks. Additional benefits is improved data locality as data is only loaded once
in the L2 cache.
It has be implemented for the resyncing process and UI editors.
On an Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz 16Gig the resyncing process went
from 170 seconds to 145 seconds (during hotspot recording).
After this patch has been applied we could add similar approach
to references (references between data blocks) and functionality (tagged deletion).
In my understanding this could reduce the resyncing process to less than a second.
Opening the village production file between 10 and 20 seconds.
Flame graphs showing that UI remapping isn't visible anymore (`WM_main_remap_editor_id_reference`)
* Master {F12769210 size=full}
* This patch {F12769211 size=full}
Reviewed By: mont29
Maniphest Tasks: T94185
Differential Revision: https://developer.blender.org/D13615
This reverts commit 948211679f.
This commit introduced some regressions in the test suite.
As this change is a core part of blender Bastien and I decided to revert
it as the solution isn't clear and needs more investigation.
The following tests FAILED:
62 - blendfile_liblink (SEGFAULT)
63 - blendfile_library_overrides (SEGFAULT)
It fails in (id_us_ensure_real)
During sprite fright loading of complex scenes would spend a long time in remapping ID's
The remapping process is done on a per ID instance that resulted in a very time consuming
process that goes over every possible ID reference to find out if it needs to be updated.
If there are N of references to ID blocks and there are M ID blocks that needed to be remapped
it would take N*M checks. These checks are scattered around the place and memory.
Each reference would only be updated at most once, but most of the time no update is needed at all.
Idea: By grouping the changes together will reduce the number of checks resulting in improved performance.
This would only require N checks. Additional benefits is improved data locality as data is only loaded once
in the L2 cache.
It has be implemented for the resyncing process and UI editors.
On an Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz 16Gig the resyncing process went
from 170 seconds to 145 seconds (during hotspot recording).
After this patch has been applied we could add similar approach
to references (references between data blocks) and functionality (tagged deletion).
In my understanding this could reduce the resyncing process to less than a second.
Opening the village production file between 10 and 20 seconds.
Flame graphs showing that UI remapping isn't visible anymore (`WM_main_remap_editor_id_reference`)
* Master {F12769210 size=full}
* This patch {F12769211 size=full}
Reviewed By: mont29
Maniphest Tasks: T94185
Differential Revision: https://developer.blender.org/D13615
Fix is similar to how CollectionObject with NULL object pointers are handled.
Using one of the 'free' pad bytes in Object_Runtime struct instead of a
gset (or other external way to detect object duplicates), as this is
several times faster.
NOTE: This makes remapping slightly slower again (adds 10 extra seconds
to file case in T94059).
General improvements of remapping time complexity, especially when
remapping a lot of IDs at once, is a separate topic currently
investigated in D13615.
Goals of this refactor:
* More unified approach to updating everything that needs to be updated
after a change in a node tree.
* The updates should happen in the correct order and quadratic or worse
algorithms should be avoided.
* Improve detection of changes to the output to avoid tagging the depsgraph
when it's not necessary.
* Move towards a more declarative style of defining nodes by having a
more centralized update procedure.
The refactor consists of two main parts:
* Node tree tagging and update refactor.
* Generally, when changes are done to a node tree, it is tagged dirty
until a global update function is called that updates everything in
the correct order.
* The tagging is more fine-grained compared to before, to allow for more
precise depsgraph update tagging.
* Depsgraph changes.
* The shading specific depsgraph node for node trees as been removed.
* Instead, there is a new `NTREE_OUTPUT` depsgrap node, which is only
tagged when the output of the node tree changed (e.g. the Group Output
or Material Output node).
* The copy-on-write relation from node trees to the data block they are
embedded in is now non-flushing. This avoids e.g. triggering a material
update after the shader node tree changed in unrelated ways. Instead
the material has a flushing relation to the new `NTREE_OUTPUT` node now.
* The depsgraph no longer reports data block changes through to cycles
through `Depsgraph.updates` when only the node tree changed in ways
that do not affect the output.
Avoiding unnecessary updates seems to work well for geometry nodes and cycles.
The situation is a bit worse when there are drivers on the node tree, but that
could potentially be improved separately in the future.
Avoiding updates in eevee and the compositor is more tricky, but also less urgent.
* Eevee updates are triggered by calling `DRW_notify_view_update` in
`ED_render_view3d_update` indirectly from `DEG_editors_update`.
* Compositor updates are triggered by `ED_node_composite_job` in `node_area_refresh`.
This is triggered by calling `ED_area_tag_refresh` in `node_area_listener`.
Removing updates always has the risk of breaking some dependency that no
one was aware of. It's not unlikely that this will happen here as well. Adding
back missing updates should be quite a bit easier than getting rid of
unnecessary updates though.
Differential Revision: https://developer.blender.org/D13246
- Added space below non doc-string comments to make it clear
these aren't comments for the symbols directly below them.
- Use doxy sections for some headers.
- Minor improvements to doc-strings.
Ref T92709
rB43bc494892c3 switched `BKE_libblock_relink_to_newid` to use new ID
remapping and libquery code.
However, that new code does protect by default against remapping an
objects's data pointer when that object is in Edit mode, since this is
not a behavior that generic BKE code can handle (due to required editing
data for most obdata types when in edit mode).
So specific code that does create new IDs and need remapping in Edit
mode has to pass specific exception flags to remaping code.
This commit adds those remapping flags to `BKE_libblock_relink_to_newid`
and add said exception flag to the remapping call from
`ED_object_add_duplicate` when the object is in edit mode.
In theory we should never allow remapping of Objects' obdata ID pointer
when the object is in Edit mode. But there are some cases were this is
needed, so adding yet another exception option to remapping flags.
Preliminary change to fix T92629.
Remapping code could call collection resync code while processing
remapping, which is a good way to crash by accessing no-more-valid
pointers.
Similar issue as with liboverrides resync, fixed the same way by
preventing any collection resync until whole remapping has been done.
This was probably not an issue in practice in current code, since this
is only used by append code currently, which should not affect
layers/collections in current scene yet.
Liboverride references need a special handling during append, since
those pointers should never be made local, nor reampped to newly
localized data. And liboverride references should never be directly made
local either, to ensure their liboverride usages remain pointing to
linked data and not local one.
Issue was reported by the studio, and also probably as part of T91892.
This implements the update logic for the vizualization of which
sockets pass data or constants directly, and which pass functions.
The socket shapes may still have to be updated. That should be
done separately, because it might be a bit more involved, because
socket shapes are currently linked to keyframe shapes. Currently
the circle and diamond shapes are used with the following meanings:
- Input Sockets:
- Circle: Required to be a single value.
- Diamond: This input supports fields.
- Output Sockets:
- Circle: This output is a single value.
- Diamond: This output may be a field.
Connecting a field to a circle input socket is an error, since a
field cannot be converted to a single value. If the socket shape
is a diamond with a dot in the middle, it means it is currently
a single value, but could be a field.
In addition to socket shapes, the intention is to draw node links
differently based on the field status. However, the exact method for
conveying that isn't decided yet.
Differential Revision: https://developer.blender.org/D12584
Current `BKE_libblock_relink_to_newid` is using its own simplistic,
limited and not really correct version of ID remapping.
While doing a full replacement would have been ideal, this is
risky/time-constrained for Blender 3.0 release, so for now we'll have
both versions co-existing.
While indeally we should only skip refcounting when relevant tag is set,
doing this in remapping code is too risky for now.
Related to previous commit and T88555.
This is complex situation. Tagged ID deletion (used to delete several
data-blocks at once) removes IDs to be deleted from Main.
But when we remove deleted IDs' usages of other IDs (using
`BKE_libblock_relink_ex`), some specific post-process is required on
some types, like Collections. Those post-processes would in some cases
rely on data actually being in Main.
That failing condition would lead in existing code on missing processing
the very ID (collection) we were working on, leading to missing removing
some child collection pointers, leading to the crash (later on in
LayerCollection resync process).
For now we go with an optimization & fix that avoids processing all
collections in Main when we actually know which one we are working one
(case of `BKE_libblock_relink_ex`, but not of
`BKE_libblock_remap_locked`).
This is however yet another demonstration of the need to rework that
whole collection/layer resync process, since it is not only extremely
inneficient currently, but it also requires valid Main/ID state way too
deep into the remapping code.
NOTE: This fix may very well not catch/address all possible fail cases
here, dealing with the double parent/child relationships of collections
is challenging...
Issue reported by @eyecandy from the studio, thanks.
In some cases (advanced, low-level), we also want to remap pointers like
`ID.newid` or `ID.orig_id`.
Only known case currently is `id_delete`, to avoid leaving potential access to freed memory. See next commit and T86501.
Keep the pointer location from the initial window-manager
between file load operations.
This is needed as the Python API may hold references to keymaps for e.g.
which are transferred to the newly loaded window manager,
without their `PointerRNA.owner_id` fields being updated.
Since there is only ever one window manager, keep the memory at the same location so the Python ID pointers stay valid.
Reviewed By: mont29
Ref D10690
Remove DNA headers, using forward declarations where possible.
Also removed duplicate header, header including it's self
and unnecessary inclusion of libc system headers from BKE header.
The parameter type was incorrectly changed in rB6be56c13e96048cbc494ba5473a8deaf2cf5a6f8 by me.
This can be any id and does not have to be a node tree.
This is the initial merge from the geometry-nodes branch.
Nodes:
* Attribute Math
* Boolean
* Edge Split
* Float Compare
* Object Info
* Point Distribute
* Point Instance
* Random Attribute
* Random Float
* Subdivision Surface
* Transform
* Triangulate
It includes the initial evaluation of geometry node groups in the Geometry Nodes modifier.
Notes on the Generic attribute access API
The API adds an indirection for attribute access. That has the following benefits:
* Most code does not have to care about how an attribute is stored internally.
This is mainly necessary, because we have to deal with "legacy" attributes
such as vertex weights and attributes that are embedded into other structs
such as vertex positions.
* When reading from an attribute, we generally don't care what domain the
attribute is stored on. So we want to abstract away the interpolation that
that adapts attributes from one domain to another domain (this is not
actually implemented yet).
Other possible improvements for later iterations include:
* Actually implement interpolation between domains.
* Don't use inheritance for the different attribute types. A single class for read
access and one for write access might be enough, because we know all the ways
in which attributes are stored internally. We don't want more different internal
structures in the future. On the contrary, ideally we can consolidate the different
storage formats in the future to reduce the need for this indirection.
* Remove the need for heap allocations when creating attribute accessors.
It includes commits from:
* Dalai Felinto
* Hans Goudey
* Jacques Lucke
* Léo Depoix
Issue was with our dear posebones again... when applying overrides we
keep the same address/pointer for the IDs themselves, (which avoids us
the need to remap their usages), but their inner data is often
re-allocated.
Therefore, we need once again to go over armature objects and invalidate
their posebone pointers.
This should also be back-ported to Blender LTS 2.83.
Maniphest Tasks: T80078
Differential Revision: https://developer.blender.org/D8734
'Private' can be a rather confusing term, especially when considering
its meaning in programming languages.
So now root node trees and master collections are 'embedded' IDs
instead.
The former is always a real, in-Main data-block, while the later, when
different, should be one of those embedded 'private' IDs (like root node
ree or master collection).
Using a struct here allows to change given parameters to the callbacks
without having to edit all callbacks functions, which is always noisy
and time consuming.
Having functions defined in `BKE_lib_id.h` implemented into
`lib_remap.c` was confusing at best.
Besides trivial code splitting and header includes cleanup, had to add a
new `lib_intern.h` header for callbacks used by both remapping and
deletion code.
