Code handling read/write of libraries is still particular... but trying
to call `library_runtime_reset` on a random address at readtime was an
obvious mistake I should have caught during review :(
Regression from rB7f8d05131a77.
An implementation of T73412, roughly as outlined there:
Track the names that are in use, as well as base names (before
numeric suffix) plus a bit map for each base name, indicating which
numeric suffixes are already used. This is done per-Main/Library,
per-object-type.
Timings (Windows, VS2022 Release build, AMD Ryzen 5950X):
- Scene with 10k cubes, Shift+D to duplicate them all: 8.7s -> 1.9s.
Name map memory usage for resulting 20k objects: 4.3MB.
- Importing a 2.5GB .obj file of exported Blender 3.0 splash scene
(24k objects), using the new C++ importer: 34.2s-> 22.0s. Name map
memory usage for resulting scene: 8.6MB.
- Importing Disney Moana USD scene (almost half a million objects):
56min -> 10min. Name map usage: ~100MB. Blender crashes later on
when trying to render it, in the same place in both cases, but
that's for another day.
Reviewed By: Bastien Montagne
Differential Revision: https://developer.blender.org/D14162
New remapper code would also fail in some cases when remapping
libraries, similar to the issue yesterday, because ID_LI type had no
mask value.
That would fail to remap `parent` member of a library to NULL when
deleting that parent, leading to a crash e.g. in Outliner tree building
code.
Reported by @JulienKaspar from Blender studio.
Use a shorter/simpler license convention, stops the header taking so
much space.
Follow the SPDX license specification: https://spdx.org/licenses
- C/C++/objc/objc++
- Python
- Shell Scripts
- CMake, GNUmakefile
While most of the source tree has been included
- `./extern/` was left out.
- `./intern/cycles` & `./intern/atomic` are also excluded because they
use different header conventions.
doc/license/SPDX-license-identifiers.txt has been added to list SPDX all
used identifiers.
See P2788 for the script that automated these edits.
Reviewed By: brecht, mont29, sergey
Ref D14069
The main goal of this refactor is to make BPath module use `IDTypeInfo`,
and move each ID-specific part of the `foreach_path` looper into their
own IDTypeInfo struct, using a new `foreach_path` callback.
Additionally, following improvements/cleanups are included:
* Attempt to get better, more consistent namings.
** In particular, move from `path_visitor` to more standard `foreach_path`.
* Update and extend documentation.
** API doc was moved to header, according to recent discussions on this
topic.
* Remove `BKE_bpath_relocate_visitor` from API, this is specific
callback that belongs in `lib_id.c` user code.
NOTE: This commit is expected to be 100% non-behavioral-change. This
implies that several potential further changes were only noted as
comments (like using a more generic solution for
`lib_id_library_local_paths`, addressing inconsistencies like path of
packed libraries always being skipped, regardless of the
`BKE_BPATH_FOREACH_PATH_SKIP_PACKED` `eBPathForeachFlag` flag value,
etc.).
NOTE: basic unittests were added to master already in
rBdcc500e5a265093bc9cc.
Reviewed By: brecht
Differential Revision: https://developer.blender.org/D13381
Two issues addressed here:
I) `asset_type_info` is sub-data, not a callback. Therefore, move it
before the callbacks in the `IDTypeInfo` struct.
II) More important, initialize this new attribute in *ALL* `IDTypeInfo`
instances. No member of this struct should ever be left implicitely
uninitilazed, ever.
Aftermath of rBa84f1c02d251.
This flag became a full duplicate of `IDTYPE_FLAGS_NO_LIBLINKING`, which
is a good thing (don't think we ever want to be able to link some data,
without being able to make it local...).
So we can now remove it and use `IDTYPE_FLAGS_NO_LIBLINKING` instead.
This concerns currently only collections (`master_collection` of scenes)
and root node trees. It removes the matching type-specific helpers
(`BKE_collection_master_scene_search` and `BKE_node_tree_find_owner_ID`).
No functional change expected here.
NOTE: Current implementation of `owner_get` is far from optimal, we
could probably do it better, see {T69169}.
NOTE: While it could also have it, shapekeys IDTypeInfo was left out of
this change for now. Mainly because it sould not be used currently, and
we ultimately want to demote shape keys from ID status anyway.
Currently this is needed to properly tag PointCache's for info update
(fixes an issue reported in T82503).
Suspect we may need this in more cases in the future though, RNA
assign/update processes are not always 100% enough to deal with
complicated corner cases.
This is essentially adding that new callback, and using it only for already
existing Scene's 3DCursor.
Note that the place where this is called has been moved again, after all
have been lib-linked, such that those callbacks may also work on ID pointers.
Maniphest Tasks: T71759
Differential Revision: https://developer.blender.org/D9237
This is part of T76372.
It adds the `blend_write`, `blend_read_data`, `blend_read_lib`
and `blend_read_expand` which correspond to the various
steps when reading and writing .blend files.
Having these callbacks allows us to decentralize the blenloader
code a lot more. This has the affect that code related to any
specific ID type is less scattered.
Reviewers: mont29
Differential Revision: https://developer.blender.org/D8670
Even though we do not have much of those, this might change in the
future, and in any case having specific functions for this ID type in
generic `BKE_lib` area was really confusing.
We can't use the fast path when the mesh is used by mulitple objects and so
slower sculpting is expected then. But fake users should not affect this. This
also fixes the same type of error in a few other areas.
This commit affects `id_sort_by_name()` and `check_for_dupid()` helper:
* Add a new parameter, `ID *id_sorting_hint`, to `id_sort_by_name()`,
and when non-NULL, check if we can insert `id` immediately before or
after it. This can dramatically reduce time spent in that function.
* Use loop over whole list in `check_for_dupid()` to also define the
likely ID pointer that will be neighbor with our new one.
This gives another decent speedup to all massive addition cases:
| Number and type of names of IDs | old code | new code | speed improvement |
| -------------------------------- | -------- | -------- | ----------------- |
| 40K, mixed (14k rand, 26k const) | 39s | 33s | 18% |
| 40K, fully random | 51s | 42s | 21% |
| 40K, fully constant | 40s | 34s | 18% |
Combined with the previous commits, this makes massive addition of IDs more
than twice as fast as previously.
This commit affects `check_for_dupid()` helper:
* Add a special, quicker code path dedicated to sequential addition of a
large number of IDs using the same base name.
This gives a significant speedup to adding 'randomly'-named IDs:
| Number and type of names of IDs | old code | new code | speed improvement |
| -------------------------------- | -------- | -------- | ----------------- |
| 40K, mixed (14k rand, 26k const) | 49s | 39s | 26% |
| 40K, fully random | 51s | 51s | 0% |
| 40K, fully constant | 71s | 40s | 78% |
Note that 'random' names give no improvement as expected, since this new code
path will never be used in such cases.
This commit affects `check_for_dupid()` helper:
* Further simplify the general logic of the code (we now typically only do
one loop over the list of data-blocks, instead of two).
This gives a significant speedup to adding 'randomly'-named IDs:
| Number and type of names of IDs | old code | new code | speed improvement |
| -------------------------------- | -------- | -------- | ----------------- |
| 40K, mixed (14k rand, 26k const) | 62s | 49s | 27% |
| 40K, fully random | 76s | 51s | 49% |
| 40K, fully constant | 77s | 71s | 8% |
Note that 'constant' names give little improvement, as in that case the first
loop over the list of IDs (checking whether base given name was already in use)
was aborting very quickly.
This commit affects `check_for_dupid()` helper:
* Fix (serious though rare) bug where several IDs could end up with
exact same name (happened with over 10k IDs with same very long name).
* Fix (relatively harmless) func reporting it did not change the given
name when it actually had truncated it.
* Sanitize handling of supported min/max number suffixes (it now handles
between 1 and 1 billion, which should be way more than enough).
* Sanitize general logic to (hopefully!) make it easier to follow.
* General cleanup (naming, comments, variables scope, remove dead code, etc.).
Note that general performances here remain the same, there is no
measurable gain or loss. Algorithm remain also the same globally.
Attempt to use a GHash to speed up checks of used names proved to be
much worse, just building the GHash would already take as much time as
the whole process with current code...
This alone can make e.g. adding 40k IDs with default name (i.e. 'fully
constant' case in table below) about 15-20% faster:
| Number and type of names of IDs | old code | new code | speed improvement |
| -------------------------------- | -------- | -------- | ----------------- |
| 40K, mixed (14k rand, 26k const) | 75s | 62s | 21% |
| 40K, fully random | 90s | 76s | 18% |
| 40K, fully constant | 94s | 77s | 22% |
Idea is to use a first pass, where we just check one item every nth ones,
until we have found the chunk in which we want to insert the new item,
then do a basic linear comparison with all items in that chunk as before.
Also, list is now walked backward, as in most common 'massive insertion'
cases new ID's names have higher number, hence ends up towards the end of
the list.
This new sorting function can be between a few percents and 50% quicker than
original code, depending on various factors (like length of common parts of
ID names, whether new IDs should be added towards the end or not, how high
in numbering we are, etc.).
Note: another, full bisecting approach was also tried, which gives a massive
reduction of comparisons (from n items to log2(n)), but it only gave minor
improvements of speed from original fully linear code, while adding a fair
amount of complexity to the logic. Only case it was shining was the unlikely
'very long, almost similar ID names' case (since `strcasecmp()` is rather
costly then).
Note that the issue also affected animdata handling...
Checked all usages of the `BKE_libblock_copy_ex()` function, and
think never handling user count here is valid, although a bit risky
maybe. But other solution would be to add yet another copy flag, so
would rather go with that one for now.
The problem was that the object and collection pointers in Base and
LayerCollection would get lost of file read. Normally such ID pointers would
be resolved by pointing to an ID_ID placeholder which has the datablock name,
and then replacing it will the real datablock. However ID_ID is only written
for directly linked datablocks.
This adds the concept of an indirectly linked datablock with a weak reference
to it. For this we write an ID_ID_WEAK_REF code, which is a reference that
will only be resolved if the datablock was read for another reason.
Differential Revision: https://developer.blender.org/D4416
Another sneaky bite from the infamous private ID data: While those
monsters are not in bmain, the actions used by their animdata are
regular cute little ID's, living with the herd in the safe and sound
Main DB...
So we have to be careful not to propagate the nasty black magic
required to handle the formers when we duplicate their animdata.
Saying it again: private ID datablocks should never have had their own
animdata & actions, this is endless issue also with RNA paths... And
makes copying of animation between materials and such needlessly
complicated.
Private ID data (nodetrees and scene collections...) need special care
and handling of their copy flags, and checks must be adapted too.
In that case, issue came from the fact that even though those IDs have
to be copied outside of bmain, we may still require usercount handling.
That commit also fixes a somewhat related issue - we cannot use the
non-id private data copying flag for private IDs copying, due to
difference in handling of usercount again.
NodeTree structures of materials and some other data blocks are
effectively node group datablock objects that are contained inside
the parent block. Thus, direct references to them are only valid
while blender is running, and are lost on save.
Fix Copy As New Driver to create a reference that goes through
the owner datablock, by adding a new ID flag to mark private
pseudo-datablocks.
Also fix functions that return full paths to structures and
properties, e.g. used in python tooltips. Functions for paths
from ID to struct or property can't be changed because of
Animation Data related code.
Reviewers: mont29
Differential Revision: https://developer.blender.org/D5559
