Adds a new `source/blender/asset_system` directory and moves asset
related files from BKE to it. More asset related code can follow
(e.g. asset indexing, ED_assetlist stuff) but needs further work to
untangle it. I also kept `BKE_asset.h` and `asset.cc` as is, since they
deal with asset DNA data mostly, thus make sense in BKE.
Motivation:
- Makes the asset system design more present (term wasn't even used in
code before).
- An `asset_system` directory is quite descriptive (trivial to identify
core asset system features) and makes it easy to find asset code.
- Asset system is mostly runtime data, with little relation to other
`Main`/BKE/DNA types.
- There's a lot of stuff in BKE already. It shouldn't be just a dump for
all stuff that seems core enough.
- Being its own directly helps us be more mindful about encapsulating
the module well, and avoiding dependencies on other modules.
- We can be more free with splitting files here than in BKE.
- In future there might be an asset system BPY module, which would then
map quite nicely to the `asset_system` directory.
Checked with some other core devs, consensus seems that this makes
sense.
Introduces a new `AssetRepresentation` type, as a runtime only container
to hold asset information. It is supposed to become _the_ main way to
represent and refer to assets in the asset system, see T87235. It can
store things like the asset name, asset traits, preview and other asset
metadata.
Technical documentation:
https://wiki.blender.org/wiki/Source/Architecture/Asset_System/Back_End#Asset_Representation.
By introducing a proper asset representation type, we do an important
step away from the previous, non-optimal representation of assets as
files in the file browser backend, and towards the asset system as
backend. It should replace the temporary & hacky `AssetHandle` design in
the near future. Note that the loading of asset data still happens
through the file browser backend, check the linked to Wiki page for more
information on that.
As a side-effect, asset metadata isn't stored in file browser file
entries when browsing with link/append anymore. Don't think this was
ever used, but scripts may have accessed this. Can be brought back if
there's a need for it.
Mistake in own rB358155a8da60, change ended up discarding the case where
we need to also delete IDs using a tagged-to-be-deleted ID in a 'never
NULL' way. Typical example: Whene deleting a Mesh ID, one also needs to
delete all the Objects using that mesh, since obdata pointer is of type
'never NULL'.
Note that luckily, this fix does not affect the performance improvements
from rB358155a8da60.
Noted by Brecht and Clement because of failing unittests, shame on me.
Batch-delete by-passes a lot of ID usages handling in to-be-deleted IDs,
so usercount of deleted IDs needs to be manually reset to avoid the
annoying error messages in the console.
Simplify and optimize remapping handling in bacth ID deletion, by moving
it outside of the initial loop gathering all IDs to be deleted, and and
by using batch remapping code.
Speedup can be over 100 times faster in complex production scenes using
thousands of IDs, when e.g. deleting a whole library.
Examples before/after times on my machine (deleting two different libraries):
lib1: 5.55 sec/0.03 sec
lib2: 13.60 sec/0.13 sec
Found while investigating T101903.
Remapping in batch deletion could end up calling viewlayer resync code
on partially invalid BMain (some IDs still in Main using IDs removed
from Main).
Think this code can actually be further optimized, but this fix should
be safe enough for 3.3 (and potentially 2.93).
Thanks to Jeroen (@jbakker) for the initial investigation.
Liboverrides are doing some very low-level manipulation of IDs in apply
code, to reduce over-head of name and sorting handling.
This requires specific care to ensure thatr the new namemap runtime data
remains up-to-date and valid. Otherwise, names of existing IDs would be
missing from the map, which would later lead to having several different
IDs with the same name. Critical corruption in Blender ID management.
Reported by animators at the Blender studio.
Regression from rB7f8d05131a77.
An implementation of T73412, roughly as outlined there:
Track the names that are in use, as well as base names (before
numeric suffix) plus a bit map for each base name, indicating which
numeric suffixes are already used. This is done per-Main/Library,
per-object-type.
Timings (Windows, VS2022 Release build, AMD Ryzen 5950X):
- Scene with 10k cubes, Shift+D to duplicate them all: 8.7s -> 1.9s.
Name map memory usage for resulting 20k objects: 4.3MB.
- Importing a 2.5GB .obj file of exported Blender 3.0 splash scene
(24k objects), using the new C++ importer: 34.2s-> 22.0s. Name map
memory usage for resulting scene: 8.6MB.
- Importing Disney Moana USD scene (almost half a million objects):
56min -> 10min. Name map usage: ~100MB. Blender crashes later on
when trying to render it, in the same place in both cases, but
that's for another day.
Reviewed By: Bastien Montagne
Differential Revision: https://developer.blender.org/D14162
`BKE_id_delete` should only check for consistency of user count with
regards to the tags and flags of the ID, not 'protect' nor even warn in
case a 'fake user' ID is deleted (such higher-level checks are to be
handled by higher-level code).
Also replace the assert + debug print by a CLOG error, this avoids
'assert crash' while still failing tests, and always producing a useful
message.
Fixes T98374 and T98260.
Relinking code would weirdly enough allow clearing of extra/fake user
status on IDs used by affected ID, which would be utterly wrong.
Fairly unclear why this was working OK in reported case before rBa71a513def20,
could not spot any obvious reason just from reading code...
Also, in `libblock_remap_data_update_tags`, only transfer fake user
status if `new_id` is not NULL (otherwise that would have removed that
falg from `old_id`, without actually transferring it to anything).
Use a shorter/simpler license convention, stops the header taking so
much space.
Follow the SPDX license specification: https://spdx.org/licenses
- C/C++/objc/objc++
- Python
- Shell Scripts
- CMake, GNUmakefile
While most of the source tree has been included
- `./extern/` was left out.
- `./intern/cycles` & `./intern/atomic` are also excluded because they
use different header conventions.
doc/license/SPDX-license-identifiers.txt has been added to list SPDX all
used identifiers.
See P2788 for the script that automated these edits.
Reviewed By: brecht, mont29, sergey
Ref D14069
Part of T91671.
Not much else to say, this is mainly a massive deletion of code.
Note that a few cleanups possible after this proxy removal were kept out
of this commit to try to reduce a bit its size.
Reviewed By: sergey, brecht
Maniphest Tasks: T91671
Differential Revision: https://developer.blender.org/D13995
During sprite fright loading of complex scenes would spend a long time in remapping ID's
The remapping process is done on a per ID instance that resulted in a very time consuming
process that goes over every possible ID reference to find out if it needs to be updated.
If there are N of references to ID blocks and there are M ID blocks that needed to be remapped
it would take N*M checks. These checks are scattered around the place and memory.
Each reference would only be updated at most once, but most of the time no update is needed at all.
Idea: By grouping the changes together will reduce the number of checks resulting in improved performance.
This would only require N checks. Additional benefits is improved data locality as data is only loaded once
in the L2 cache.
It has be implemented for the resyncing process and UI editors.
On an Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz 16Gig the resyncing process went
from 170 seconds to 145 seconds (during hotspot recording).
After this patch has been applied we could add similar approach
to references (references between data blocks) and functionality (tagged deletion).
In my understanding this could reduce the resyncing process to less than a second.
Opening the village production file between 10 and 20 seconds.
Flame graphs showing that UI remapping isn't visible anymore (`WM_main_remap_editor_id_reference`)
* Master {F12769210 size=full}
* This patch {F12769211 size=full}
Reviewed By: mont29
Maniphest Tasks: T94185
Differential Revision: https://developer.blender.org/D13615
This reverts commit 948211679f.
This commit introduced some regressions in the test suite.
As this change is a core part of blender Bastien and I decided to revert
it as the solution isn't clear and needs more investigation.
The following tests FAILED:
62 - blendfile_liblink (SEGFAULT)
63 - blendfile_library_overrides (SEGFAULT)
It fails in (id_us_ensure_real)
During sprite fright loading of complex scenes would spend a long time in remapping ID's
The remapping process is done on a per ID instance that resulted in a very time consuming
process that goes over every possible ID reference to find out if it needs to be updated.
If there are N of references to ID blocks and there are M ID blocks that needed to be remapped
it would take N*M checks. These checks are scattered around the place and memory.
Each reference would only be updated at most once, but most of the time no update is needed at all.
Idea: By grouping the changes together will reduce the number of checks resulting in improved performance.
This would only require N checks. Additional benefits is improved data locality as data is only loaded once
in the L2 cache.
It has be implemented for the resyncing process and UI editors.
On an Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz 16Gig the resyncing process went
from 170 seconds to 145 seconds (during hotspot recording).
After this patch has been applied we could add similar approach
to references (references between data blocks) and functionality (tagged deletion).
In my understanding this could reduce the resyncing process to less than a second.
Opening the village production file between 10 and 20 seconds.
Flame graphs showing that UI remapping isn't visible anymore (`WM_main_remap_editor_id_reference`)
* Master {F12769210 size=full}
* This patch {F12769211 size=full}
Reviewed By: mont29
Maniphest Tasks: T94185
Differential Revision: https://developer.blender.org/D13615
- Added space below non doc-string comments to make it clear
these aren't comments for the symbols directly below them.
- Use doxy sections for some headers.
- Minor improvements to doc-strings.
Ref T92709
This commit adds to ID struct a new optional 'weak reference' to a
linked ID (in the form of a blend file library path and full ID name).
This can then be used on next append to try to find a matching local ID
instead of re-making the linked data local again.
Ref. T90545
NOTE: ID re-use will be disabled for regular append for the time being
(3.0 release), and only used for assets. Therefore, this commit should
not change anything user-wise.
Differential Revision: https://developer.blender.org/D12545
ID data-blocks that could be accessed from Python and weren't freed
using BKE_id_free_ex did not release the Python reference count.
Add BKE_libblock_free_data_py function to clear the Python reference
in this case.
Add asserts to ensure no Python reference is held in situations
when ID's are copied for internal use (not exposed through the RNA API),
to ensure these kinds of leaks don't go by unnoticed again.
Code would end up freeing some of the newly created overrides, which
were assigned to the matching linked ID's `newid` pointer, accessed
again further down the code.
Note that this is not a normal expected situation, and it won't give a
proper resync result anyway, but it might happen in some complicated
corner cases, and also quite often when dealing with older .blend files.
The `MAX_LIBARRAY` define was an annoying doublon to the `INDEX_ID_MAX` enum value
now defined in `DNA_ID.h`, and it is no more useful.
And comments were somewhat outdated. Also added an explanation about
chosen order for the `INDEX_ID_<IDTYPE>` order.
This was rather obscure and non-critical issue, but in some cases ID
usercount of some deleted IDs from batch-deletion code would not be
properly nullified, which would then assert later in actual deletion
code.
Sometimes multiple calls to `bpy.data.orphans_purge()` are needed to
delete all orphans, because a call can turn previously-used datablocks
into orphans. Returning the number of deleted datablocks makes it
possible to keep looping until nothing can be deleted any more.
This patch keeps track of deletions in `id_delete()` so that it can be
returned up the call stack.
Reviewed By: mont29
Differential Revision: https://developer.blender.org/D9918
Asset metadata is what turns a regular data-block into an asset. It is a small
data-structure, but a key part of the technical design of the asset system.
The design foresees that asset data-blocks store an `ID.asset_data` pointer of
type `AssetMetaData`. This data **must not** have dependencies on other
data-blocks or data-block data, it must be an independent unit. That way we can
read asset-metadata from .blends without reading anything else from the file.
The Asset Browser will use this metadata (together with the data-block name,
preview and file path) to represent assets in the file list.
Includes:
* New `ID.asset_data` for asset metadata.
* Asset tags, description and custom properties.
* BKE code to manage asset meta-data and asset tags.
* Code to read asset data from files, without reading IDs.
* RNA for asset metadata (including tags)
Part of the first Asset Browser milestone. Check the #asset_browser_milestone_1
project milestone on developer.blender.org.
Differential Revision: https://developer.blender.org/D9716
Reviewed by: Bastien Montagne, Brecht Van Lommel
This reverts commit 2ddecfffc3.
The correct fix is to compile with -fno-strict-aliasing on
release/relwithdebinfo builds also.
See the last commit/ {D9372}.
Maniphest Task T81077
This looks like a optimizer bug where it makes wrong assumptions.
The code inside lib_id_delete:264 on rBafd13710b897cc1c11b
`for (id = last_remapped_id->next; id; id = id->next) {..}`
is not executed in release/relwithdebinfo builds.
This can be "fixed" by several ways:
- Adding a line that prints the `last_remapped_id->name` right before
the said for-loop starts.
- Turning off optimization for the whole function `id_delete`:
`#pragma clang optimize off/on` Ray Molenkamp
- Marking `last_remapped_id` volatile. Julian Eisel
- Marking `tagged_deleted_ids` volatile. But it adds a warning when
calling `BLI_addtail`: discards volatile qualifier. Discovered by
accident.
Fix T81077
Reviewed By: mont29
Maniphest Tasks: T81077
Differential Revision: https://developer.blender.org/D9315
Trying to get shape key pointer after having unlinked its owner from
Main data-base is rather useless... So those shapekeys ended up never
being deleted.
Consistency with e.g. copying (copying a mesh also duplicates its
shapekey typically).
Baby step towards getting rid of this horrible ID, some day...
Note that for now we only do that when taking BMain into account, fancy
code dealing with non-main IDs and such has to deal with those itself.
This removes grease pencil brush creation/dat-block delete on load,
since this causes duplicate data-blocks.
Add assert to prevent this happening in the future
since the error is isn't obvious.
The `BKE_animsys.h` and `anim_sys.c` files already had a an "AnimData
API" section. The code in that section has now been split off, and
placed into `BKE_anim_data.h` and `anim_data.c`.
All files that used to include `BKE_animsys.h` have been adjusted to
only include the animation headers they need (sometimes none).
No functional changes.
This has been long standing TODO...
Note that remaining usages of BKE_xxx_delete should all be carefully
checked for and utilmately nuked in favor of `BKE_id_delete()`, think we
still have quiet a few bugs hidden in those (code seems to usually
assume those functions do a full ID deletion, which is not the case).
Only covers direct usages of new callbacks from IDTypeInfo.
We still have a lot of those switch/case, many can probably go away
with minimal refactor now, but that will be for later.
