When saving asset catalogs to an already-existing file, only perform
that save when there were local changes to the catalogs. This prevents
unnecessary writes to the catalog definition file, as that can cause
conflicts when multiple Blender instances share the same asset library
(either directly or via Syncthing/Dropbox/etc.)
Pull Request: https://projects.blender.org/blender/blender/pulls/148205
This adds support for packed linked data. This is a key part of an improved
asset workflow in Blender.
Packed IDs remain considered as linked data (i.e. they cannot be edited),
but they are stored in the current blendfile. This means that they:
* Are not lost in case the library data becomes unavailable.
* Are not changed in case the library data is updated.
These packed IDs are de-duplicated across blend-files, so e.g. if a shot
file and several of its dependencies all use the same util geometry node,
there will be a single copy of that geometry node in the shot file.
In case there are several versions of a same ID (e.g. linked at different
moments from a same library, which has been modified in-between), there
will be several packed IDs.
Name collisions are averted by storing these packed IDs into a new type of
'archive' libraries (and their namespaces). These libraries:
* Only contain packed IDs.
* Are owned and managed by their 'real' library data-block, called an
'archive parent'.
For more in-depth, technical design: #132167
UI/UX design: #140870
Co-authored-by: Bastien Montagne <bastien@blender.org>
Pull Request: https://projects.blender.org/blender/blender/pulls/133801
The numeric levels have no obvious meaning. This removes the distinction
between severity and levels, instead there is a single list of named levels
with defined meaning.
Debug means information that's mainly useful for developers, and trace is for
very verbose code execution tracing.
Pull Request: https://projects.blender.org/blender/blender/pulls/140244
* Remove bke, ed and wm prefixes
* Add prefixes like: geom, object, blend, lib.
* Shorten some category names
* A few log level changes to improve --log-level info output
Pull Request: https://projects.blender.org/blender/blender/pulls/140244
Change the maximum data-block name from 64 to 256 bytes by increasing MAX_ID_NAME value.
Also increase a few related non-ID data name max size, essentially the action slots identifiers, as these are the primary key used to match an Action's slot to an ID by name.
Other sub-data (bones, modifiers, etc.) lengths are not modified here, as these can be made actual dynamic strings in the future, while keeping (a reasonable level of) forward compatibility, during the course of Blender 5 release cycles.
Implements #137608.
Co-authored-by: Bastien Montagne <bastien@blender.org>
Pull Request: https://projects.blender.org/blender/blender/pulls/137196
When initially saving a file, we would now always create a catalog
definition file that would be writted to disk on file save. This
shouldn't be done if there are no new catalogs to write.
Main issue was that the handling of catalogs for on-disk libraries
relies on an in memory version of a asset catalog definition file. This
wasn't present for the runtime current file library storage. We can
actually construct this quite easily when converting it from a runtime
to a on-disk library.
Pull Request: https://projects.blender.org/blender/blender/pulls/139881
The "All Libraries" library didn't free its assets correctly on refresh,
so the asset previews didn't refresh correctly either. That's because it
didn't forward the removal request to the asset library that actually
owns the asset. It only freed assets from its own storage, which is
always empty.
This might make refreshing the all library feel a little slower, since
previews are now refreshed too. But in general this is fairly fast still
and there's an optimization to only load visible previews too.
Asset libraries from the Preferences were not recognized as such. This
could cause various issues like the asset library not being refreshed
correctly on relevant changes. There was also a failed assert when
right-clicking the context menu of a custom brush asset, and the "Edit
Metadata" brush asset operator would be unavailable.
A pointer was unintentionally cast to a boolean, and instead null passed
for the pointer (because of a default argument value).
For automated tests we have support for directly loading a directory as
asset library. We might need this for other things too in future, so
this should be supported generally.
Directories loaded as asset libraries this way would internally be
treated as a Preferences on-disk library, which isn't correct. This
didn't cause any known issues, but with blender/blender!138150 it would
make tests fail incorrectly.
Test was failing since efdda78175. The asset library path was now using
backslashes on Windows, and got compared to a path with forward slashes. Use
our higher level path comparison function to make sure they point to the same
directory.
Convert all slashes to native format when initializing an asset library. This
might convert slashes that are valid parts of the file name, but this just
leads to an error about a not found asset library, which is better than
crashing. This is a typical tradeoff when dealing with cross platform paths.
This patch adds a new `BLI_mutex.hh` header which adds `blender::Mutex` as alias
for either `tbb::mutex` or `std::mutex` depending on whether TBB is enabled.
Description copied from the patch:
```
/**
* blender::Mutex should be used as the default mutex in Blender. It implements a subset of the API
* of std::mutex but has overall better guaranteed properties. It can be used with RAII helpers
* like std::lock_guard. However, it is not compatible with e.g. std::condition_variable. So one
* still has to use std::mutex for that case.
*
* The mutex provided by TBB has these properties:
* - It's as fast as a spin-lock in the non-contended case, i.e. when no other thread is trying to
* lock the mutex at the same time.
* - In the contended case, it spins a couple of times but then blocks to avoid draining system
* resources by spinning for a long time.
* - It's only 1 byte large, compared to e.g. 40 bytes when using the std::mutex of GCC. This makes
* it more feasible to have many smaller mutexes which can improve scalability of algorithms
* compared to using fewer larger mutexes. Also it just reduces "memory slop" across Blender.
* - It is *not* a fair mutex, i.e. it's not guaranteed that a thread will ever be able to lock the
* mutex when there are always more than one threads that try to lock it. In the majority of
* cases, using a fair mutex just causes extra overhead without any benefit. std::mutex is not
* guaranteed to be fair either.
*/
```
The performance benchmark suggests that the impact is negilible in almost
all cases. The only benchmarks that show interesting behavior are the once
testing foreach zones in Geometry Nodes. These tests are explicitly testing
overhead, which I still have to reduce over time. So it's not unexpected that
changing the mutex has an impact there. What's interesting is that on macos the
performance improves a lot while on linux it gets worse. Since that overhead
should eventually be removed almost entirely, I don't really consider that
blocking.
Links:
* Documentation of different mutex flavors in TBB:
https://www.intel.com/content/www/us/en/docs/onetbb/developer-guide-api-reference/2021-12/mutex-flavors.html
* Older implementation of a similar mutex by me:
https://archive.blender.org/developer/differential/0016/0016711/index.html
* Interesting read regarding how a mutex can be this small:
https://webkit.org/blog/6161/locking-in-webkit/
Pull Request: https://projects.blender.org/blender/blender/pulls/138370
This change moves the tests data files and publish folder of assets
repository to the main blender.git repository as LFS files.
The goal of this change is to eliminate toil of modifying tests,
cherry-picking changes to LFS branches, adding tests as part of a
PR which brings new features or fixes.
More detailed explanation and conversation can be found in the
design task.
Ref #137215
Pull Request: https://projects.blender.org/blender/blender/pulls/137219
When saving a file, move the catalog service from the runtime asset library into the new on-disk
asset library. This makes all catalog data like the undo history and deleted catalogs be preserved.
The fact that a new asset library type gets allocated is an implementation detail that shouldn't
affect behavior.
Once the service is moved, an undo push is added (so this state can be restored to), and if the new
.blend file location can be associated with a catalog definition file, its catalogs are merged in.
Includes unit tests.
Pull Request: https://projects.blender.org/blender/blender/pulls/135589
Rather new code to build the asset library reference from a library
didn't cover the case where the current file library is saved to disk,
and as such implemented as on-disk library.
Second part to fix#130007, after 50f7666785.
When saving a new file with the current file asset library loaded, the
library wasn't converted properly to an on-disk library. While the
library was loaded again as on-disk library, the previous runtime
version of it wasn't cleared, so catalog definitions could be duplicated
across available libraries.
Make sure the runtime only library is destructed properly, and the UI
refreshed.
There seem to be more issues with converting the runtime current file library to
a on-disk current file library, but these can be addressed after the crashes and
hangs are fixed, see report.
Pull Request: https://projects.blender.org/blender/blender/pulls/133341
Partial fix for #130007.
The catalog tree mutex would be set in `catalog_tree()` and
`invalidate_catalog_tree()`. The former could end up adding catalogs and thus
calling `invalidate_catalog_tree()`, which would attempt to set a mutex that the
caller set already. This seems like a good use-case for `std::recursive_mutex`.
would make sure there's a catalog for each catalog path by calling
`create_missing_catalogs()`
Brush and pose asset operators were doing some avoidable roundtrips
through asset types, lookups and rather low level operations. This
indicates that the asset system APIs need some improvements. Moving
lower-level logic there can help avoiding errors, since implementation
details are kept inside the corresponding module.
- Avoid lookups for asset library reference in operator code, make asset
library itself construct it.
- Avoid redundant lookups for asset library definition (was looking up
the asset library definition in the Preferences from the library
reference type, just to turn it into the reference type again).
- Add utility for refreshing loading asset libraries
- Add utility for saving asset catalogs for an asset
- Remove unused function
- Fixes preview flickering on actions like undo/redo in the asset shelf (#93726), not yet for the
file browser.
- Fixes#130861.
Makes the asset shelf use the asynchronous preview loading system of the UI instead of the file
browser one. The issues above where mostly caused by the file browser caching design.
The asset system and its UIs can now manage previews independently of the file browser back-end.
This is another step towards making the asset system independent of the file browser, see
https://developer.blender.org/docs/features/asset_system/fundamentals/from_file_browser_to_asset_system/.
Code to query asset previews through file browser types is removed.
Quite some work was done to prepare the UI preview system for this, to make it on par with the file
browser preview system. E.g.: 9d83061ed4, 315e7e04a8, 5055adc1c0, 16ab6111f7.
Note that the same change should be done to the asset/file browser, but this requires more work.
Pull Request: https://projects.blender.org/blender/blender/pulls/131871
When using clangd or running clang-tidy on headers there are
currently many errors. These are noisy in IDEs, make auto fixes
impossible, and break features like code completion, refactoring
and navigation.
This makes source/blender headers work by themselves, which is
generally the goal anyway. But #includes and forward declarations
were often incomplete.
* Add #includes and forward declarations
* Add IWYU pragma: export in a few places
* Remove some unused #includes (but there are many more)
* Tweak ShaderCreateInfo macros to work better with clangd
Some types of headers still have errors, these could be fixed or
worked around with more investigation. Mostly preprocessor
template headers like NOD_static_types.h.
Note that that disabling WITH_UNITY_BUILD is required for clangd to
work properly, otherwise compile_commands.json does not contain
the information for the relevant source files.
For more details see the developer docs:
https://developer.blender.org/docs/handbook/tooling/clangd/
Pull Request: https://projects.blender.org/blender/blender/pulls/132608
I found the naming of these files awkward, where we prefixed them with
`asset_library_` to keep them grouped together, and then a library type
name. This resulted in rather un-natural names. Also, there are files
like `asset_library_service.cc`, which is in fact not another asset
library type, but seems like one with the old convention.
Moving these files to an own directory keeps the grouping while allowing
more natural sounding names.
For C/C++ doc-strings should be located in headers,
move function comments into the headers, in some cases merging
with existing doc-strings, in other cases, moving implementation
notes into the function body.
Fixes#128751.
As asset libraries were added or removed (through the UI or BPY), the
"All" asset library wouldn't refresh to reflect the changes. In general
this wasn't handled well. Even a manual refresh wouldn't give the right
result.
There were multiple issues really:
- Only the first load of the "All" library would query the preferences
for the available libraries. Further loads would only refresh the
catalogs, ignoring any added/removed libraries.
- Operators and BPY functions didn't clear the asset libraries to
enforce a re-fetch.
- When clearing an asset library, the "All" library wasn't cleared in
some cases, it would show the old state still.
- The API function to clear an asset library's asset list would not
clear the storage of asset browsers so they wouldn't refresh. It makes
no sense to only do one, so let the API handle both cases.
The way we handle asset library updating could be improved generally,
and be more internal to the asset system. For now this explicit clearing
seems fine.
We also need to handle removal of libraries better still, I think they
remain in memory.
Pull Request: https://projects.blender.org/blender/blender/pulls/129541
Design of the essentials asset library is to treat it as part of
Blender (as if it were compiled into the binary), so the location and
state of these assets is clear and can be assumed in code. User edits
are not expected.
Because of that we should only look for this asset library in the
installation ("system") location, not in the user configuration which
would take priority if present. On some devs machines this location
would actually be present, making the essentials unavailable and causing
confusing/misleading warning prints, see #128420.
Match function and declaration names, picking names based on
consistency with related code & clarity.
Also changes for old conventions, missed in previous cleanups:
- name -> filepath
- tname -> newname
- maxlen -> maxncpy
Rather than a manually managed union, use `std::variant` which is
generally safer. E.g. an invalid access will now throw an exception
instead of causing undefined behavior (which may or may not crash, or
cause a data corruption). Code is also simplified this way.
Pull Request: https://projects.blender.org/blender/blender/pulls/125494
Use "asset storage" instead of just "storage", because there are other
members acting as storage (e.g. holding all catalog data), while this is
asset specific.
This was just rather useless level of abstraction. I heard from other
devs that these helper classes caused confusion, so better to avoid
this.
Now the asset representation has all the needed bits to create its full
path, blend-file path and asset library relative path. In fact only the
asset library relative path needs to be stored to make all this
available, since the asset representation already stores a reference to
the asset library owning it, so the paths can be recreated easily.
