Check if either the memory is zero or already matches the default value,
and copy. This simplifies a common pattern to a single line.
Preparing for default initializers in DNA (#134531).
Pull Request: https://projects.blender.org/blender/blender/pulls/138830
These are generic properties of grids (not stored in voxels) which are
useful to know in geometry nodes. The transform in particular defines
the voxel size. Background value is used outside of active voxels.
Pull Request: https://projects.blender.org/blender/blender/pulls/138592
This patch adds a new `BLI_mutex.hh` header which adds `blender::Mutex` as alias
for either `tbb::mutex` or `std::mutex` depending on whether TBB is enabled.
Description copied from the patch:
```
/**
* blender::Mutex should be used as the default mutex in Blender. It implements a subset of the API
* of std::mutex but has overall better guaranteed properties. It can be used with RAII helpers
* like std::lock_guard. However, it is not compatible with e.g. std::condition_variable. So one
* still has to use std::mutex for that case.
*
* The mutex provided by TBB has these properties:
* - It's as fast as a spin-lock in the non-contended case, i.e. when no other thread is trying to
* lock the mutex at the same time.
* - In the contended case, it spins a couple of times but then blocks to avoid draining system
* resources by spinning for a long time.
* - It's only 1 byte large, compared to e.g. 40 bytes when using the std::mutex of GCC. This makes
* it more feasible to have many smaller mutexes which can improve scalability of algorithms
* compared to using fewer larger mutexes. Also it just reduces "memory slop" across Blender.
* - It is *not* a fair mutex, i.e. it's not guaranteed that a thread will ever be able to lock the
* mutex when there are always more than one threads that try to lock it. In the majority of
* cases, using a fair mutex just causes extra overhead without any benefit. std::mutex is not
* guaranteed to be fair either.
*/
```
The performance benchmark suggests that the impact is negilible in almost
all cases. The only benchmarks that show interesting behavior are the once
testing foreach zones in Geometry Nodes. These tests are explicitly testing
overhead, which I still have to reduce over time. So it's not unexpected that
changing the mutex has an impact there. What's interesting is that on macos the
performance improves a lot while on linux it gets worse. Since that overhead
should eventually be removed almost entirely, I don't really consider that
blocking.
Links:
* Documentation of different mutex flavors in TBB:
https://www.intel.com/content/www/us/en/docs/onetbb/developer-guide-api-reference/2021-12/mutex-flavors.html
* Older implementation of a similar mutex by me:
https://archive.blender.org/developer/differential/0016/0016711/index.html
* Interesting read regarding how a mutex can be this small:
https://webkit.org/blog/6161/locking-in-webkit/
Pull Request: https://projects.blender.org/blender/blender/pulls/138370
OpenVDB has a voxel size limit defined by the determinant of the grid
transform, which is equivalent to a uniform voxel size of
`sqrt3(3e-15) ~= 1.44e-5`.
The `mesh_to_density_grid` function was using an arbitrary threshold of
`1.0e-5` for the uniform voxel size.
In this case the voxel size is `~1.343e-5` so it passes the Blender
threshold but crashes in OpenVDB.
This fix adds some convenience functions to check for valid grid voxel
size and transform based on the same determinant metric. This is now
employed consistently in the mesh_to_density_grid, mesh_to_sdf_grid, and
points_to_sdf_grid functions to avoid exceptions in OpenVDB.
MOD_volume_to_mesh, node_geo_volume_to_mesh, BKE_mesh_remesh_voxel have
not been modified, since they have their own error checks with larger
thresholds.
Pull Request: https://projects.blender.org/blender/blender/pulls/138481
This adds an Import VDB node. It loads all the grids from a .vdb file and hence
outputs a Volume geometry instead of an individual grid.
The grids are cached through the existing volume grid file cache, so they are
automatically deduplicated when volume grids are loaded from files in other
ways.
Pull Request: https://projects.blender.org/blender/blender/pulls/138380
This adds a version of `BKE_id_new_nomain` that takes the ID type parameter as
template argument. This allows the function the return the newly created ID with
the correct type, removing the need to use `static_cast` on the call-site.
To make this work, I added a static `id_type` member to every ID struct. This
can also be used to create a similar API for other id management functions in
future patches.
```cpp
// Old
Mesh *mesh = static_cast<Mesh *>(BKE_id_new_nomain(ID_ME, "Mesh"));
// New
Mesh *mesh = BKE_id_new_nomain<Mesh>("Mesh");
```
Pull Request: https://projects.blender.org/blender/blender/pulls/138383
The goal here is to avoid having to cast to and from `ID` when getting the
evaluated or original ID using the depsgraph API, which is often verbose and not
type safe. To solve this, there are now `DEG_get_original` and
`DEG_get_evaluated` methods which are templated on the type and use a new
`is_ID_v` static type check to make sure it's only used with valid types.
This allows removing quite some verbosity on all the call sites. I also removed
`DEG_get_original_object`, because that does not have to be a special case
anymore.
Pull Request: https://projects.blender.org/blender/blender/pulls/137629
Move `Library.runtime` to be a pointer, move the related
`LibraryRuntime` struct to `BKE_library.hh`. Similar to e.g.
Mesh.runtime, that pointer is expected to always be valid, and is
allocated at readtime or when creating a new Library ID.
Related smaller changes:
* Write code now uses standard ID writing codepath for Library IDs too.
* Runtime pointer is reset to nullptr before writing.
* Looking up a library by its absolute path is now handled through a
dedicated utils, `search_filepath_abs`, instead of using
`BLI_findstring`.
Pull Request: https://projects.blender.org/blender/blender/pulls/134188
When using clangd or running clang-tidy on headers there are
currently many errors. These are noisy in IDEs, make auto fixes
impossible, and break features like code completion, refactoring
and navigation.
This makes source/blender headers work by themselves, which is
generally the goal anyway. But #includes and forward declarations
were often incomplete.
* Add #includes and forward declarations
* Add IWYU pragma: export in a few places
* Remove some unused #includes (but there are many more)
* Tweak ShaderCreateInfo macros to work better with clangd
Some types of headers still have errors, these could be fixed or
worked around with more investigation. Mostly preprocessor
template headers like NOD_static_types.h.
Note that that disabling WITH_UNITY_BUILD is required for clangd to
work properly, otherwise compile_commands.json does not contain
the information for the relevant source files.
For more details see the developer docs:
https://developer.blender.org/docs/handbook/tooling/clangd/
Pull Request: https://projects.blender.org/blender/blender/pulls/132608
We often have the situation where it would be good if we could easily estimate
the memory usage of some value (e.g. a mesh, or volume). Examples of where we
ran into this in the past:
* Undo step size.
* Caching of volume grids.
* Caching of loaded geometries for import geometry nodes.
Generally, most caching systems would benefit from the ability to know how much
memory they currently use to make better decisions about which data to free and
when. The goal of this patch is to introduce a simple general API to count the
memory usage that is independent of any specific caching system. I'm doing this
to "fix" the chicken and egg problem that caches need to know the memory usage,
but we don't really need to count the memory usage without using it for caches.
Implementing caching and memory counting at the same time make both harder than
implementing them one after another.
The main difficulty with counting memory usage is that some memory may be shared
using implicit sharing. We want to avoid double counting such memory. How
exactly shared memory is treated depends a bit on the use case, so no specific
assumptions are made about that in the API. The gathered memory usage is not
expected to be exact. It's expected to be a decent approximation. It's neither a
lower nor an upper bound unless specified by some specific type. Cache systems
generally build on top of heuristics to decide when to free what anyway.
There are two sides to this API:
1. Get the amount of memory used by one or more values. This side is used by
caching systems and/or systems that want to present the used memory to the
user.
2. Tell the caller how much memory is used. This side is used by all kinds of
types that can report their memory usage such as meshes.
```cpp
/* Get how much memory is used by two meshes together. */
MemoryCounter memory;
mesh_a->count_memory(memory);
mesh_b->count_memory(memory);
int64_t bytes_used = memory.counted_bytes();
/* Tell the caller how much memory is used. */
void Mesh::count_memory(blender::MemoryCounter &memory) const
{
memory.add_shared(this->runtime->face_offsets_sharing_info,
this->face_offsets().size_in_bytes());
/* Forward memory counting to lower level types. This should be fairly common. */
CustomData_count_memory(this->vert_data, this->verts_num, memory);
}
void CustomData_count_memory(const CustomData &data,
const int totelem,
blender::MemoryCounter &memory)
{
for (const CustomDataLayer &layer : Span{data.layers, data.totlayer}) {
memory.add_shared(layer.sharing_info, [&](blender::MemoryCounter &shared_memory) {
/* Not quite correct for all types, but this is only a rough approximation anyway. */
const int64_t elem_size = CustomData_get_elem_size(&layer);
shared_memory.add(totelem * elem_size);
});
}
}
```
Pull Request: https://projects.blender.org/blender/blender/pulls/126295
Previously, values for `ID.flag` and `ID.tag` used the prefixes `LIB_` and
`LIB_TAG` respectively. This was somewhat confusing because it's not really
related to libraries in general. This patch changes the prefix to `ID_FLAG_` and
`ID_TAG_`. This makes it more obvious what they correspond to, simplifying code.
Pull Request: https://projects.blender.org/blender/blender/pulls/125811
Seems to work OK in basic cases, but needs more work when copying
outside of Main at least.
Note: There is no behavioral changes expected from this commit.
Note that there are at least two known usecases for this change:
* Liboverrides, as with recursive resync and proxies conversion it
often ends up creating 'virtual' linked data that does not actually
exists in the library blend files.
* Complex versionning code (`do_versions_after_setup`) when it needs
to create new IDs (currently handling linked data that way is just not
supported!).
Implements #107847.
The depsgraph CoW mechanism is a bit of a misnomer. It creates an
evaluated copy for data-blocks regardless of whether the copy will
actually be written to. The point is to have physical separation between
original and evaluated data. This is in contrast to the commonly used
performance improvement of keeping a user count and copying data
implicitly when it needs to be changed. In Blender code we call this
"implicit sharing" instead. Importantly, the dependency graph has no
idea about the _actual_ CoW behavior in Blender.
Renaming this functionality in the despgraph removes some of the
confusion that comes up when talking about this, and will hopefully
make the depsgraph less confusing to understand initially too. Wording
like "the evaluated copy" (as opposed to the original data-block) has
also become common anyway.
Pull Request: https://projects.blender.org/blender/blender/pulls/118338
This data was 'hidden' away in a util in
`lib_query.cc`, which made it hard to discover and keep up-to-date.
However, as shown by e.g. #108407, critical low-level features in ID
management code, such as remapping, now rely on this information being
valid.
Also simplify `BKE_library_id_can_use_filter_id` and
`BKE_library_id_can_use_idtype` to make them more generic, relying on
IDTypeInfo to retrieve IDtype-specific info.
No behavioral changes expected here.
This adds support for baking the volume component of a geometry. Previously,
volumes were just removed in the simulation and bake node.
On disk, each volume geometry is written to a separate `.vdb` file that is stored in
the bakes `blobs` directory and referenced from the corresponding meta `.json` file.
Technically, it would also be easy to write the volume data to the same `.blob`
files that we also write e.g. mesh attributes to. However, since `.vdb` is a well
known file format, it seems reasonable to just store it as a separate file. The
serialization code doesn't really care whether it's a separate file or embedded into
a bigger file, so this decision could be made at a higher level.
Just like with other geometry types, materials are preserved. Just note that when
using the written stand-alone .vdb files, materials are not preserved.
Currently, volume grids are not deduplicated on disk. This could be added in the
future if necessary.
Pull Request: https://projects.blender.org/blender/blender/pulls/117781
`UUID` generally stands for "universally unique identifier". The session identifier that
we use is neither universally unique, nor does it follow the standard. Therefor, the term
"session uuid" is confusing and should be replaced.
In #116888 we briefly talked about a better name and ended up with "session uid".
The reason for "uid" instead of "id" is that the latter is a very overloaded term in Blender
already.
This patch changes all uses of "uuid" to "uid" where it's used in the context of a
"session uid". It's not always trivial to see whether a specific mention of "uuid" refers
to an actual uuid or something else. Therefore, I might have missed some renames.
I can't think of an automated way to differentiate the case.
BMesh also uses the term "uuid" sometimes in a the wrong context (e.g. `UUIDFaceStepItem`)
but there it also does not mean "session uid", so it's *not* changed by this patch.
Pull Request: https://projects.blender.org/blender/blender/pulls/117350
The name `offset_in_ID` is very misleading since it does not have to be an offset at all,
it just has to be some unique value. Often it's also a hash or a constant value depending
on the use.
Pull Request: https://projects.blender.org/blender/blender/pulls/117306
There was one functional issue with the previous API which was its
use in `VolumeGrid<T>::grid_for_write(tree_token)`. The issue was
that the tree token had to be received before the grid is accessed.
However, this `grid_for_write` method might create a copy of the
`VolumeGridData` internally and if it does, the passed in `tree_token`
corresponds to the wrong tree.
The solution is to output the token as part of the method. This has two
additional benefits:
* The API is more safe, because one can't pass an r-value into the methods
anymore. This generally shouldn't be done, because the token should
live at least as long as the OpenVDB tree is used and shouldn't be freed
immediatly.
* The API is a bit simpler, because it's not necessary to call the
`VolumeGrid.tree_access_token()` method anymore.
The same is done for other geometry types. This allows us to use C++ types in
the run-time data more easily and avoids dumping runtime data into .blend files.
Pull Request: https://projects.blender.org/blender/blender/pulls/116840
Along with the 4.1 libraries upgrade, we are bumping the clang-format
version from 8-12 to 17. This affects quite a few files.
If not already the case, you may consider pointing your IDE to the
clang-format binary bundled with the Blender precompiled libraries.
This refactors how volume grids are stored with the following new goals in mind:
* Get a **stand-alone volume grid** data structure that can be used by geometry nodes.
Previously, the `VolumeGrid` data structure was tightly coupled with the `Volume` data block.
* Support **implicit sharing of grids and trees**. Previously, it was possible to share data
when multiple `Volume` data blocks loaded grids from the same `.vdb` files but this was
not flexible enough.
* Get a safe API for **lazy-loading and unloading** of grids without requiring explicit calls
to some "load" function all the time.
* Get a safe API for **caching grids from files** that is not coupled to the `Volume` data block.
* Get a **tiered API** for different levels of `openvdb` involvement:
* No `OpenVDB`: Since `WITH_OPENVDB` is optional, it's helpful to have parts of the API that
still work in this case. This makes it possible to write high level code for volumes that does
not require `#ifdef WITH_OPENVDB` checks everywhere. This is in `BKE_volume_grid_fwd.hh`.
* Shallow `OpenVDB`: Code using this API requires `WITH_OPENVDB` checks. However, care
is taken to not include the expensive parts of `OpenVDB` and to use forward declarations as
much as possible. This is in `BKE_volume_grid.hh` and uses `openvdb_fwd.hh`.
* "Full" `OpenVDB`: This API requires more heavy `OpenVDB` includes. Fortunately, it turned
out to be not necessary for the common API. So this is only used for task specific APIs.
At the core of the new API is the `VolumeGridData` type. It's a wrapper around an
`openvdb::Grid` and adds some features on top like implicit sharing, lazy-loading and unloading.
Then there are `GVolumeGrid` and `VolumeGrid` which are containers for a volume grid.
Semantically, each `VolumeGrid` has its own independent grid, but this is cheap due to implicit
sharing. At highest level we currently have the `Volume` data-block which contains a list of
`VolumeGrid`.
```mermaid
flowchart LR
Volume --> VolumeGrid --> VolumeGridData --> openvdb::Grid
```
The loading of `.vdb` files is abstracted away behind the volume file cache API. This API makes
it easy to load and reuse entire files and individual grids from disk. It also supports caching
simplify levels for grids on disk.
An important new concept are the "tree access tokens". Whenever some code wants to work
with an openvdb tree, it has to retrieve an access token from the corresponding `VolumeGridData`.
This access token has to be kept alive for as long as the code works with the grid data. The same
token is valid for read and write access. The purpose of these access tokens is to make it possible
to detect when some code is currently working with the openvdb tree. This allows freeing it if it's
possible to reload it later on (e.g. from disk). It's possible to free a tree that is referenced by
multiple owners, but only no one is actively working with. In some sense, this is similar to the
existing `ImageUser` concept.
The most important new files to read are `BKE_volume_grid.hh` and `BKE_volume_grid_file_cache.hh`.
Most other changes are updates to existing code to use the new API.
Pull Request: https://projects.blender.org/blender/blender/pulls/116315
Implement the next phases of bounds improvement design #96968.
Mainly the following changes:
Don't use `Object.runtime.bb` for performance caching volume bounds.
This is redundant with the cache in most geometry data-block types.
Instead, this becomes `Object.runtime.bounds_eval`, and is only used
where it's actually needed: syncing the bounds from the evaluated
geometry in the active depsgraph to the original object.
Remove all redundant functions to access geometry bounds with an
Object argument. These make the whole design confusing, since they
access geometry bounds at an object level.
Use `std::optional<Bounds<float3>>` to pass and store bounds instead
of an allocated `BoundBox` struct. This uses less space, avoids
small heap allocations, and generally simplifies code, since we
usually only want the min and max anyway.
After this, to avoid performance regressions, we should also cache
bounds in volumes, and maybe the legacy curve and GP data types
(though it might not be worth the effort for those legacy types).
Pull Request: https://projects.blender.org/blender/blender/pulls/114933
Move object runtime data to a separate header and allocate it separately
as `blender::bke::ObjectRuntime`. This is how node, mesh, curves, and
point cloud runtime data is stored.
Benefits:
- Allow using C++ types in object runtime data
- Reduce space required for Object struct in files
- Increase conceptual separation between DNA and runtime data
- Remove the need to add manual padding in runtime data
- Include runtime struct definition only in files that require it
Pull Request: https://projects.blender.org/blender/blender/pulls/113957
