Move `Library.runtime` to be a pointer, move the related
`LibraryRuntime` struct to `BKE_library.hh`. Similar to e.g.
Mesh.runtime, that pointer is expected to always be valid, and is
allocated at readtime or when creating a new Library ID.
Related smaller changes:
* Write code now uses standard ID writing codepath for Library IDs too.
* Runtime pointer is reset to nullptr before writing.
* Looking up a library by its absolute path is now handled through a
dedicated utils, `search_filepath_abs`, instead of using
`BLI_findstring`.
Pull Request: https://projects.blender.org/blender/blender/pulls/134188
Previously, there was a `StringRef.copy` method which would copy the string into
the given buffer. However, it was not defined for the case when the buffer was
too small. It moved the responsibility of making sure the buffer is large enough
to the caller.
Unfortunately, in practice that easily hides bugs in builds without asserts
which don't come up in testing much. Now, the method is replaced with
`StringRef.copy_utf8_truncated` which has much more well defined semantics and
also makes sure that the string remains valid utf-8.
This also renames `unsafe_copy` to `copy_unsafe` to make the naming more similar
to `copy_utf8_truncated`.
Pull Request: https://projects.blender.org/blender/blender/pulls/133677
Move `CD_CUSTOMLOOPNORMAL` to the newly added
`CD_PROP_INT16_2D` generic attribute type. This is similar to
previous commits moving specific custom data types.
The attribute name is `custom_normal`. When the attribute with
that name is on the face corner domain, the code will interpret it
as stored in the existing deformation-invariant spherical coordinate
space.
The API remains the same, with the additional opportunity to edit
custom normal data as an attribute directly (which admittedly is fairly
unintuitive currently).
See #130484.
Pull Request: https://projects.blender.org/blender/blender/pulls/130689
Add a `short2` attribute type, intended to store the existing
`CD_CUSTOMLOOPNORMAL` custom data type in order to move the custom
normal storage to generic attributes. This type probably won't get much
use besides that, but generally we don't have reasons not to add these
generic types, besides binary size. In the future we should consolidate
usage of `convert_to_static_type` to avoid large binary size increases
(about 700KB here) for this sort of addition.
This is the first step of #130484.
Pull Request: https://projects.blender.org/blender/blender/pulls/130530
This removes `AnonymousAttributeID` which was "attached" to every anonymous
attribute before. It adds more complexity than is justified for its
functionality.
It was originally introduced to keep the reference count of the anonymous
attribute so that it can be deleted automatically when the attribute is not
referenced anymore. For quite some time we have had deterministic attribute
life-times though which don't rely on the reference count anymore.
Anonymous attributes are sometimes shown in the UI as "friendly looking" string
like `"UV Map" from Cube`. Some information necessary for this was also stored
in `AnonymousAttributeID`. However, this can also be solved differently.
Specifically, this functionality has now been added directly to
`AttributeFieldInput`.
This refactor also allows removing `AttributeIDRef` which was mainly introduced
because we had to keep the `AnonymousAttributeID` attached with the attribute
name. Just using simple string types to identify attributes can reduce the
mental overhead quite significantly. This will be done as a separate refactor
though.
Pull Request: https://projects.blender.org/blender/blender/pulls/127081
And same for the `copy_layout` function. These functions do not free any
potentially existing data in destination, but expect it to be uninitialized.
Hopefully these new names will make it more clear and avoid bugs like #122160.
We often have the situation where it would be good if we could easily estimate
the memory usage of some value (e.g. a mesh, or volume). Examples of where we
ran into this in the past:
* Undo step size.
* Caching of volume grids.
* Caching of loaded geometries for import geometry nodes.
Generally, most caching systems would benefit from the ability to know how much
memory they currently use to make better decisions about which data to free and
when. The goal of this patch is to introduce a simple general API to count the
memory usage that is independent of any specific caching system. I'm doing this
to "fix" the chicken and egg problem that caches need to know the memory usage,
but we don't really need to count the memory usage without using it for caches.
Implementing caching and memory counting at the same time make both harder than
implementing them one after another.
The main difficulty with counting memory usage is that some memory may be shared
using implicit sharing. We want to avoid double counting such memory. How
exactly shared memory is treated depends a bit on the use case, so no specific
assumptions are made about that in the API. The gathered memory usage is not
expected to be exact. It's expected to be a decent approximation. It's neither a
lower nor an upper bound unless specified by some specific type. Cache systems
generally build on top of heuristics to decide when to free what anyway.
There are two sides to this API:
1. Get the amount of memory used by one or more values. This side is used by
caching systems and/or systems that want to present the used memory to the
user.
2. Tell the caller how much memory is used. This side is used by all kinds of
types that can report their memory usage such as meshes.
```cpp
/* Get how much memory is used by two meshes together. */
MemoryCounter memory;
mesh_a->count_memory(memory);
mesh_b->count_memory(memory);
int64_t bytes_used = memory.counted_bytes();
/* Tell the caller how much memory is used. */
void Mesh::count_memory(blender::MemoryCounter &memory) const
{
memory.add_shared(this->runtime->face_offsets_sharing_info,
this->face_offsets().size_in_bytes());
/* Forward memory counting to lower level types. This should be fairly common. */
CustomData_count_memory(this->vert_data, this->verts_num, memory);
}
void CustomData_count_memory(const CustomData &data,
const int totelem,
blender::MemoryCounter &memory)
{
for (const CustomDataLayer &layer : Span{data.layers, data.totlayer}) {
memory.add_shared(layer.sharing_info, [&](blender::MemoryCounter &shared_memory) {
/* Not quite correct for all types, but this is only a rough approximation anyway. */
const int64_t elem_size = CustomData_get_elem_size(&layer);
shared_memory.add(totelem * elem_size);
});
}
}
```
Pull Request: https://projects.blender.org/blender/blender/pulls/126295
This fixes#121695. `float4x4` matrices are generally expected to be 16 byte aligned.
Currently, there is no mechanism (afaik) that allows allocating these overaligned types
when loading files from disk. This patch adds an array with alignment information for
each type in `SDNA`. Currently, the alignment is just `__STDCPP_DEFAULT_NEW_ALIGNMENT__`
for all types and is manually set for the `mat4x4f` DNA type. The .blend file format is
not changed at all. The alignment information is purely runtime data.
In the future it would probably be good to generalize this a bit more instead of
hardcoding the alignment for `mat4x4f`, but would make it unnecessarily complex for
now because this is intended for the release branch.
Pull Request: https://projects.blender.org/blender/blender/pulls/123271
This fixes a bug that happens when two drawings in the same grease pencil
data-block reference the same array. In this case, the same attribute array
pointer is written to .blend files for both drawings. However, when loading the
.blend file, it was not detected that the data is shared. Instead each drawing
would think that it was the single owner of the array even though it was shared.
The same problem can technically occur for all kinds of shared data, and not
just for grease pencil. However, only grease pencil shows this issue currently,
because it can easily happen that the same attribute is shared between different
`CustomData` within the same data-block. This very rarely happens with meshes or
curves, because different `CustomData` usually have different sizes and don't
share the same arrays.
A previous fix (cc891aa699) solved pretty much the same issues for the case
when two layers in the same `CustomData` share data. This original fix is now
generalized and is integrated with `BLO_read_shared` which makes it less error
prone. For each data-block, the `BlendDataReader` now remembers which shared
data it has loaded before so that it is not loaded again.
In the future this could be extended to support data that is shared between
data-blocks, but that's not as straight forward as one would hope currently. For
various reasons, different data-blocks could store independent data at the same
pointer. I specifically noticed that with a regression test file:
`blender/tests/data/modeling/geometry_nodes/attributes/attribute_statistics.blend`.
To support it, one solution could be to tag `BHead` in .blend files to indicate
that a specific pointer is shared and unique within the entire file. But that's
for another day.
Pull Request: https://projects.blender.org/blender/blender/pulls/122780
This makes the read and write API functions match more closely, and adds
asserts to check that the data size is as expected.
There are still a few places remaining that use BLO_read_data_address
and similar generic functions, these should eventually be replaced as well.
Pull Request: https://projects.blender.org/blender/blender/pulls/120994
For some reason, the custom data type was set as `float[4][4]`
for tangents, but they are written as `float[4]`. The overallocation
might not have been a problem, since it looks like the extra memory
was never written to. But it's still confusing and potentially bad for
performance.
It's possible that multiple custom data layers share the same underlying data.
This data is only written once (per data-block) currently. However, when reading
the data again, two separate `ImplicitSharingInfo` were constructed which
referenced the same underlying memory. The fix is to reuse previously created
`ImplicitSharingInfo` that manage the same data.
Pull Request: https://projects.blender.org/blender/blender/pulls/118991
This adds implicit sharing support for the `MemFile` undo-step. This decreases memory
usage and increases performance.
Implicit sharing allows the undo system to take (shared) ownership of some data.
Previously, the data would always be serialized and compared to the previous undo-step.
So this turns an O(n) operation into O(1) (in terms of memory usage and time).
Read/write code that wants to make use of this has to use the new `BLO_read_shared`
and `BLO_write_shared` functions respectively. Those either make use of implicit-sharing
internally or do the "full" read/write based on a passed-in function. It seems possible to
use the same API in the future to store shared data to .blend files.
Improvements:
* Much faster undo step creation in many cases by avoiding the majority data copies
and equality checks. This fixes#98574. I found undo step creation and undo step
decoding to be 2-5 times faster in some demo files from the blender website and in
some production files from the Heist project.
* Reduced memory usage when there is large data in `bmain`. For example, when
loading the same highly subdivided mesh that I used in #106228 the memory usage
is 1.03 GB now (compared to 1.62 GB in `main` currently). The main remaining copy
of the data now is done by rendering code.
* Some significant performance improvements were also measured for the new grease
pencil type (#105540).
There is one main downside of using implicit-sharing as implemented here: `MemFile`
undo steps can't be written as .blend files anymore. This has a few consequences:
* Auto-save becomes slower (up to 3x), because it can't just save the previous undo step
anymore and does a normal save instead. This has been discussed in more detail here:
https://devtalk.blender.org/t/remove-support-for-saving-memfile-undo-steps-as-blend-files-proposal/33544
It would be nice to work towards asynchronous auto-save to alleviate this problem.
Some previous work has been done to reduce the impact of this change in 41b10424c7
and f0f304e240. This has been committed separately in efb511a76d.
* Writing `quit.blend` has to do a normal file save now. So it's a bit slower too, but it's
less of a problem in practice.
* The `USE_WRITE_CRASH_BLEND` functionality does not work anymore. It doesn't seem
to be used by anyone (removed in e90f5d03c4)
There are also benefits to not writing `MemFile` from undo steps to disk. It allows us to
more safely do undo-specific optimizations without risking corrupted .blend files. This
is especially useful when we want to preserve forward compatibility in some cases.
This requires converting data before writing the .blend files, but this conversion is not
necessary for undo steps. Trying to implement this kind of optimization in the past has
often lead to bugs (e.g. 43b37fbc93).
Another new problem is that it is harder to know the size of each undo step. Currently, a
heuristic is used to approximate the memory usage, but better solutions could be found
if necessary.
Pull Request: https://projects.blender.org/blender/blender/pulls/106903
Due to a recent change the name of a layer is passed using a string ref
but the check if it is filled was done by checking if the first char
was zero. This fails as string ref first does the bound checks.
This is fixed by using `!is_empty()`. There might be more places, but this
one fired when loading production files.
Issue introduced by a39e8a4ab9
Pull Request: https://projects.blender.org/blender/blender/pulls/118605
This simple node finds the values of a volume grid at
positions in the local space used in geometry nodes
evaluation. There are three interpolation modes to
choose how to mix values between neighboring voxels.
For the implementation, first the values are sampled
with the grid's type directly, then implicit type conversions
are used to get the final type. This makes gives us flexibility
in case there aren't exact matches in support between grid
types and Blender types.
Pull Request: https://projects.blender.org/blender/blender/pulls/118397
Implements the design from #116067.
The socket type is called "Matrix" but it is often referred to as "Transform"
when that's what it is semantically. The attribute type is "4x4 Matrix" since
that's a lower level choice. Currently matrix sockets are always passed
around internally as `float4x4`, but that can be optimized in the future
when smaller types would give the same behavior.
A new "Matrix" utilities category has the following set of initial nodes"
- **Combine Transform**
- **Separate Transform**
- **Multiply Matrices**
- **Transform Direction**
- **Transform Vector**
- **Invert Matrix**
- **Transpose Matrix**
The nodes and socket type are behind an experimental flag for now,
which will give us time to make sure it's the right set of initial nodes.
The viewer node overlay doesn't support matrices-- they aren't supported
for rendering in general. They also aren't supported in the modifier interface
currently. But they are supported in the spreadsheet, where the value is
displayed in a tooltip.
Pull Request: https://projects.blender.org/blender/blender/pulls/116166
Along with the 4.1 libraries upgrade, we are bumping the clang-format
version from 8-12 to 17. This affects quite a few files.
If not already the case, you may consider pointing your IDE to the
clang-format binary bundled with the Blender precompiled libraries.
Some common headers were including this. Separating the includes
will ideally lead to better conceptual separation between CustomData
and the attribute API too. Mostly the change is adding the file to
places where it was included indirectly before. But some code is
shuffled around to hopefully better places as well.
This is the initial implementation for the volume grid sockets that has been
discussed during the November 2023 geometry nodes workshop.
It adds initial support for passing volume grids around in sockets. Furthermore,
it adds two new nodes. Both are initially hidden under the "New Volume Nodes"
experimental option until we have a few mode nodes.
* **Get Named Grid**: Gets or extracts a volume grid from a volume geometry
based on the grid's name.
* **Store Named Grid**: Puts a volume grid back into a volume with a name.
`SocketValueVariant` is extended to support grids besides single values and fields.
Next steps:
* Implement grid socket shape and inferencing (currently, they just look like
single values).
* Add implicit conversions between grid types.
* Implement nodes that operate on the grids (#116021).
* Improved spreadsheet and viewer support.
Links:
* https://devtalk.blender.org/t/volumes-in-geometry-nodes-proposal/31917
* https://devtalk.blender.org/t/2023-11-06-geometry-nodes-workshop-notes/32007#volumes-3
Co-authored-by: Jacques Lucke <jacques@blender.org>
Pull Request: https://projects.blender.org/blender/blender/pulls/115270
Fixes#115776
Create a minimal structure that contains the instructions necessary to
copy from one custom data format to another. This structure is similar
to the one used in dfacaf4f40. It should have been used in
9175d9b7c2, which instead introduced quadratic performance
relative to the number of layers when copying every element.
In this commit, copying the entire mesh and adding new custom data
are explicitly changed to use the new map to speed up copying many
elements at a time.
The non-map attribute copy functions are also changed to check for when
the source and result BMeshes are the same. In that case it's much
faster to call the "same format" function from 9175d9b7c2.
For numbers, the timings are arbitrarily influenced by how many layers
I add in my testing. With 50 or so layers, a 10x difference is easily
observable though.
Pull Request: https://projects.blender.org/blender/blender/pulls/115824
When the BMesh source and result arguments are the same, restore
performance lost by 9175d9b7c2, which made copying layers
have quadratic time complexity. When we know the custom data format
is the same between the source and result, the copying can be much
simpler, so it's worth specializing this case. There is still more
to be done, because often we only know that the two meshes are the
same at runtime. A followup commit will add that check.
The quadratic runtime means performance is fine for low layer counts,
and terrible with higher layer counts. For example, in my testing with
47 boolean attributes, copying 250k vertices went from 2.3 seconds to
316 ms.
The implementation uses a new CustomData function that copies an entire
BMesh custom data block, called by a function in the BMesh module
overloaded for every BMesh element type. That handles the extra data
like flags, normals, and material indices.
Related to #115776
For context, see 6d09fa3577. Overall, these values were still
written in some cases, but never used. Nowadays the viewer node and
attribute overlays give even better answers to these needs.
Store paint masks as generic float attributes, with the name
`".sculpt_mask"`. This is similar to 060a534141, which made
the same change for face sets. The benefits are general
consistency, nicer code, and more support in newer areas
that deal with attributes like geometry nodes.
The RNA API is replaced with one created in Python. The new
API only presents a single layer as an attribute class, so it
should be simpler to use in general:
- Before: `object.data.vertex_paint_masks[0].data[0].value`
- After: `object.data.vertex_paint_mask.data[0].value`
Pull Request: https://projects.blender.org/blender/blender/pulls/115119
