* Move colorband and theme DNA to own headers
* Move some anim, curve, modifier and space enums to new headers
* Move data transfer enums to DNA
* Duplicate imbuf proxy and GPU backend enums
For a few reasons:
* Reduce number of includes in DNA headers
* Don't define enums used in DNA outside of DNA
* Move theme settings to separate header for userdef_default_theme.c
* Prepare for using default initializers in DNA headers. (#134531)
Pull Request: https://projects.blender.org/blender/blender/pulls/138831
As described in #122398, implement read and write support for a new
attribute storage system. Currently this is only implemented to support
forward compatibility; the format used at runtime isn't changed at all.
That can be done one step at a time during the 4.5 and 5.0 development
cycles. A new experimental option for testing tells Blender to always
save with the new format.
The main benefit of the new structure is that it matches the attribute
system design, it allows for future attribute storage optimization, and
each attribute is an allocated struct, which will give pointer stability
for the Python API.
The next step is to connect the attribute API and the RNA API to
AttributeStorage for the simplest geometry type, point clouds.
Pull Request: https://projects.blender.org/blender/blender/pulls/133874
It's safer to pass a type so that it can be checked if delete should be
used instead. Also changes a few void pointer casts to const_cast so that
if the data becomes typed it's an error.
Pull Request: https://projects.blender.org/blender/blender/pulls/137404
The arrays for CustomData layers aren't necessarily allocated with the guarded
allocator. Their ownership is managed by the implicit sharing system which is
required for CustomData layer data.
The main issue of 'type-less' standard C allocations is that there is no check on
allocated type possible.
This is a serious source of annoyance (and crashes) when making some
low-level structs non-trivial, as tracking down all usages of these
structs in higher-level other structs and their allocation is... really
painful.
MEM_[cm]allocN<T> templates on the other hand do check that the
given type is trivial, at build time (static assert), which makes such issue...
trivial to catch.
NOTE: New code should strive to use MEM_new (i.e. allocation and
construction) as much as possible, even for trivial PoD types.
Pull Request: https://projects.blender.org/blender/blender/pulls/136134
The general idea is to keep the 'old', C-style MEM_callocN signature, and slowly
replace most of its usages with the new, C++-style type-safer template version.
* `MEM_cnew<T>` allocation version is renamed to `MEM_callocN<T>`.
* `MEM_cnew_array<T>` allocation version is renamed to `MEM_calloc_arrayN<T>`.
* `MEM_cnew<T>` duplicate version is renamed to `MEM_dupallocN<T>`.
Similar templates type-safe version of `MEM_mallocN` will be added soon
as well.
Following discussions in !134452.
NOTE: For now static type checking in `MEM_callocN` and related are slightly
different for Windows MSVC. This compiler seems to consider structs using the
`DNA_DEFINE_CXX_METHODS` macro as non-trivial (likely because their default
copy constructors are deleted). So using checks on trivially
constructible/destructible instead on this compiler/system.
Pull Request: https://projects.blender.org/blender/blender/pulls/134771
Previously we generally expected CustomData layers to have implicit
sharing info, but we didn't require it. This PR clarifies that we do
require layers with non-null data to have implicit sharing info. This
generally makes code simpler because we don't have to have a separate
code path for non-shared layers. For example, it makes the "totelem"
arguments for layer freeing functions unnecessary, since shared data
knows how to free itself. Those arguments are removed in this PR.
Pull Request: https://projects.blender.org/blender/blender/pulls/134578
Move `Library.runtime` to be a pointer, move the related
`LibraryRuntime` struct to `BKE_library.hh`. Similar to e.g.
Mesh.runtime, that pointer is expected to always be valid, and is
allocated at readtime or when creating a new Library ID.
Related smaller changes:
* Write code now uses standard ID writing codepath for Library IDs too.
* Runtime pointer is reset to nullptr before writing.
* Looking up a library by its absolute path is now handled through a
dedicated utils, `search_filepath_abs`, instead of using
`BLI_findstring`.
Pull Request: https://projects.blender.org/blender/blender/pulls/134188
Previously, there was a `StringRef.copy` method which would copy the string into
the given buffer. However, it was not defined for the case when the buffer was
too small. It moved the responsibility of making sure the buffer is large enough
to the caller.
Unfortunately, in practice that easily hides bugs in builds without asserts
which don't come up in testing much. Now, the method is replaced with
`StringRef.copy_utf8_truncated` which has much more well defined semantics and
also makes sure that the string remains valid utf-8.
This also renames `unsafe_copy` to `copy_unsafe` to make the naming more similar
to `copy_utf8_truncated`.
Pull Request: https://projects.blender.org/blender/blender/pulls/133677
Move `CD_CUSTOMLOOPNORMAL` to the newly added
`CD_PROP_INT16_2D` generic attribute type. This is similar to
previous commits moving specific custom data types.
The attribute name is `custom_normal`. When the attribute with
that name is on the face corner domain, the code will interpret it
as stored in the existing deformation-invariant spherical coordinate
space.
The API remains the same, with the additional opportunity to edit
custom normal data as an attribute directly (which admittedly is fairly
unintuitive currently).
See #130484.
Pull Request: https://projects.blender.org/blender/blender/pulls/130689
Add a `short2` attribute type, intended to store the existing
`CD_CUSTOMLOOPNORMAL` custom data type in order to move the custom
normal storage to generic attributes. This type probably won't get much
use besides that, but generally we don't have reasons not to add these
generic types, besides binary size. In the future we should consolidate
usage of `convert_to_static_type` to avoid large binary size increases
(about 700KB here) for this sort of addition.
This is the first step of #130484.
Pull Request: https://projects.blender.org/blender/blender/pulls/130530
This removes `AnonymousAttributeID` which was "attached" to every anonymous
attribute before. It adds more complexity than is justified for its
functionality.
It was originally introduced to keep the reference count of the anonymous
attribute so that it can be deleted automatically when the attribute is not
referenced anymore. For quite some time we have had deterministic attribute
life-times though which don't rely on the reference count anymore.
Anonymous attributes are sometimes shown in the UI as "friendly looking" string
like `"UV Map" from Cube`. Some information necessary for this was also stored
in `AnonymousAttributeID`. However, this can also be solved differently.
Specifically, this functionality has now been added directly to
`AttributeFieldInput`.
This refactor also allows removing `AttributeIDRef` which was mainly introduced
because we had to keep the `AnonymousAttributeID` attached with the attribute
name. Just using simple string types to identify attributes can reduce the
mental overhead quite significantly. This will be done as a separate refactor
though.
Pull Request: https://projects.blender.org/blender/blender/pulls/127081
And same for the `copy_layout` function. These functions do not free any
potentially existing data in destination, but expect it to be uninitialized.
Hopefully these new names will make it more clear and avoid bugs like #122160.
We often have the situation where it would be good if we could easily estimate
the memory usage of some value (e.g. a mesh, or volume). Examples of where we
ran into this in the past:
* Undo step size.
* Caching of volume grids.
* Caching of loaded geometries for import geometry nodes.
Generally, most caching systems would benefit from the ability to know how much
memory they currently use to make better decisions about which data to free and
when. The goal of this patch is to introduce a simple general API to count the
memory usage that is independent of any specific caching system. I'm doing this
to "fix" the chicken and egg problem that caches need to know the memory usage,
but we don't really need to count the memory usage without using it for caches.
Implementing caching and memory counting at the same time make both harder than
implementing them one after another.
The main difficulty with counting memory usage is that some memory may be shared
using implicit sharing. We want to avoid double counting such memory. How
exactly shared memory is treated depends a bit on the use case, so no specific
assumptions are made about that in the API. The gathered memory usage is not
expected to be exact. It's expected to be a decent approximation. It's neither a
lower nor an upper bound unless specified by some specific type. Cache systems
generally build on top of heuristics to decide when to free what anyway.
There are two sides to this API:
1. Get the amount of memory used by one or more values. This side is used by
caching systems and/or systems that want to present the used memory to the
user.
2. Tell the caller how much memory is used. This side is used by all kinds of
types that can report their memory usage such as meshes.
```cpp
/* Get how much memory is used by two meshes together. */
MemoryCounter memory;
mesh_a->count_memory(memory);
mesh_b->count_memory(memory);
int64_t bytes_used = memory.counted_bytes();
/* Tell the caller how much memory is used. */
void Mesh::count_memory(blender::MemoryCounter &memory) const
{
memory.add_shared(this->runtime->face_offsets_sharing_info,
this->face_offsets().size_in_bytes());
/* Forward memory counting to lower level types. This should be fairly common. */
CustomData_count_memory(this->vert_data, this->verts_num, memory);
}
void CustomData_count_memory(const CustomData &data,
const int totelem,
blender::MemoryCounter &memory)
{
for (const CustomDataLayer &layer : Span{data.layers, data.totlayer}) {
memory.add_shared(layer.sharing_info, [&](blender::MemoryCounter &shared_memory) {
/* Not quite correct for all types, but this is only a rough approximation anyway. */
const int64_t elem_size = CustomData_get_elem_size(&layer);
shared_memory.add(totelem * elem_size);
});
}
}
```
Pull Request: https://projects.blender.org/blender/blender/pulls/126295
This fixes#121695. `float4x4` matrices are generally expected to be 16 byte aligned.
Currently, there is no mechanism (afaik) that allows allocating these overaligned types
when loading files from disk. This patch adds an array with alignment information for
each type in `SDNA`. Currently, the alignment is just `__STDCPP_DEFAULT_NEW_ALIGNMENT__`
for all types and is manually set for the `mat4x4f` DNA type. The .blend file format is
not changed at all. The alignment information is purely runtime data.
In the future it would probably be good to generalize this a bit more instead of
hardcoding the alignment for `mat4x4f`, but would make it unnecessarily complex for
now because this is intended for the release branch.
Pull Request: https://projects.blender.org/blender/blender/pulls/123271
This fixes a bug that happens when two drawings in the same grease pencil
data-block reference the same array. In this case, the same attribute array
pointer is written to .blend files for both drawings. However, when loading the
.blend file, it was not detected that the data is shared. Instead each drawing
would think that it was the single owner of the array even though it was shared.
The same problem can technically occur for all kinds of shared data, and not
just for grease pencil. However, only grease pencil shows this issue currently,
because it can easily happen that the same attribute is shared between different
`CustomData` within the same data-block. This very rarely happens with meshes or
curves, because different `CustomData` usually have different sizes and don't
share the same arrays.
A previous fix (cc891aa699) solved pretty much the same issues for the case
when two layers in the same `CustomData` share data. This original fix is now
generalized and is integrated with `BLO_read_shared` which makes it less error
prone. For each data-block, the `BlendDataReader` now remembers which shared
data it has loaded before so that it is not loaded again.
In the future this could be extended to support data that is shared between
data-blocks, but that's not as straight forward as one would hope currently. For
various reasons, different data-blocks could store independent data at the same
pointer. I specifically noticed that with a regression test file:
`blender/tests/data/modeling/geometry_nodes/attributes/attribute_statistics.blend`.
To support it, one solution could be to tag `BHead` in .blend files to indicate
that a specific pointer is shared and unique within the entire file. But that's
for another day.
Pull Request: https://projects.blender.org/blender/blender/pulls/122780
This makes the read and write API functions match more closely, and adds
asserts to check that the data size is as expected.
There are still a few places remaining that use BLO_read_data_address
and similar generic functions, these should eventually be replaced as well.
Pull Request: https://projects.blender.org/blender/blender/pulls/120994
For some reason, the custom data type was set as `float[4][4]`
for tangents, but they are written as `float[4]`. The overallocation
might not have been a problem, since it looks like the extra memory
was never written to. But it's still confusing and potentially bad for
performance.
It's possible that multiple custom data layers share the same underlying data.
This data is only written once (per data-block) currently. However, when reading
the data again, two separate `ImplicitSharingInfo` were constructed which
referenced the same underlying memory. The fix is to reuse previously created
`ImplicitSharingInfo` that manage the same data.
Pull Request: https://projects.blender.org/blender/blender/pulls/118991
This adds implicit sharing support for the `MemFile` undo-step. This decreases memory
usage and increases performance.
Implicit sharing allows the undo system to take (shared) ownership of some data.
Previously, the data would always be serialized and compared to the previous undo-step.
So this turns an O(n) operation into O(1) (in terms of memory usage and time).
Read/write code that wants to make use of this has to use the new `BLO_read_shared`
and `BLO_write_shared` functions respectively. Those either make use of implicit-sharing
internally or do the "full" read/write based on a passed-in function. It seems possible to
use the same API in the future to store shared data to .blend files.
Improvements:
* Much faster undo step creation in many cases by avoiding the majority data copies
and equality checks. This fixes#98574. I found undo step creation and undo step
decoding to be 2-5 times faster in some demo files from the blender website and in
some production files from the Heist project.
* Reduced memory usage when there is large data in `bmain`. For example, when
loading the same highly subdivided mesh that I used in #106228 the memory usage
is 1.03 GB now (compared to 1.62 GB in `main` currently). The main remaining copy
of the data now is done by rendering code.
* Some significant performance improvements were also measured for the new grease
pencil type (#105540).
There is one main downside of using implicit-sharing as implemented here: `MemFile`
undo steps can't be written as .blend files anymore. This has a few consequences:
* Auto-save becomes slower (up to 3x), because it can't just save the previous undo step
anymore and does a normal save instead. This has been discussed in more detail here:
https://devtalk.blender.org/t/remove-support-for-saving-memfile-undo-steps-as-blend-files-proposal/33544
It would be nice to work towards asynchronous auto-save to alleviate this problem.
Some previous work has been done to reduce the impact of this change in 41b10424c7
and f0f304e240. This has been committed separately in efb511a76d.
* Writing `quit.blend` has to do a normal file save now. So it's a bit slower too, but it's
less of a problem in practice.
* The `USE_WRITE_CRASH_BLEND` functionality does not work anymore. It doesn't seem
to be used by anyone (removed in e90f5d03c4)
There are also benefits to not writing `MemFile` from undo steps to disk. It allows us to
more safely do undo-specific optimizations without risking corrupted .blend files. This
is especially useful when we want to preserve forward compatibility in some cases.
This requires converting data before writing the .blend files, but this conversion is not
necessary for undo steps. Trying to implement this kind of optimization in the past has
often lead to bugs (e.g. 43b37fbc93).
Another new problem is that it is harder to know the size of each undo step. Currently, a
heuristic is used to approximate the memory usage, but better solutions could be found
if necessary.
Pull Request: https://projects.blender.org/blender/blender/pulls/106903
Due to a recent change the name of a layer is passed using a string ref
but the check if it is filled was done by checking if the first char
was zero. This fails as string ref first does the bound checks.
This is fixed by using `!is_empty()`. There might be more places, but this
one fired when loading production files.
Issue introduced by a39e8a4ab9
Pull Request: https://projects.blender.org/blender/blender/pulls/118605
This simple node finds the values of a volume grid at
positions in the local space used in geometry nodes
evaluation. There are three interpolation modes to
choose how to mix values between neighboring voxels.
For the implementation, first the values are sampled
with the grid's type directly, then implicit type conversions
are used to get the final type. This makes gives us flexibility
in case there aren't exact matches in support between grid
types and Blender types.
Pull Request: https://projects.blender.org/blender/blender/pulls/118397
Implements the design from #116067.
The socket type is called "Matrix" but it is often referred to as "Transform"
when that's what it is semantically. The attribute type is "4x4 Matrix" since
that's a lower level choice. Currently matrix sockets are always passed
around internally as `float4x4`, but that can be optimized in the future
when smaller types would give the same behavior.
A new "Matrix" utilities category has the following set of initial nodes"
- **Combine Transform**
- **Separate Transform**
- **Multiply Matrices**
- **Transform Direction**
- **Transform Vector**
- **Invert Matrix**
- **Transpose Matrix**
The nodes and socket type are behind an experimental flag for now,
which will give us time to make sure it's the right set of initial nodes.
The viewer node overlay doesn't support matrices-- they aren't supported
for rendering in general. They also aren't supported in the modifier interface
currently. But they are supported in the spreadsheet, where the value is
displayed in a tooltip.
Pull Request: https://projects.blender.org/blender/blender/pulls/116166
