This fixes a bug that happens when two drawings in the same grease pencil
data-block reference the same array. In this case, the same attribute array
pointer is written to .blend files for both drawings. However, when loading the
.blend file, it was not detected that the data is shared. Instead each drawing
would think that it was the single owner of the array even though it was shared.
The same problem can technically occur for all kinds of shared data, and not
just for grease pencil. However, only grease pencil shows this issue currently,
because it can easily happen that the same attribute is shared between different
`CustomData` within the same data-block. This very rarely happens with meshes or
curves, because different `CustomData` usually have different sizes and don't
share the same arrays.
A previous fix (cc891aa699) solved pretty much the same issues for the case
when two layers in the same `CustomData` share data. This original fix is now
generalized and is integrated with `BLO_read_shared` which makes it less error
prone. For each data-block, the `BlendDataReader` now remembers which shared
data it has loaded before so that it is not loaded again.
In the future this could be extended to support data that is shared between
data-blocks, but that's not as straight forward as one would hope currently. For
various reasons, different data-blocks could store independent data at the same
pointer. I specifically noticed that with a regression test file:
`blender/tests/data/modeling/geometry_nodes/attributes/attribute_statistics.blend`.
To support it, one solution could be to tag `BHead` in .blend files to indicate
that a specific pointer is shared and unique within the entire file. But that's
for another day.
Pull Request: https://projects.blender.org/blender/blender/pulls/122780
This makes the read and write API functions match more closely, and adds
asserts to check that the data size is as expected.
There are still a few places remaining that use BLO_read_data_address
and similar generic functions, these should eventually be replaced as well.
Pull Request: https://projects.blender.org/blender/blender/pulls/120994
For some reason, the custom data type was set as `float[4][4]`
for tangents, but they are written as `float[4]`. The overallocation
might not have been a problem, since it looks like the extra memory
was never written to. But it's still confusing and potentially bad for
performance.
It's possible that multiple custom data layers share the same underlying data.
This data is only written once (per data-block) currently. However, when reading
the data again, two separate `ImplicitSharingInfo` were constructed which
referenced the same underlying memory. The fix is to reuse previously created
`ImplicitSharingInfo` that manage the same data.
Pull Request: https://projects.blender.org/blender/blender/pulls/118991
This adds implicit sharing support for the `MemFile` undo-step. This decreases memory
usage and increases performance.
Implicit sharing allows the undo system to take (shared) ownership of some data.
Previously, the data would always be serialized and compared to the previous undo-step.
So this turns an O(n) operation into O(1) (in terms of memory usage and time).
Read/write code that wants to make use of this has to use the new `BLO_read_shared`
and `BLO_write_shared` functions respectively. Those either make use of implicit-sharing
internally or do the "full" read/write based on a passed-in function. It seems possible to
use the same API in the future to store shared data to .blend files.
Improvements:
* Much faster undo step creation in many cases by avoiding the majority data copies
and equality checks. This fixes#98574. I found undo step creation and undo step
decoding to be 2-5 times faster in some demo files from the blender website and in
some production files from the Heist project.
* Reduced memory usage when there is large data in `bmain`. For example, when
loading the same highly subdivided mesh that I used in #106228 the memory usage
is 1.03 GB now (compared to 1.62 GB in `main` currently). The main remaining copy
of the data now is done by rendering code.
* Some significant performance improvements were also measured for the new grease
pencil type (#105540).
There is one main downside of using implicit-sharing as implemented here: `MemFile`
undo steps can't be written as .blend files anymore. This has a few consequences:
* Auto-save becomes slower (up to 3x), because it can't just save the previous undo step
anymore and does a normal save instead. This has been discussed in more detail here:
https://devtalk.blender.org/t/remove-support-for-saving-memfile-undo-steps-as-blend-files-proposal/33544
It would be nice to work towards asynchronous auto-save to alleviate this problem.
Some previous work has been done to reduce the impact of this change in 41b10424c7
and f0f304e240. This has been committed separately in efb511a76d.
* Writing `quit.blend` has to do a normal file save now. So it's a bit slower too, but it's
less of a problem in practice.
* The `USE_WRITE_CRASH_BLEND` functionality does not work anymore. It doesn't seem
to be used by anyone (removed in e90f5d03c4)
There are also benefits to not writing `MemFile` from undo steps to disk. It allows us to
more safely do undo-specific optimizations without risking corrupted .blend files. This
is especially useful when we want to preserve forward compatibility in some cases.
This requires converting data before writing the .blend files, but this conversion is not
necessary for undo steps. Trying to implement this kind of optimization in the past has
often lead to bugs (e.g. 43b37fbc93).
Another new problem is that it is harder to know the size of each undo step. Currently, a
heuristic is used to approximate the memory usage, but better solutions could be found
if necessary.
Pull Request: https://projects.blender.org/blender/blender/pulls/106903
Due to a recent change the name of a layer is passed using a string ref
but the check if it is filled was done by checking if the first char
was zero. This fails as string ref first does the bound checks.
This is fixed by using `!is_empty()`. There might be more places, but this
one fired when loading production files.
Issue introduced by a39e8a4ab9
Pull Request: https://projects.blender.org/blender/blender/pulls/118605
This simple node finds the values of a volume grid at
positions in the local space used in geometry nodes
evaluation. There are three interpolation modes to
choose how to mix values between neighboring voxels.
For the implementation, first the values are sampled
with the grid's type directly, then implicit type conversions
are used to get the final type. This makes gives us flexibility
in case there aren't exact matches in support between grid
types and Blender types.
Pull Request: https://projects.blender.org/blender/blender/pulls/118397
Implements the design from #116067.
The socket type is called "Matrix" but it is often referred to as "Transform"
when that's what it is semantically. The attribute type is "4x4 Matrix" since
that's a lower level choice. Currently matrix sockets are always passed
around internally as `float4x4`, but that can be optimized in the future
when smaller types would give the same behavior.
A new "Matrix" utilities category has the following set of initial nodes"
- **Combine Transform**
- **Separate Transform**
- **Multiply Matrices**
- **Transform Direction**
- **Transform Vector**
- **Invert Matrix**
- **Transpose Matrix**
The nodes and socket type are behind an experimental flag for now,
which will give us time to make sure it's the right set of initial nodes.
The viewer node overlay doesn't support matrices-- they aren't supported
for rendering in general. They also aren't supported in the modifier interface
currently. But they are supported in the spreadsheet, where the value is
displayed in a tooltip.
Pull Request: https://projects.blender.org/blender/blender/pulls/116166
Along with the 4.1 libraries upgrade, we are bumping the clang-format
version from 8-12 to 17. This affects quite a few files.
If not already the case, you may consider pointing your IDE to the
clang-format binary bundled with the Blender precompiled libraries.
Some common headers were including this. Separating the includes
will ideally lead to better conceptual separation between CustomData
and the attribute API too. Mostly the change is adding the file to
places where it was included indirectly before. But some code is
shuffled around to hopefully better places as well.
This is the initial implementation for the volume grid sockets that has been
discussed during the November 2023 geometry nodes workshop.
It adds initial support for passing volume grids around in sockets. Furthermore,
it adds two new nodes. Both are initially hidden under the "New Volume Nodes"
experimental option until we have a few mode nodes.
* **Get Named Grid**: Gets or extracts a volume grid from a volume geometry
based on the grid's name.
* **Store Named Grid**: Puts a volume grid back into a volume with a name.
`SocketValueVariant` is extended to support grids besides single values and fields.
Next steps:
* Implement grid socket shape and inferencing (currently, they just look like
single values).
* Add implicit conversions between grid types.
* Implement nodes that operate on the grids (#116021).
* Improved spreadsheet and viewer support.
Links:
* https://devtalk.blender.org/t/volumes-in-geometry-nodes-proposal/31917
* https://devtalk.blender.org/t/2023-11-06-geometry-nodes-workshop-notes/32007#volumes-3
Co-authored-by: Jacques Lucke <jacques@blender.org>
Pull Request: https://projects.blender.org/blender/blender/pulls/115270
Fixes#115776
Create a minimal structure that contains the instructions necessary to
copy from one custom data format to another. This structure is similar
to the one used in dfacaf4f40. It should have been used in
9175d9b7c2, which instead introduced quadratic performance
relative to the number of layers when copying every element.
In this commit, copying the entire mesh and adding new custom data
are explicitly changed to use the new map to speed up copying many
elements at a time.
The non-map attribute copy functions are also changed to check for when
the source and result BMeshes are the same. In that case it's much
faster to call the "same format" function from 9175d9b7c2.
For numbers, the timings are arbitrarily influenced by how many layers
I add in my testing. With 50 or so layers, a 10x difference is easily
observable though.
Pull Request: https://projects.blender.org/blender/blender/pulls/115824
When the BMesh source and result arguments are the same, restore
performance lost by 9175d9b7c2, which made copying layers
have quadratic time complexity. When we know the custom data format
is the same between the source and result, the copying can be much
simpler, so it's worth specializing this case. There is still more
to be done, because often we only know that the two meshes are the
same at runtime. A followup commit will add that check.
The quadratic runtime means performance is fine for low layer counts,
and terrible with higher layer counts. For example, in my testing with
47 boolean attributes, copying 250k vertices went from 2.3 seconds to
316 ms.
The implementation uses a new CustomData function that copies an entire
BMesh custom data block, called by a function in the BMesh module
overloaded for every BMesh element type. That handles the extra data
like flags, normals, and material indices.
Related to #115776
For context, see 6d09fa3577. Overall, these values were still
written in some cases, but never used. Nowadays the viewer node and
attribute overlays give even better answers to these needs.
Store paint masks as generic float attributes, with the name
`".sculpt_mask"`. This is similar to 060a534141, which made
the same change for face sets. The benefits are general
consistency, nicer code, and more support in newer areas
that deal with attributes like geometry nodes.
The RNA API is replaced with one created in Python. The new
API only presents a single layer as an attribute class, so it
should be simpler to use in general:
- Before: `object.data.vertex_paint_masks[0].data[0].value`
- After: `object.data.vertex_paint_mask.data[0].value`
Pull Request: https://projects.blender.org/blender/blender/pulls/115119
This was the last use of `CustomDataAttributes`. It's not very useful
nowadays because we have a lower level (`CustomData`) and higher level
(`AttributeAccessor`) API.
As part of this I removed the ability to `reserve` instances which was
useful when adding instances one by one. Generally, such code should
eventually be rewritten to handle more instances at once, instead of
handling them one by one. This will lead to better performance and
is more in line with how other geometry types (like mesh) are handled.
This also adds the ability to move and assign `Instances` just like we can
move and assign `CurvesGeometry`.
The issue was that custom data layers were modified without checking
that the data was not shared. Now the data is unshared first.
I also added an assert to catches this kind of issue more directly.
The RNA attribute API did not make sure that attributes are unshared when
the data is accessed. This is necessary to avoid accidentally changing data
on other geometries.
Unfortunately, we currently can't detect if the access is read-only or if the
attribute data is modified. Therefore, the data is always copied when it
was shared. This makes it a little bit less efficient in some cases, but that
should still be ok. The old behavior could by brought back by implementing
#112150.
Pull Request: https://projects.blender.org/blender/blender/pulls/111514
`LayerTypeInfo` can define a `layers_max()` function which determines the
maximum allowed number of layers.
Upon merging, this limit was respected from the source, but not on the
destination, so it was possible to exceed the max (if there were layers
on the destination already).
NOTE: `layers_max()` is currently only defined for legacy CD_MTFACE, but
we might want to enforce this for UVs / CD_PROP_FLOAT2 again.
This came up in #111608.
Pull Request: https://projects.blender.org/blender/blender/pulls/111609
Listing the "Blender Foundation" as copyright holder implied the Blender
Foundation holds copyright to files which may include work from many
developers.
While keeping copyright on headers makes sense for isolated libraries,
Blender's own code may be refactored or moved between files in a way
that makes the per file copyright holders less meaningful.
Copyright references to the "Blender Foundation" have been replaced with
"Blender Authors", with the exception of `./extern/` since these this
contains libraries which are more isolated, any changed to license
headers there can be handled on a case-by-case basis.
Some directories in `./intern/` have also been excluded:
- `./intern/cycles/` it's own `AUTHORS` file is planned.
- `./intern/opensubdiv/`.
An "AUTHORS" file has been added, using the chromium projects authors
file as a template.
Design task: #110784
Ref !110783.
Using ClangBuildAnalyzer on the whole Blender build, it was pointing
out that BLI_math.h is the heaviest "header hub" (i.e. non tiny file
that is included a lot).
However, there's very little (actually zero) source files in Blender
that need "all the math" (base, colors, vectors, matrices,
quaternions, intersection, interpolation, statistics, solvers and
time). A common use case is source files needing just vectors, or
just vectors & matrices, or just colors etc. Actually, 181 files
were including the whole math thing without needing it at all.
This change removes BLI_math.h completely, and instead in all the
places that need it, includes BLI_math_vector.h or BLI_math_color.h
and so on.
Change from that:
- BLI_math_color.h was included 1399 times -> now 408 (took 114.0sec
to parse -> now 36.3sec)
- BLI_simd.h 1403 -> 418 (109.7sec -> 34.9sec).
Full rebuild of Blender (Apple M1, Xcode, RelWithDebInfo) is not
affected much (342sec -> 334sec). Most of benefit would be when
someone's changing BLI_simd.h or BLI_math_color.h or similar files,
that now there's 3x fewer files result in a recompile.
Pull Request #110944
This function is used to retrieve the index of the n'th layer of
the specified type, if it exists. Currently the way it handles
an n value that is too big is inconsistent: if there are more
layers of different types following the ones of the correct type,
and n isn't too big, it will safely detect that the corresponding
layer is of a different type and return -1. However, if the value
results in a buffer overrun, it causes an assertion or crash.
This means that safe code should currently ensure n is always
correct, making the type check in this function pointless.
Moreover, checking the range in the caller may incur more
overhead than making this code safe.
This changes the assert into a check to ensure consistent behavior.
Pull Request: https://projects.blender.org/blender/blender/pulls/110812
Replace `BKE_mesh_face_flip_ex()` with `bke::mesh_flip_faces()`.
This gives better performance since it avoids custom data name
lookups for every face index, and processes data in a more cache
friendly way. Also remove the `CustomData_swap` API function
as mentioned in 09b6eac5c4.
Flipping 16 million faces, I observed a 3.25x speedup, from 760 to 230 ms.
Pull Request: https://projects.blender.org/blender/blender/pulls/110702
The issue was that `CustomData_swap` modifies all layers, but does not ensure
that the layers are actually mutable. This resulted in it modifying shared layers.
Now it ensures that the layers are mutable first (which copies the layers if they
are shared).
Longer term, it would be good to remove `CustomData_swap` since it's not a
good API performance wise, but that is outside of the scope of this patch.
Pull Request: https://projects.blender.org/blender/blender/pulls/110678
