Check if either the memory is zero or already matches the default value,
and copy. This simplifies a common pattern to a single line.
Preparing for default initializers in DNA (#134531).
Pull Request: https://projects.blender.org/blender/blender/pulls/138830
As described in #122398, implement read and write support for a new
attribute storage system. Currently this is only implemented to support
forward compatibility; the format used at runtime isn't changed at all.
That can be done one step at a time during the 4.5 and 5.0 development
cycles. A new experimental option for testing tells Blender to always
save with the new format.
The main benefit of the new structure is that it matches the attribute
system design, it allows for future attribute storage optimization, and
each attribute is an allocated struct, which will give pointer stability
for the Python API.
The next step is to connect the attribute API and the RNA API to
AttributeStorage for the simplest geometry type, point clouds.
Pull Request: https://projects.blender.org/blender/blender/pulls/133874
This adds a version of `BKE_id_new_nomain` that takes the ID type parameter as
template argument. This allows the function the return the newly created ID with
the correct type, removing the need to use `static_cast` on the call-site.
To make this work, I added a static `id_type` member to every ID struct. This
can also be used to create a similar API for other id management functions in
future patches.
```cpp
// Old
Mesh *mesh = static_cast<Mesh *>(BKE_id_new_nomain(ID_ME, "Mesh"));
// New
Mesh *mesh = BKE_id_new_nomain<Mesh>("Mesh");
```
Pull Request: https://projects.blender.org/blender/blender/pulls/138383
The goal here is to avoid having to cast to and from `ID` when getting the
evaluated or original ID using the depsgraph API, which is often verbose and not
type safe. To solve this, there are now `DEG_get_original` and
`DEG_get_evaluated` methods which are templated on the type and use a new
`is_ID_v` static type check to make sure it's only used with valid types.
This allows removing quite some verbosity on all the call sites. I also removed
`DEG_get_original_object`, because that does not have to be a special case
anymore.
Pull Request: https://projects.blender.org/blender/blender/pulls/137629
Currently this code uses the `Mesh` struct as a value embedded in the
undo step struct. This causes the need for various hacky workarounds and
abstraction leaks. It's simpler to just allocate it in the regular way
The Shape Key ID can also be freed with the regular ID free function,
avoiding the need to expose the internal free function.
Pull Request: https://projects.blender.org/blender/blender/pulls/137613
Clearing a mesh's geometry without freeing the mesh has the requirement
that the mesh remain initialized afterwards. Freeing the mesh doesn't
have that requirement and corresponds to a destructor rather than move
construction with the default value. Avoiding conflating the two
operations simplifies some future feature additions (including #122398).
Add a "dumb vector" storage option for custom normals, with the
"custom_normal" attribute. Adjust the mesh normals caching to
provide this attribute if it's available, and add a geometry node to
store custom normals.
## Free Normals
They're called "free" in the sense that they're just direction vectors
in the object's local space, rather than the existing "smooth corner
fan space" storage. They're also "free" in that they make further
normals calculation very inexpensive, since we just use the custom
normals instead. That's a big improvement from the existing custom
normals storage, which usually significantly decreases
viewport performance. For example, in a simple test file just storing
the vertex normals on a UV sphere, using free normals gives 25 times
better playback performance and 10% lower memory usage.
Free normals are adjusted when applying a transformation to the entire
mesh or when realizing instances, but in general they're not updated for
vertex deformations.
## Set Mesh Normal Node
The new geometry node allows storing free custom normals as well as
the existing corner fan space normals. When free normals are chosen,
free normals can be stored on vertices, faces, or face corners. Using
the face corner domain is necessary to bake existing mixed sharp and
smooth edges into the custom normal vectors.
The node also has a mode for storing edge and mesh sharpness, meant
as a "soft" replacement to the "Set Shade Smooth" node that's a bit
more convenient.
## Normal Input Node
The normal node outputs free custom normals mixed to whatever domain is
requested. A "true normal" output that ignores custom normals and
sharpness is added as well.
Across Blender, custom normals are generally accessed via face and
vertex normals, when "true normals" are not requested explicitly.
In many cases that means they are mixed from the face corner domain.
## Future Work
1. There are many places where propagation of free normals could be
improved. They should probably be normalized after mixing, and it
may be useful to not just use 0 vectors for new elements. To keep
the scope of this change smaller, that sort of thing generally isn't
handled here. Searching `CD_NORMAL` gives a hint of where better
propagation could be useful.
2. Free normals are displayed properly in edit mode, but the existing
custom normal editing operators don't work with free normals yet.
This will hopefully be fairly straightforward since custom normals
are usually converted to `float3` for editing anyway. Edit mode
changes aren't included here because they're unnecessary for the
procedural custom normals use cases.
3. Most importers can probably switch to using free normals instead,
or at least provide an option for it. That will give a significant
import performance improvement, and an improvement of Blender's
FPS for imported scenes too.
Pull Request: https://projects.blender.org/blender/blender/pulls/132583
Due to !133498 layers containing incorrect material indices could lead to generating
to many GPU resources crashing Blender. As there is a maximum number of material slots
we can limit it.
This is just a safeguard for incorrect values in material_index attribute layer. Working with
this material index attributes should automatically limit, but I leave that to the modeling
module.
Pull Request: https://projects.blender.org/blender/blender/pulls/136302
The main issue of 'type-less' standard C allocations is that there is no check on
allocated type possible.
This is a serious source of annoyance (and crashes) when making some
low-level structs non-trivial, as tracking down all usages of these
structs in higher-level other structs and their allocation is... really
painful.
MEM_[cm]allocN<T> templates on the other hand do check that the
given type is trivial, at build time (static assert), which makes such issue...
trivial to catch.
NOTE: New code should strive to use MEM_new (i.e. allocation and
construction) as much as possible, even for trivial PoD types.
Pull Request: https://projects.blender.org/blender/blender/pulls/136134
* Fix mesh_copy_data setting the subd wrapper type without copying the
associated tessellated mesh. It's unclear if this currently causes any
issues, but with future changes it would crash.
* Use double checked lock to avoid mutex locks on every call to
BKE_mesh_wrapper_ensure_mdata and BKE_mesh_wrapper_ensure_subdivision.
* Set wrapper type to subd also if subdivision was skipped due to bad
topology or level set to zero. Previous it would do mutex locking and
TBB task isolation on every function call.
* Add assert when BKE_mesh_wrapper_ensure_mdata changes the wrapper type
from subd to mdata. This doesn't really make any sense, it just loses
the subdivision detail. If any code relies on this we need to understand
why and maybe change it.
Pull Request: https://projects.blender.org/blender/blender/pulls/135895
This adds a new `Visual Geometry to Objects` operator which is added to the
`Apply` menu.
It creates new objects from the evaluated geometry of the active object. Other
than e.g. applying modifiers, this operator does keep all generated data and
creates new objects and collections for instances. It does have some
similarities to `Make Instances Real` with the main difference that it doesn't
actually realize instances, i.e. instanced geometry is still shared between all
objects using it.
It does intentionally not replace the original object, because the semantics of
that aren't all that obvious. It's especially tricky when the object was
referenced by other objects. Instance attributes are not preserved currently.
Pull Request: https://projects.blender.org/blender/blender/pulls/134119
Previously we generally expected CustomData layers to have implicit
sharing info, but we didn't require it. This PR clarifies that we do
require layers with non-null data to have implicit sharing info. This
generally makes code simpler because we don't have to have a separate
code path for non-shared layers. For example, it makes the "totelem"
arguments for layer freeing functions unnecessary, since shared data
knows how to free itself. Those arguments are removed in this PR.
Pull Request: https://projects.blender.org/blender/blender/pulls/134578
Replace `const char *` with `StringRef` for the API in `BKE_attribute.h`.
The benefits are slightly simpler code and possibly slightly improved
performance through avoiding the need to measure string length.
Pull Request: https://projects.blender.org/blender/blender/pulls/134183
The issue was that the function that computes the maximum used material index
did not take the edit mesh into account yet. This led the drawing code to
believe that only the first material is used by the mesh.
Pull Request: https://projects.blender.org/blender/blender/pulls/133711
The core issue was that the geometry batch cache (e.g. `MeshBatchCache` or
`PointCloudBatchCache`) was dependent on the object. This is problematic when
the the same geometry is used with multiple different objects because the cache
can't be consistent with all of them.
Fortunately, the only thing that was retrieved from the object was the number of
material slots, so if that can be avoided we should be fine. We can't just use
the number of material slots stored on the geometry because that may have no
material slots but still has material indices which are overridden on the object
level.
The solution is to take make the number of materials for a geometry only
dependent on the actual `material_index` attribute and not on the number of
available slots. More specifically, we find the maximal referenced material
index and handle that many materials. This number does not depend on how many
material slots there are on the object, but it still allows the object to
override materials slots that the mesh references.
A downside is that the maximum material index has to be computed which often
requires an iteration over the mesh. Fortunately, we can cache that quite easily
and the computation can be done in parallel. Also we are probably able to
eagerly update the material index in many cases when it's set instead of
computing it lazily. That is not implemented in this patch though.
The largest part of the patch is making the maximal material index easily
available on all the geometry types. Besides that, the material API is slightly
replaced and the drawing code now makes use of the updated API.
Pull Request: https://projects.blender.org/blender/blender/pulls/133498
Regression in [0] that clamped the shape-key index when exiting
edit-mode but not when entering, causing the "Basis" shape not to
be treated as active when entering edit-mode, then treated as active
when exiting - overwriting it's values from the mesh.
Resolve by clamping on edit-mode enter & exit.
[0]: 225e3460d0
When moving UV seams from special custom data types to attributes,
I considered them similar to mesh selection or visibility which are
"internal" attributes that aren't accessible procedurally and are
hidden from the UI. In retrospect that was the wrong decision; users
expect UV seams to behave more like bevel weights, and that makes sense.
This PR makes UV seams accessible in modifiers (geometry nodes) by
removing the leading period from the attribute name that indicated their
internal status.
The change of the attribute name is a breaking change of the API to some
extent, even though it's technically only mesh data. To mitigate that
issue, the `mesh.attributes["name"]` lookup function is modified to
support both the old and new names. Versioning code renames the
attribute to the new name when loading older files, and renames the
new name to the old name when saving files. That handling will be
removed as a breaking change in 5.0.
Pull Request: https://projects.blender.org/blender/blender/pulls/129803
Move `CD_CUSTOMLOOPNORMAL` to the newly added
`CD_PROP_INT16_2D` generic attribute type. This is similar to
previous commits moving specific custom data types.
The attribute name is `custom_normal`. When the attribute with
that name is on the face corner domain, the code will interpret it
as stored in the existing deformation-invariant spherical coordinate
space.
The API remains the same, with the additional opportunity to edit
custom normal data as an attribute directly (which admittedly is fairly
unintuitive currently).
See #130484.
Pull Request: https://projects.blender.org/blender/blender/pulls/130689
Avoid rebuilding BVH trees when meshes are copied.
Similar to the other uses of the shared cache system,
this can arbitrarily improve performance when meshes
are copied but not deformed and BVH building is the
main bottleneck. In a simple test file I got a 6x speedup.
The amount of code is also reduced and the system is
much simpler overall-- built out of common threading
patterns like `SharedCache` with its double-checked lock.
RAII is used in a few places to simplify memory management
too.
The downside is storing more `SharedCache` items in the
mesh runtime struct. That has a slight cost when copying
a small mesh many times, but we have ideas to improve that
in the future anyway (#104327).
Pull Request: https://projects.blender.org/blender/blender/pulls/130865
Currently replacing the Mesh during evaluation with the object info node
can cause us to use invalid original indices when the source object is
in edit mode. This is really a more fundamental problem though: we have
no way to tell whether an evaluated mesh actually corresponds to the
object's original mesh.
This commit changes to explicitly propagating the edit mesh pointer
through copied and changed meshes during modifier and nodes evaluation,
instead of just blindly copying the edit mode pointer from the original
mesh to the evaluated mesh. A benefit of not writing to the evaluated
mesh means it can be shared, potentially offering a future performance
improvement for uses of the object info node.
When we detect an invalid correspondence between the evaluated/original
meshes, we skip extracting the cage mesh's data and skip extracting edit
mesh data from the evaluated object.
This commit also moves the source of "object is in edit mode" truth in
the draw module from whether the evaluated mesh has an edit mode pointer
to the object's mode flag. That's a simplification that's also helpful
to reduce the strong linking between BMesh and edit mode.
Pull Request: https://projects.blender.org/blender/blender/pulls/120999
Previously, the attribute accessor were defined in the `geometry_component_*.cc`
files. This made sense back in the day, because this attribute API was only used
through `GeometryComponent`. However, nowadays this attribute API is independent
of `GeometryComponent`. E.g. one can use `mesh.attributes()` without ever
creating a component.
The refactor contains the following changes:
* Move attribute accessors to separate files for each geometry type. E.g. from
`geometry_component_mesh.cc` to `mesh_attributes.cc`.
* Move implementations of e.g. `Mesh::attributes()` to `mesh.cc`.
* Provide access to the `AttributeAccessorFunctions` without actually having a
geometry. This will be useful to e.g. implement
`attribute_is_builtin_on_component_type` without dummy components.
Pull Request: https://projects.blender.org/blender/blender/pulls/130516
Previously, the `AttributeIDRef` wrapper was needed because it also had to
contain a pointer to an `AnonymousAttributeID`. However, since
b279a6d703 this is not necessary anymore.
Therefore we can use "raw" `StringRef` now which reduces the mental overhead
when working with attributes and also simplifies code.
Pull Request: https://projects.blender.org/blender/blender/pulls/127140
And same for the `copy_layout` function. These functions do not free any
potentially existing data in destination, but expect it to be uninitialized.
Hopefully these new names will make it more clear and avoid bugs like #122160.
We often have the situation where it would be good if we could easily estimate
the memory usage of some value (e.g. a mesh, or volume). Examples of where we
ran into this in the past:
* Undo step size.
* Caching of volume grids.
* Caching of loaded geometries for import geometry nodes.
Generally, most caching systems would benefit from the ability to know how much
memory they currently use to make better decisions about which data to free and
when. The goal of this patch is to introduce a simple general API to count the
memory usage that is independent of any specific caching system. I'm doing this
to "fix" the chicken and egg problem that caches need to know the memory usage,
but we don't really need to count the memory usage without using it for caches.
Implementing caching and memory counting at the same time make both harder than
implementing them one after another.
The main difficulty with counting memory usage is that some memory may be shared
using implicit sharing. We want to avoid double counting such memory. How
exactly shared memory is treated depends a bit on the use case, so no specific
assumptions are made about that in the API. The gathered memory usage is not
expected to be exact. It's expected to be a decent approximation. It's neither a
lower nor an upper bound unless specified by some specific type. Cache systems
generally build on top of heuristics to decide when to free what anyway.
There are two sides to this API:
1. Get the amount of memory used by one or more values. This side is used by
caching systems and/or systems that want to present the used memory to the
user.
2. Tell the caller how much memory is used. This side is used by all kinds of
types that can report their memory usage such as meshes.
```cpp
/* Get how much memory is used by two meshes together. */
MemoryCounter memory;
mesh_a->count_memory(memory);
mesh_b->count_memory(memory);
int64_t bytes_used = memory.counted_bytes();
/* Tell the caller how much memory is used. */
void Mesh::count_memory(blender::MemoryCounter &memory) const
{
memory.add_shared(this->runtime->face_offsets_sharing_info,
this->face_offsets().size_in_bytes());
/* Forward memory counting to lower level types. This should be fairly common. */
CustomData_count_memory(this->vert_data, this->verts_num, memory);
}
void CustomData_count_memory(const CustomData &data,
const int totelem,
blender::MemoryCounter &memory)
{
for (const CustomDataLayer &layer : Span{data.layers, data.totlayer}) {
memory.add_shared(layer.sharing_info, [&](blender::MemoryCounter &shared_memory) {
/* Not quite correct for all types, but this is only a rough approximation anyway. */
const int64_t elem_size = CustomData_get_elem_size(&layer);
shared_memory.add(totelem * elem_size);
});
}
}
```
Pull Request: https://projects.blender.org/blender/blender/pulls/126295
Previously, values for `ID.flag` and `ID.tag` used the prefixes `LIB_` and
`LIB_TAG` respectively. This was somewhat confusing because it's not really
related to libraries in general. This patch changes the prefix to `ID_FLAG_` and
`ID_TAG_`. This makes it more obvious what they correspond to, simplifying code.
Pull Request: https://projects.blender.org/blender/blender/pulls/125811
All of these changes should be trivial, like using `string` read/write
code for strings, and a convert a few usages of 'raw data' read/write
calls to the 'struct' ones.
No behavioral changes expected here.
Advanced ID copying code can now take a `new_owner_id` ID pointer parameter,
and use it to set the relevant 'loopback' pointer to its owner ID by the
copy code itself.
Besides avoiding the need for all code copying embedded IDs to set the
loopback pointer themselves, this also means that `lib_id` copying code
itself does not need to use `IDWALK_IGNORE_MISSING_OWNER_ID` anymore.
This change is not expected to have any effect in current codebase.
This makes the read and write API functions match more closely, and adds
asserts to check that the data size is as expected.
There are still a few places remaining that use BLO_read_data_address
and similar generic functions, these should eventually be replaced as well.
Pull Request: https://projects.blender.org/blender/blender/pulls/120994
The main motivation for this is that it's part of a fix for #113377,
where I want to propagate the edit mesh pointers through copied
meshes in modifiers and geometry nodes, instead of just setting the
edit mesh pointer at the end of the modifier stack. That would have
two main benefits:
1. We avoid the need to write to the evaluated mesh, after evaluation
which means it can be shared directly among evaluated objects.
2. When an object's mesh is completely replaced by the mesh from another
object during evaluation (with the object info node), the final edit
mesh pointer will not be "wrong", allowing us to skip index-mapped
GPU data extraction.
Beyond that, using a shared pointer just makes things more automatic.
Handling of edit mesh data is already complicated enough, this way some
of the worry and complexity can be handled by RAII.
One thing to keep in mind is that the edit mesh's BMesh is still freed
manually with `EDBM_mesh_free_data` when leaving edit mode. I figured
that was a more conservative approach for now. Maybe eventually that
could be handled automatically with RAII too.
Pull Request: https://projects.blender.org/blender/blender/pulls/120276
After some discussions and investigation over the last couple months,
it's not clear what the "wrapper type finalize" logic is necessary for.
For edit meshes and regular meshes, normals are calculated lazily when
building the draw cache.
Apart from the unnecessary complication for mesh GPU draw data
extraction, this code also causes normals to always be calculated
when turning an edit mesh wrapper into a regular mesh. However, those
normals are immediately discarded since the edit deform cache is deleted
in the next line.
Beyond the obvious simplification, the motivation for this change is to
avoid requesting write access on the evaluated mesh and cage mesh. This
works better with implicit sharing, allowing other improvements.
Pull Request: https://projects.blender.org/blender/blender/pulls/120066
Seems to work OK in basic cases, but needs more work when copying
outside of Main at least.
Note: There is no behavioral changes expected from this commit.
Note that there are at least two known usecases for this change:
* Liboverrides, as with recursive resync and proxies conversion it
often ends up creating 'virtual' linked data that does not actually
exists in the library blend files.
* Complex versionning code (`do_versions_after_setup`) when it needs
to create new IDs (currently handling linked data that way is just not
supported!).
Implements #107847.
This adds implicit sharing support for the `MemFile` undo-step. This decreases memory
usage and increases performance.
Implicit sharing allows the undo system to take (shared) ownership of some data.
Previously, the data would always be serialized and compared to the previous undo-step.
So this turns an O(n) operation into O(1) (in terms of memory usage and time).
Read/write code that wants to make use of this has to use the new `BLO_read_shared`
and `BLO_write_shared` functions respectively. Those either make use of implicit-sharing
internally or do the "full" read/write based on a passed-in function. It seems possible to
use the same API in the future to store shared data to .blend files.
Improvements:
* Much faster undo step creation in many cases by avoiding the majority data copies
and equality checks. This fixes#98574. I found undo step creation and undo step
decoding to be 2-5 times faster in some demo files from the blender website and in
some production files from the Heist project.
* Reduced memory usage when there is large data in `bmain`. For example, when
loading the same highly subdivided mesh that I used in #106228 the memory usage
is 1.03 GB now (compared to 1.62 GB in `main` currently). The main remaining copy
of the data now is done by rendering code.
* Some significant performance improvements were also measured for the new grease
pencil type (#105540).
There is one main downside of using implicit-sharing as implemented here: `MemFile`
undo steps can't be written as .blend files anymore. This has a few consequences:
* Auto-save becomes slower (up to 3x), because it can't just save the previous undo step
anymore and does a normal save instead. This has been discussed in more detail here:
https://devtalk.blender.org/t/remove-support-for-saving-memfile-undo-steps-as-blend-files-proposal/33544
It would be nice to work towards asynchronous auto-save to alleviate this problem.
Some previous work has been done to reduce the impact of this change in 41b10424c7
and f0f304e240. This has been committed separately in efb511a76d.
* Writing `quit.blend` has to do a normal file save now. So it's a bit slower too, but it's
less of a problem in practice.
* The `USE_WRITE_CRASH_BLEND` functionality does not work anymore. It doesn't seem
to be used by anyone (removed in e90f5d03c4)
There are also benefits to not writing `MemFile` from undo steps to disk. It allows us to
more safely do undo-specific optimizations without risking corrupted .blend files. This
is especially useful when we want to preserve forward compatibility in some cases.
This requires converting data before writing the .blend files, but this conversion is not
necessary for undo steps. Trying to implement this kind of optimization in the past has
often lead to bugs (e.g. 43b37fbc93).
Another new problem is that it is harder to know the size of each undo step. Currently, a
heuristic is used to approximate the memory usage, but better solutions could be found
if necessary.
Pull Request: https://projects.blender.org/blender/blender/pulls/106903
