And same for the `copy_layout` function. These functions do not free any
potentially existing data in destination, but expect it to be uninitialized.
Hopefully these new names will make it more clear and avoid bugs like #122160.
We often have the situation where it would be good if we could easily estimate
the memory usage of some value (e.g. a mesh, or volume). Examples of where we
ran into this in the past:
* Undo step size.
* Caching of volume grids.
* Caching of loaded geometries for import geometry nodes.
Generally, most caching systems would benefit from the ability to know how much
memory they currently use to make better decisions about which data to free and
when. The goal of this patch is to introduce a simple general API to count the
memory usage that is independent of any specific caching system. I'm doing this
to "fix" the chicken and egg problem that caches need to know the memory usage,
but we don't really need to count the memory usage without using it for caches.
Implementing caching and memory counting at the same time make both harder than
implementing them one after another.
The main difficulty with counting memory usage is that some memory may be shared
using implicit sharing. We want to avoid double counting such memory. How
exactly shared memory is treated depends a bit on the use case, so no specific
assumptions are made about that in the API. The gathered memory usage is not
expected to be exact. It's expected to be a decent approximation. It's neither a
lower nor an upper bound unless specified by some specific type. Cache systems
generally build on top of heuristics to decide when to free what anyway.
There are two sides to this API:
1. Get the amount of memory used by one or more values. This side is used by
caching systems and/or systems that want to present the used memory to the
user.
2. Tell the caller how much memory is used. This side is used by all kinds of
types that can report their memory usage such as meshes.
```cpp
/* Get how much memory is used by two meshes together. */
MemoryCounter memory;
mesh_a->count_memory(memory);
mesh_b->count_memory(memory);
int64_t bytes_used = memory.counted_bytes();
/* Tell the caller how much memory is used. */
void Mesh::count_memory(blender::MemoryCounter &memory) const
{
memory.add_shared(this->runtime->face_offsets_sharing_info,
this->face_offsets().size_in_bytes());
/* Forward memory counting to lower level types. This should be fairly common. */
CustomData_count_memory(this->vert_data, this->verts_num, memory);
}
void CustomData_count_memory(const CustomData &data,
const int totelem,
blender::MemoryCounter &memory)
{
for (const CustomDataLayer &layer : Span{data.layers, data.totlayer}) {
memory.add_shared(layer.sharing_info, [&](blender::MemoryCounter &shared_memory) {
/* Not quite correct for all types, but this is only a rough approximation anyway. */
const int64_t elem_size = CustomData_get_elem_size(&layer);
shared_memory.add(totelem * elem_size);
});
}
}
```
Pull Request: https://projects.blender.org/blender/blender/pulls/126295
Previously, values for `ID.flag` and `ID.tag` used the prefixes `LIB_` and
`LIB_TAG` respectively. This was somewhat confusing because it's not really
related to libraries in general. This patch changes the prefix to `ID_FLAG_` and
`ID_TAG_`. This makes it more obvious what they correspond to, simplifying code.
Pull Request: https://projects.blender.org/blender/blender/pulls/125811
The change to use generic "capture field on geometry" utilities for this
node and other nodes like it means `AttributeWriter` with its update
tagging isn't being used anymore, the attribute is just being created
with the new values (for some cases anyway). To fix this, call the
attribute provider's update function when creating the attribute too.
This was noted as useful in 130701763b too.
The initialization of curve and point cloud runtime structs is moved
because they now have to be allocated before any attributes are added.
Seems to work OK in basic cases, but needs more work when copying
outside of Main at least.
Note: There is no behavioral changes expected from this commit.
Note that there are at least two known usecases for this change:
* Liboverrides, as with recursive resync and proxies conversion it
often ends up creating 'virtual' linked data that does not actually
exists in the library blend files.
* Complex versionning code (`do_versions_after_setup`) when it needs
to create new IDs (currently handling linked data that way is just not
supported!).
Implements #107847.
This data was 'hidden' away in a util in
`lib_query.cc`, which made it hard to discover and keep up-to-date.
However, as shown by e.g. #108407, critical low-level features in ID
management code, such as remapping, now rely on this information being
valid.
Also simplify `BKE_library_id_can_use_filter_id` and
`BKE_library_id_can_use_idtype` to make them more generic, relying on
IDTypeInfo to retrieve IDtype-specific info.
No behavioral changes expected here.
With this patch, materials are kept intact in simulation zones and bake nodes
without any additional user action.
This implements the design proposed in #108410 to support referencing
data-blocks (only materials for now) in the baked data. The task also describes
why this is not a trivial issue. A previous attempt was implemented in #109703
but it didn't work well-enough.
The solution is to have an explicit `name (+ library name) -> data-block`
mapping that is stored in the modifier for each bake node and simulation zone.
The `library name` is necessary for it to be unique within a .blend file. Note
that this refers to the name of the `Library` data-block and not a file path.
The baked data only contains the names of the used data-blocks. When the baked
data is loaded, the correct material data-block is looked up from the mapping.
### Automatic Mapping Generation
The most tricky aspect of this approach is to make it feel mostly automatic.
From the user point-of-view, it should just work. Therefore, we don't want the
user to have to create the mapping manually in the majority of cases. Creating
the mapping automatically is difficult because the data-blocks that should
become part of the mapping are only known during depsgraph evaluation. So we
somehow have to gather the missing data blocks during evaluation and then write
the new mappings back to the original data.
While writing back to original data is something we do in some cases already,
the situation here is different, because we are actually creating new relations
between data-blocks. This also means that we'll have to do user-counting. Since
user counts in data-blocks are *not* atomic, we can't do that from multiple
threads at the same time. Also, under some circumstances, it may be necessary to
trigger depsgraph evaluation again after the write-back because it actually
affects the result.
To solve this, a small new API is added in `DEG_depsgraph_writeback_sync.hh`. It
allows gathering tasks which write back to original data in a synchronous way
which may also require a reevaluation.
### Accessing the Mapping
A new `BakeDataBlockMap` is passed to geometry nodes evaluation by the modifier.
This map allows getting the `ID` pointer that should be used for a specific
data-block name that is stored in baked data. It's also used to gather all the
missing data mappings during evaluation.
### Weak ID References
The baked/cached geometries may have references to other data-blocks (currently
only materials, but in the future also e.g. instanced objects/collections).
However, the pointers of these data-blocks are not stable over time. That is
especially true when storing/loading the data from disk, but also just when
playing back the animation. Therefore, the used data-blocks have to referenced
in a different way at run-time.
This is solved by adding `std::unique_ptr<bake::BakeMaterialsList>` to the
run-time data of various geometry data-blocks. If the data-block is cached over
a longer period of time (such that material pointers can't be used directly), it
stores the material name (+ library name) used by each material slot. When the
geometry is used again, the material pointers are restored using these weak name
references and the `BakeDataBlockMap`.
### Manual Mapping Management
There is a new `Data-Blocks` panel in the bake settings in the node editor
sidebar that allows inspecting and modifying the data-blocks that are used when
baking. The user can change what data-block a specific name is mapped to.
Pull Request: https://projects.blender.org/blender/blender/pulls/117043
Some common headers were including this. Separating the includes
will ideally lead to better conceptual separation between CustomData
and the attribute API too. Mostly the change is adding the file to
places where it was included indirectly before. But some code is
shuffled around to hopefully better places as well.
Each value is now out of the global namespace, so they can be shorter
and easier to read. Most of this commit just adds the necessary casting
and namespace specification. `enum class` can be forward declared since
it has a specified size. We will make use of that in the next commit.
Implement the next phases of bounds improvement design #96968.
Mainly the following changes:
Don't use `Object.runtime.bb` for performance caching volume bounds.
This is redundant with the cache in most geometry data-block types.
Instead, this becomes `Object.runtime.bounds_eval`, and is only used
where it's actually needed: syncing the bounds from the evaluated
geometry in the active depsgraph to the original object.
Remove all redundant functions to access geometry bounds with an
Object argument. These make the whole design confusing, since they
access geometry bounds at an object level.
Use `std::optional<Bounds<float3>>` to pass and store bounds instead
of an allocated `BoundBox` struct. This uses less space, avoids
small heap allocations, and generally simplifies code, since we
usually only want the min and max anyway.
After this, to avoid performance regressions, we should also cache
bounds in volumes, and maybe the legacy curve and GP data types
(though it might not be worth the effort for those legacy types).
Pull Request: https://projects.blender.org/blender/blender/pulls/114933
Move object runtime data to a separate header and allocate it separately
as `blender::bke::ObjectRuntime`. This is how node, mesh, curves, and
point cloud runtime data is stored.
Benefits:
- Allow using C++ types in object runtime data
- Reduce space required for Object struct in files
- Increase conceptual separation between DNA and runtime data
- Remove the need to add manual padding in runtime data
- Include runtime struct definition only in files that require it
Pull Request: https://projects.blender.org/blender/blender/pulls/113957
Translation markers `N_()` were added to IDs' plural names in !113912.
I also renamed some of those because I thought there were only display
names.
This commit reverts the renaming, leaving only the addition of the
markers.
Pull Request: https://projects.blender.org/blender/blender/pulls/114030
Extract:
- Sculpt filter types from the Sculpt menu. Some of these types use a
custom label, different from those defined in the operator RNA,
which was never extracted.
- "Today" and "Yesterday" from the file browser modification date.
- All name_plural from IDs, as these are used in the UI to list which
data block is to be removed, when calling outliner.orphans_purge.
Disambiguate:
- "Area", meaning the measurement of a surface as opposed to a place.
Some messages reported by Satoshi Yamasaki in #43295.
Pull Request: https://projects.blender.org/blender/blender/pulls/113912
Currently object bounds (`object.runtime.bb`) are lazily initialized
when accessed. This access happens from arbitrary threads, and
is unprotected by a mutex. This can cause access to stale data at
best, and crashes at worst. Eager calculation is meant to keep this
working, but it's fragile.
Since e8f4010611, geometry bounds are cached in the geometry
itself, which makes this object-level cache redundant. So, it's clearer
to build the `BoundBox` from those cached bounds and return it by
value, without interacting with the object's cached bounding box.
The code change is is mostly a move from `const BoundBox *` to
`std::optional<BoundBox>`. This is only one step of a larger change
described in #96968. Followup steps would include switching to
a simpler and smaller `Bounds` type, removing redundant object-
level access, and eventually removing `object.runtime.bb`.
Access of bounds from the object for mesh, curves, and point cloud
objects should now be thread-safe. Other object types still lazily
initialize the object `BoundBox` cache since they don't have
a data-level cache.
Pull Request: https://projects.blender.org/blender/blender/pulls/113465
The `lib_link` callback cannot always be fully replaced/removed, as in
some case it is also doing some validation checks, or data editing based
on the result of lib_linking internal ID pointers.
The callback has been renamed for that purpose, from `read_lib` to
`read_after_liblink`. It is now called after all ID pointers have been
fully lib-linked for the current ID, but still before the call to
`do_versions_after_linking`.
This change should not have any behavioral effect. Although in theory
the side-effect of this commit (to split lib linking itself, and the
validation/further processing code) into two completely separated steps
could have some effects, in practice none are expected, and tests did
not show any changes in behavior either..
Part of implementing #105134: Removal of readfile's lib_link & expand code.
The `expand` callback is 'trivial' to replace, since it is only iterating
over ID pointers and calling a callback.
The only change in behavior here is that some pointers that were not
processed previously will now be.
In practice this is not expected to have any real effect (usually
the IDs used by these pointers would have been expanded through other
usages anyway). But it may solve a few corner cases, undocumented issues
though.
Part of implementing #105134: Removal of readfile's lib_link & expand code.
Listing the "Blender Foundation" as copyright holder implied the Blender
Foundation holds copyright to files which may include work from many
developers.
While keeping copyright on headers makes sense for isolated libraries,
Blender's own code may be refactored or moved between files in a way
that makes the per file copyright holders less meaningful.
Copyright references to the "Blender Foundation" have been replaced with
"Blender Authors", with the exception of `./extern/` since these this
contains libraries which are more isolated, any changed to license
headers there can be handled on a case-by-case basis.
Some directories in `./intern/` have also been excluded:
- `./intern/cycles/` it's own `AUTHORS` file is planned.
- `./intern/opensubdiv/`.
An "AUTHORS" file has been added, using the chromium projects authors
file as a template.
Design task: #110784
Ref !110783.
There is no reason at all for each ID read/write callbacks to have to
deal with this generic, common data explicitely. Generic ID read/write
code is the place to handle such data (just like asset, liboverride,
etc. data are handled already).
Note behavioral change expected here.
Remove the "_for_read" suffix from methods to get geometry and geometry
components. That should be considered the default, so the suffix just
adds unnecessary text. This is consistent with the attribute API and
various implicit sharing data access methods.
Use "from_mesh" instead of "create_with_mesh". This is consistent with
the recently used naming for the `IndexMask` API.
Pull Request: https://projects.blender.org/blender/blender/pulls/110738
More consistently return geometry bounds with the `Bounds` type that
holds the min and max in one variable. This simplifies some code and
reduces the need to initialize separate min and max variables first.
Meshes now use the same `bounds_min_max()` function as curves and
point clouds, though the wrapper mesh isn't affected yet.
The motivation is to make some of the changes for #96968 simpler.
Move `GeometrySet` and `GeometryComponent` and subclasses
to the `blender::bke` namespace. This wasn't done earlier since
these were one of the first C++ classes used throughout Blender,
but now it is common.
Also remove the now-unnecessary C-header, since all users of
the geometry set header are now in C++.
Pull Request: https://projects.blender.org/blender/blender/pulls/109020
A lot of files were missing copyright field in the header and
the Blender Foundation contributed to them in a sense of bug
fixing and general maintenance.
This change makes it explicit that those files are at least
partially copyrighted by the Blender Foundation.
Note that this does not make it so the Blender Foundation is
the only holder of the copyright in those files, and developers
who do not have a signed contract with the foundation still
hold the copyright as well.
Another aspect of this change is using SPDX format for the
header. We already used it for the license specification,
and now we state it for the copyright as well, following the
FAQ:
https://reuse.software/faq/
The usage of the lib pointer was cryptic to say the least, it was
essentialy used to allow or not local IDs results in the mapping
old_id_pointer -> new_id_pointer lookup result.
Now:
- Explicitely pass a boolean to allow or not local ID as result in
lower-level code.
- Pass the 'self ID' pointer instead of its library to the whole
liblinking code (that was already the case in a few places).
Note that naming of the 'self id' pointer is currently very inconsistent
throughout the readfile liblink code, this will have to be cleaned up in
a separate step later. For now, `self_id` has been chosen for new code
as it matches the terminology in lib_query code.
The later change can also allow finer handling of lookup on undo, based
on how it was re-read (or not), should the need for this arise.
Implicit sharing means attribute ownership is shared between geometry
data-blocks, and the sharing happens automatically. So it's unnecessary
to choose whether to enable it when copying a mesh.
Add the ability to retrieve implicit sharing info directly from the
C++ attribute API, which simplifies memory usage and performance
optimizations making use of it. This commit uses the additions to
the API to avoid copies in a few places:
- The "rest_position" attribute in the mesh modifier stack
- Instance on Points node
- Instances to points node
- Mesh to points node
- Points to vertices node
Many files are affected because in order to include the new information
in the API's returned data, I had to switch a bunch of types from
`VArray` to `AttributeReader`. This generally makes sense anyway, since
it allows retrieving the domain, which wasn't possible before in some
cases. I overloaded the `*` deference operator for some syntactic sugar
to avoid the (very ugly) `.varray` that would be necessary otherwise.
Pull Request: https://projects.blender.org/blender/blender/pulls/107059
