Remove the addition of the position attribute from the default
"init data" callback where we don't know the desired number
of points. Add it in the other functions that add the data-block
(except the version that purposefully doesn't add attributes).
Pull Request: https://projects.blender.org/blender/blender/pulls/138697
This adds a version of `BKE_id_new_nomain` that takes the ID type parameter as
template argument. This allows the function the return the newly created ID with
the correct type, removing the need to use `static_cast` on the call-site.
To make this work, I added a static `id_type` member to every ID struct. This
can also be used to create a similar API for other id management functions in
future patches.
```cpp
// Old
Mesh *mesh = static_cast<Mesh *>(BKE_id_new_nomain(ID_ME, "Mesh"));
// New
Mesh *mesh = BKE_id_new_nomain<Mesh>("Mesh");
```
Pull Request: https://projects.blender.org/blender/blender/pulls/138383
This makes accessing these properties more convenient. Since we only ever have
const references to `CPPType`, there isn't really a benefit to using methods to
avoid mutation.
Pull Request: https://projects.blender.org/blender/blender/pulls/137482
This implements bundles and closures which are described in more detail in this
blog post: https://code.blender.org/2024/11/geometry-nodes-workshop-october-2024/
tl;dr:
* Bundles are containers that allow storing multiple socket values in a single
value. Each value in the bundle is identified by a name. Bundles can be
nested.
* Closures are functions that are created with the Closure Zone and can be
evaluated with the Evaluate Closure node.
To use the patch, the `Bundle and Closure Nodes` experimental feature has to be
enabled. This is necessary, because these features are not fully done yet and
still need iterations to improve the workflow before they can be officially
released. These iterations are easier to do in `main` than in a separate branch
though. That's because this patch is quite large and somewhat prone to merge
conflicts. Also other work we want to do, depends on this.
This adds the following new nodes:
* Combine Bundle: can pack multiple values into one.
* Separate Bundle: extracts values from a bundle.
* Closure Zone: outputs a closure zone for use in the `Evaluate Closure` node.
* Evaluate Closure: evaluates the passed in closure.
Things that will be added soon after this lands:
* Fields in bundles and closures. The way this is done changes with #134811, so
I rather implement this once both are in `main`.
* UI features for keeping sockets in sync (right now there are warnings only).
One bigger issue is the limited support for lazyness. For example, all inputs of
a Combine Bundle node will be evaluated, even if they are not all needed. The
same is true for all captured values of a closure. This is a deeper limitation
that needs to be resolved at some point. This will likely be done after an
initial version of this patch is done.
Pull Request: https://projects.blender.org/blender/blender/pulls/128340
The general idea is to keep the 'old', C-style MEM_callocN signature, and slowly
replace most of its usages with the new, C++-style type-safer template version.
* `MEM_cnew<T>` allocation version is renamed to `MEM_callocN<T>`.
* `MEM_cnew_array<T>` allocation version is renamed to `MEM_calloc_arrayN<T>`.
* `MEM_cnew<T>` duplicate version is renamed to `MEM_dupallocN<T>`.
Similar templates type-safe version of `MEM_mallocN` will be added soon
as well.
Following discussions in !134452.
NOTE: For now static type checking in `MEM_callocN` and related are slightly
different for Windows MSVC. This compiler seems to consider structs using the
`DNA_DEFINE_CXX_METHODS` macro as non-trivial (likely because their default
copy constructors are deleted). So using checks on trivially
constructible/destructible instead on this compiler/system.
Pull Request: https://projects.blender.org/blender/blender/pulls/134771
Previously we generally expected CustomData layers to have implicit
sharing info, but we didn't require it. This PR clarifies that we do
require layers with non-null data to have implicit sharing info. This
generally makes code simpler because we don't have to have a separate
code path for non-shared layers. For example, it makes the "totelem"
arguments for layer freeing functions unnecessary, since shared data
knows how to free itself. Those arguments are removed in this PR.
Pull Request: https://projects.blender.org/blender/blender/pulls/134578
Vertex groups are written as generic attributes in bakes currently.
In order to restore them as vertex groups properly when reading bakes,
keep track of the vertex group names in the bake metadata.
Pull Request: https://projects.blender.org/blender/blender/pulls/131393
Add a `short2` attribute type, intended to store the existing
`CD_CUSTOMLOOPNORMAL` custom data type in order to move the custom
normal storage to generic attributes. This type probably won't get much
use besides that, but generally we don't have reasons not to add these
generic types, besides binary size. In the future we should consolidate
usage of `convert_to_static_type` to avoid large binary size increases
(about 700KB here) for this sort of addition.
This is the first step of #130484.
Pull Request: https://projects.blender.org/blender/blender/pulls/130530
This patch improves working with grease pencil layers in geometry nodes.
* Allow layers to have duplicate names in geometry nodes. In original data, unique names are enforced.
* This allows e.g. duplicating layers and then merging them by name in the end.
* It also resolves a big serial bottleneck when working with many grease pencil layers in geometry nodes. Enforcing unique names is inefficient.
* New `Merge Layers` node that can merge multiple layers by name or by a custom group id.
* Applying a grease pencil modifier now first merges all layers with the same name to ensure all names are unique.
Co-authored-by: Jacques Lucke <jacques@blender.org>
Pull Request: https://projects.blender.org/blender/blender/pulls/127873
This improve the API in multiple aspects:
* No need for an additional `lookup` call to get the current attribute. This
would internally iterate over all attributes again. This leads to O(n^2)
behavior. Note that there are still other reasons for O(n^2) behavior when
processing attributes (where n is the number of attributes).
* Remove the need to return a value from the iteration code to indicate that the
iteration should continue. This is now the default behavior. The iteration can
still be stopped by calling `iter.stop()`.
* Easier access to `is_builtin` property.
* Iterator callback only has a single parameter instead of two (of which one is
sometimes unused).
Pull Request: https://projects.blender.org/blender/blender/pulls/128128
Previously, it was only possible to bake to disk with geometry nodes. This patch
adds support for storing the baked data directly in the .blend file.
By default, new bakes are stored in the .blend file now. Whether a new bake
should be packed or stored on disk can be configured in two places: in the
properties of the bake node and in the bake panel of the modifier. These
settings don't affect existing bakes, only the next bake.
To unpack or pack an individual bake, there is a new operator button next to the
bake button. The icon and the label below indicate where the bake is currently
stored. The label now also contains the size of the bake.
To unpack or pack all bakes, the `File > External Data > Pack Resources / Unpack
Resources` operators can be used. The unpack operator also has a new title that
mentions the number if individual files separate from the number of bakes. This
works better than just listing a number of files because a bake can consist of
many files.
Pull Request: https://projects.blender.org/blender/blender/pulls/124230
Previously, the `AttributeIDRef` wrapper was needed because it also had to
contain a pointer to an `AnonymousAttributeID`. However, since
b279a6d703 this is not necessary anymore.
Therefore we can use "raw" `StringRef` now which reduces the mental overhead
when working with attributes and also simplifies code.
Pull Request: https://projects.blender.org/blender/blender/pulls/127140
The problem was the automatic instance deduplication. There were two
instance references before baking, both of which were referenced by instances.
When loading the bake, they were deduplicated, so there was only one geometry,
but the instances still referenced two.
The fix is to not do deduplication when loading instances from a bake.
`xxHash` is a fast non-cryptographic hashing library. It significantly outperforms
md5 which we use in some places currently while also having great collision
resistance if not attacked explicitly.
The library is added to `extern` because that was the easiest way to do it and has
the least impact on others. I expect this library to become a required dependency
instead of an optional one. It's licence is `BSD 2-Clause` which seems to be the
first of its kind in Blender (there is `BSD 3-Clause` a couple of times).
For now, I used the library only for data deduplication when baking geometry nodes
where the same geometry is generated for each frame. The bake time in my test
goes down from >6s to <1s (note that this includes more than just the hashing time).
Pull Request: https://projects.blender.org/blender/blender/pulls/120139
4d0936c7d7 explicitly avoids turning non-geometry object
instances into geometry instances. This code is called to prepare
geometry sets for baking, and baking currently assumes that the baked
instances are always geometry sets.
To fix this, just check the instance type and serialize an empty
geometry set for the crashing object instance case. Compared to before
the crash causing commit, there is no change in behavior, since that
would have created empty geometry sets too.
Pull Request: https://projects.blender.org/blender/blender/pulls/119892
Similar to 2e6223d90f, but potentially 16 times more effective.
The new attribute is named "instance_transform". It isn't displayed in the
spreadsheet since that wouldn't really be useful. This simplifies a lot of
code since it doesn't have to handle transforms specially anymore. But
complexity is added in the store named attribute node and attribute input
node to keep the old "position" attribute working for compatibility.
Pull Request: https://projects.blender.org/blender/blender/pulls/118531
Implements the design from #116067.
The socket type is called "Matrix" but it is often referred to as "Transform"
when that's what it is semantically. The attribute type is "4x4 Matrix" since
that's a lower level choice. Currently matrix sockets are always passed
around internally as `float4x4`, but that can be optimized in the future
when smaller types would give the same behavior.
A new "Matrix" utilities category has the following set of initial nodes"
- **Combine Transform**
- **Separate Transform**
- **Multiply Matrices**
- **Transform Direction**
- **Transform Vector**
- **Invert Matrix**
- **Transpose Matrix**
The nodes and socket type are behind an experimental flag for now,
which will give us time to make sure it's the right set of initial nodes.
The viewer node overlay doesn't support matrices-- they aren't supported
for rendering in general. They also aren't supported in the modifier interface
currently. But they are supported in the spreadsheet, where the value is
displayed in a tooltip.
Pull Request: https://projects.blender.org/blender/blender/pulls/116166
This means the array can be shared between geometries when unchanged,
reducing memory usage and increasing performance. It also means that
handling the data can be done more generically using the attribute
system. In the future the transforms can become attributes too.
Two other changes are related here:
- The "almost unique ids" are cached with a shared cache so they
are shared too.
- The reference indices are baked as an attribute now, making the process
more generic and potentially easier to optimize in the future.
Pull Request: https://projects.blender.org/blender/blender/pulls/117951
This adds support for baking the volume component of a geometry. Previously,
volumes were just removed in the simulation and bake node.
On disk, each volume geometry is written to a separate `.vdb` file that is stored in
the bakes `blobs` directory and referenced from the corresponding meta `.json` file.
Technically, it would also be easy to write the volume data to the same `.blob`
files that we also write e.g. mesh attributes to. However, since `.vdb` is a well
known file format, it seems reasonable to just store it as a separate file. The
serialization code doesn't really care whether it's a separate file or embedded into
a bigger file, so this decision could be made at a higher level.
Just like with other geometry types, materials are preserved. Just note that when
using the written stand-alone .vdb files, materials are not preserved.
Currently, volume grids are not deduplicated on disk. This could be added in the
future if necessary.
Pull Request: https://projects.blender.org/blender/blender/pulls/117781
This adds hash-based data deduplication when baking in
geometry nodes. All arrays that are written to `.blob` files
are hashed. If an array is detected to have the same hash
as a previously written array, it is not written again. Instead
the same memory is reused.
We already have a similar optimization, but that only worked .with data that was already implicitly shared. Doing this kind
of deduplication with implicitly shared data has the benefit,
that the equality check is constant time. The hash based
approach implemented here requires linear time in the size
of the array, but works on all kinds of data. Both optimizations
work together. So the hashing is skipped if possible.
The hash-based deduplication primarily benefits cases where
the data is regenerated on each frame, so the data between .frames is not shared. One example used to require 2.9 GB
disk space. Now it only requires 542 MB. Additionally, the
duplicate arrays will now be implicitly shared between frames
when reading the baked data later.
An extended version of this approach which also detects partial
duplicates is implemented in #117749.
Pull Request: https://projects.blender.org/blender/blender/pulls/117768
Bake items are generally identified by their (generated) identifier.
This allows changing the name and reordering sockets without breaking
baked data.
In the future we want to have some kind of Import Bake node that
ideally automatically creates its output sockets and names them correctly.
For that to work, the baked data has to contain the user-defined names
for each socket. Those names are not used yet.
With this patch, materials are kept intact in simulation zones and bake nodes
without any additional user action.
This implements the design proposed in #108410 to support referencing
data-blocks (only materials for now) in the baked data. The task also describes
why this is not a trivial issue. A previous attempt was implemented in #109703
but it didn't work well-enough.
The solution is to have an explicit `name (+ library name) -> data-block`
mapping that is stored in the modifier for each bake node and simulation zone.
The `library name` is necessary for it to be unique within a .blend file. Note
that this refers to the name of the `Library` data-block and not a file path.
The baked data only contains the names of the used data-blocks. When the baked
data is loaded, the correct material data-block is looked up from the mapping.
### Automatic Mapping Generation
The most tricky aspect of this approach is to make it feel mostly automatic.
From the user point-of-view, it should just work. Therefore, we don't want the
user to have to create the mapping manually in the majority of cases. Creating
the mapping automatically is difficult because the data-blocks that should
become part of the mapping are only known during depsgraph evaluation. So we
somehow have to gather the missing data blocks during evaluation and then write
the new mappings back to the original data.
While writing back to original data is something we do in some cases already,
the situation here is different, because we are actually creating new relations
between data-blocks. This also means that we'll have to do user-counting. Since
user counts in data-blocks are *not* atomic, we can't do that from multiple
threads at the same time. Also, under some circumstances, it may be necessary to
trigger depsgraph evaluation again after the write-back because it actually
affects the result.
To solve this, a small new API is added in `DEG_depsgraph_writeback_sync.hh`. It
allows gathering tasks which write back to original data in a synchronous way
which may also require a reevaluation.
### Accessing the Mapping
A new `BakeDataBlockMap` is passed to geometry nodes evaluation by the modifier.
This map allows getting the `ID` pointer that should be used for a specific
data-block name that is stored in baked data. It's also used to gather all the
missing data mappings during evaluation.
### Weak ID References
The baked/cached geometries may have references to other data-blocks (currently
only materials, but in the future also e.g. instanced objects/collections).
However, the pointers of these data-blocks are not stable over time. That is
especially true when storing/loading the data from disk, but also just when
playing back the animation. Therefore, the used data-blocks have to referenced
in a different way at run-time.
This is solved by adding `std::unique_ptr<bake::BakeMaterialsList>` to the
run-time data of various geometry data-blocks. If the data-block is cached over
a longer period of time (such that material pointers can't be used directly), it
stores the material name (+ library name) used by each material slot. When the
geometry is used again, the material pointers are restored using these weak name
references and the `BakeDataBlockMap`.
### Manual Mapping Management
There is a new `Data-Blocks` panel in the bake settings in the node editor
sidebar that allows inspecting and modifying the data-blocks that are used when
baking. The user can change what data-block a specific name is mapped to.
Pull Request: https://projects.blender.org/blender/blender/pulls/117043
Along with the 4.1 libraries upgrade, we are bumping the clang-format
version from 8-12 to 17. This affects quite a few files.
If not already the case, you may consider pointing your IDE to the
clang-format binary bundled with the Blender precompiled libraries.
Some common headers were including this. Separating the includes
will ideally lead to better conceptual separation between CustomData
and the attribute API too. Mostly the change is adding the file to
places where it was included indirectly before. But some code is
shuffled around to hopefully better places as well.
Each value is now out of the global namespace, so they can be shorter
and easier to read. Most of this commit just adds the necessary casting
and namespace specification. `enum class` can be forward declared since
it has a specified size. We will make use of that in the next commit.
Use the standard "elements_num" naming, and use the "corner" name rather
than the old "loop" name: `verts_num`, `edges_num`, and `corners_num`.
This matches the existing `faces_num` field which was already renamed.
Pull Request: https://projects.blender.org/blender/blender/pulls/116350
This adds a new `Bake` node which allows saving and loading intermediate geometries.
Typical use cases we want address with this currently are:
* Bake some data for use with a render engine.
* Bake parts of the node tree explicitly for better performance.
For now, the format that is written to disk is not considered to be an import/export format.
It's not guaranteed that data written with one Blender version can be read by another
Blender version. For that it's better to use proper interchange formats. Better support for
those will be added eventually as well. We also plan an `Import Bake` node that allows
reading the blender-specific baked data independent of the Bake node and at different frames.
The baking works very similar to the baking in the simulation zone (UI and implementation
wise). Major differences are:
* The Bake node has a `Bake Still` and `Bake Animation` mode.
* The Bake node doesn't do automatic caching.
Implementation details:
* Refactored how we create the Python operators for moving socket items so that it also
makes sense for non-zones.
* The `ModifierCache` stores an independent map of `SimulationNodeCache` and
`BakeNodeCache`, but both share a common data structure for the actually baked data.
* For baking, the `Bake` node is added as a side-effect-node in the modifier. This will make
sure that the node is baked even if it's currently not connected to the output.
* Had to add a new `DEG_id_tag_update_for_side_effect_request` function that is used
during baking. It's necessary because I want to evaluate the object again even though none
of its inputs changed. The reevaluation is necessary to create the baked data. Using
`DEG_id_tag_update` technically works as well, but has the problem that it also uses the
`DEG_UPDATE_SOURCE_USER_EDIT` flag which (rightly) invalidates simulation caches
which shouldn't happen here.
* Slightly refactored the timeline drawing so that it can also show the baked ranges of
Bake nodes. It does not show anything for baked nodes with a in Still mode though.
* The bake operator is refactored to bake a list of `NodeBakeRequest` which makes the
code easier to follow compared to the previous nested
`ObjectBakeData > ModifierBakeData > NodeBakeData` data structure.
* The bake operators are disabled when the .blend file is not yet saved. This is technically
only necessary when the bake path depends on the .blend file path but seems ok to force
the user anyway (otherwise the bake path may be lost as well if it's set explicitly).
* The same operators are used to bake and delete single bakes in `Bake` nodes and
`Simulation Zones`. On top of that, there are separate operators of baking and deleting all
simulation bakes (those ignore bake nodes).
* The `Bake` node remembers which inputs have been fields and thus may be baked as attributes.
For that it uses an `Is Attribute` flag on the socket item. This is needed because the baked data
may still contain attribute data, even if the inputs to the bake node are disconnected.
* Similar to simulation zones, the behavior of `Bake` nodes is passed into the geometry nodes
evaluation from the outside (from the modifier only currently). This is done by providing the
new `GeoNodesBakeParams` in `GeoNodesCallData` when executing geometry nodes.
Next Steps (mostly because they also involve simulations):
* Visualize nodes that have not been evaluated in the last evaluation.
* Fix issue with seemingly loosing baked data after undo.
* Improve error handling when baked data is not found.
* Show bake node in link drag search.
* Higher level tools for managing bakes.
Pull Request: https://projects.blender.org/blender/blender/pulls/115466
