This refactors the lifetime analysis of anonymous attributes in geometry nodes.
The refactor has a couple of goals:
* Use a better and simpler abstraction that can be used when building the
lazy-function graph. We currently have a bunch of duplicate code to handle
"field source" and "caller propagation" attributes. This is now unified so
that one only has to worry about one kind of "reference sets".
* Make the abstraction compatible with handling bundles and closures in case we
want to support them in the future. Both types can contain geometries and
fields so they need to be taken into account when determining lifetimes.
* Make more parts independent of the concept of "anonymous attributes". In
theory, there could be more kinds of referenced data whose lifetimes need to
be managed. I don't have any concrete plans for adding any though.
At its core, deterministic anonymous attributes still work the same they have
been since they became deterministic [0]. Even the generated lazy-function graph
is still pretty much or even exactly the same as before.
The patch renames `AnonymousAttributeSet` to the more general
`GeometryNodesReferenceSet` which is more. This also makes more places
independent of the concept of anonymous attributes. Functionally, this still the
same though. It's only used in the internals of geometry nodes nowadays. Most
code just gets an `AttributeFilter` that is based on it.
[0]: https://archive.blender.org/developer/D16858
Pull Request: https://projects.blender.org/blender/blender/pulls/128667
This improve the API in multiple aspects:
* No need for an additional `lookup` call to get the current attribute. This
would internally iterate over all attributes again. This leads to O(n^2)
behavior. Note that there are still other reasons for O(n^2) behavior when
processing attributes (where n is the number of attributes).
* Remove the need to return a value from the iteration code to indicate that the
iteration should continue. This is now the default behavior. The iteration can
still be stopped by calling `iter.stop()`.
* Easier access to `is_builtin` property.
* Iterator callback only has a single parameter instead of two (of which one is
sometimes unused).
Pull Request: https://projects.blender.org/blender/blender/pulls/128128
For most cases where the iteration index is needed, a For Each zone is the
better than using a Repeat zone. It's easier to use and more optimizable.
However, in some cases a Repeat zone is actually the right solution and the
index often useful in such cases too.
Pull Request: https://projects.blender.org/blender/blender/pulls/127521
This adds a new type of zone to Geometry Nodes that allows executing some nodes
for each element in a geometry.
## Features
* The `Selection` input allows iterating over a subset of elements on the set
domain.
* Fields passed into the input node are available as single values inside of the
zone.
* The input geometry can be split up into separate (completely independent)
geometries for each element (on all domains except face corner).
* New attributes can be created on the input geometry by outputting a single
value from each iteration.
* New geometries can be generated in each iteration.
* All of these geometries are joined to form the final output.
* Attributes from the input geometry are propagated to the output
geometries.
## Evaluation
The evaluation strategy is similar to the one used for repeat zones. Namely, it
dynamically builds a `lazy_function::Graph` once it knows how many iterations
are necessary. It contains a separate node for each iteration. The inputs for
each iteration are hardcoded into the graph. The outputs of each iteration a
passed to a separate lazy-function that reduces all the values down to the final
outputs. This final output can have a huge number of inputs and that is not
ideal for multi-threading yet, but that can still be improved in the future.
## Performance
There is a non-neglilible amount of overhead for each iteration. The overhead is
way larger than the per-element overhead when just doing field evaluation.
Therefore, normal field evaluation should be preferred when possible. That can
partially still be optimized if there is only some number crunching going on in
the zone but that optimization is not implemented yet.
However, processing many small geometries (e.g. each hair of a character
separately) will likely **always be slower** than working on fewer larger
geoemtries. The additional flexibility you get by processing each element
separately comes at the cost that Blender can't optimize the operation as well.
For node groups that need to handle lots of geometry elements, we recommend
trying to design the node setup so that iteration over tiny sub-geometries is
not required.
An opposite point is true as well though. It can be faster to process more
medium sized geometries in parallel than fewer very large geometries because of
more multi-threading opportunities. The exact threshold between tiny, medium and
large geometries depends on a lot of factors though.
Overall, this initial version of the new zone does not implement all
optimization opportunities yet, but the points mentioned above will still hold
true later.
Pull Request: https://projects.blender.org/blender/blender/pulls/127331
Previously, we only measured the execution time of built-in nodes. To get
execution times of node groups, the time of each nested node was accumulated.
This can lead to very bad accuracy when multiple nodes are evaluated at the same
time.
With this patch, we measure the time spend in each compute context more directly
instead of relying on accumulating many small measurements. This also opens up
some potential optimization opportunities, because we can skip measuring the
time for contexts that we don't care about. However, that is not implemented
yet.
The time shown in the UI can still be misleading when many things are going on
at the same time, but it should at least be more accurate in more situations
now.
Pull Request: https://projects.blender.org/blender/blender/pulls/127658
The tricky thing here is that this graph is only generated while geometry nodes
is evaluated and is generally only stored temporarily. To make it accessible via
Python, the accessor method will cause a reevaluation specifically to log the
generated graph.
This allows building tools that simplify Blender core development.
The depsgraph already exposed the dot graph using the `debug_relations_graphviz`
method. This has been extended to return the data as a string. The filepath is
optional now.
Similar functions have been added for node trees. Note that these should be used
on evaluated node trees.
This removes `AnonymousAttributeID` which was "attached" to every anonymous
attribute before. It adds more complexity than is justified for its
functionality.
It was originally introduced to keep the reference count of the anonymous
attribute so that it can be deleted automatically when the attribute is not
referenced anymore. For quite some time we have had deterministic attribute
life-times though which don't rely on the reference count anymore.
Anonymous attributes are sometimes shown in the UI as "friendly looking" string
like `"UV Map" from Cube`. Some information necessary for this was also stored
in `AnonymousAttributeID`. However, this can also be solved differently.
Specifically, this functionality has now been added directly to
`AttributeFieldInput`.
This refactor also allows removing `AttributeIDRef` which was mainly introduced
because we had to keep the `AnonymousAttributeID` attached with the attribute
name. Just using simple string types to identify attributes can reduce the
mental overhead quite significantly. This will be done as a separate refactor
though.
Pull Request: https://projects.blender.org/blender/blender/pulls/127081
This implements the `Warning` node that allows node groups to communicate
expectations about input values to the user.
By default, the `Warning` node is only evaluated if the node group that contains
it is evaluated in any way. This is better than always evaluating it, because
that could trigger lots of unnecessary evaluation in parts of the potentially
large node tree which should be ignored. In this basic mode, the output of the
node should not be connected to anything and it must not be in a zone.
For more fine-grained control for when the `Warning` node should be evaluated,
one can use the boolean output which is just a pass-through of the `Show` input.
If this output is used, the `Warning` node will only be evaluated if its output
is used. A simple way to use it is to control a Switch node with it that e.g.
"disables" a specific output when the inputs are invalid. In this case, the
`Warning` node may also be in a zone.
The node allows the user to choose between 3 severity levels: Error, Warning and
Info. Those are the same levels that we use internally. Currently, the error and
warning mode are pretty much the same, but that may change in the future.
Pull Request: https://projects.blender.org/blender/blender/pulls/125544
Previously, values for `ID.flag` and `ID.tag` used the prefixes `LIB_` and
`LIB_TAG` respectively. This was somewhat confusing because it's not really
related to libraries in general. This patch changes the prefix to `ID_FLAG_` and
`ID_TAG_`. This makes it more obvious what they correspond to, simplifying code.
Pull Request: https://projects.blender.org/blender/blender/pulls/125811
This shouldn't make any difference right now, because the main inputs
and outputs of a group always come first. But it's better to not rely on that
here if it can be avoided.
The lazy-function for a logical-or made the wrong assumption that
`try_get_input_data_ptr_or_request` returns null when `try_get_input_data_ptr`
returns null for the same input right before that. That's not true, because the
input might have been computed by another thread in the mean-time.
This wrong assumption lead to a bug because lazy-functions are always assumed to
either request more unavailable inputs, or compute all requested outputs. Here,
the lazy-function did neither. It wanted to request a new input, but it was
available already.
The solution is to handle the return value of
`try_get_input_data_ptr_or_request` properly.
Pull Request: https://projects.blender.org/blender/blender/pulls/124465
This adds support for attaching gizmos for input values. The goal is to make it
easier for users to set input values intuitively in the 3D viewport.
We went through multiple different possible designs until we settled on the one
implemented here. We picked it for it's flexibility and ease of use when using
geometry node assets. The core principle in the design is that **gizmos are
attached to existing input values instead of being the input value themselves**.
This actually fits the existing concept of gizmos in Blender well, but may be a
bit unintutitive in a node setup at first. The attachment is done using links in
the node editor.
The most basic usage of the node is to link a Value node to the new Linear Gizmo
node. This attaches the gizmo to the input value and allows you to change it
from the 3D view. The attachment is indicated by the gizmo icon in the sockets
which are controlled by a gizmo as well as the back-link (notice the double
link) when the gizmo is active.
The core principle makes it straight forward to control the same node setup from
the 3D view with gizmos, or by manually changing input values, or by driving the
input values procedurally.
If the input value is controlled indirectly by other inputs, it's often possible
to **automatically propagate** the gizmo to the actual input.
Backpropagation does not work for all nodes, although more nodes can be
supported over time.
This patch adds the first three gizmo nodes which cover common use cases:
* **Linear Gizmo**: Creates a gizmo that controls a float or integer value using
a linear movement of e.g. an arrow in the 3D viewport.
* **Dial Gizmo**: Creates a circular gizmo in the 3D viewport that can be
rotated to change the attached angle input.
* **Transform Gizmo**: Creates a simple gizmo for location, rotation and scale.
In the future, more built-in gizmos and potentially the ability for custom
gizmos could be added.
All gizmo nodes have a **Transform** geometry output. Using it is optional but
it is recommended when the gizmo is used to control inputs that affect a
geometry. When it is used, Blender will automatically transform the gizmos
together with the geometry that they control. To achieve this, the output should
be merged with the generated geometry using the *Join Geometry* node. The data
contained in *Transform* output is not visible geometry, but just internal
information that helps Blender to give a better user experience when using
gizmos.
The gizmo nodes have a multi-input socket. This allows **controlling multiple
values** with the same gizmo.
Only a small set of **gizmo shapes** is supported initially. It might be
extended in the future but one goal is to give the gizmos used by different node
group assets a familiar look and feel. A similar constraint exists for
**colors**. Currently, one can choose from a fixed set of colors which can be
modified in the theme settings.
The set of **visible gizmos** is determined by a multiple factors because it's
not really feasible to show all possible gizmos at all times. To see any of the
geometry nodes gizmos, the "Active Modifier" option has to be enabled in the
"Viewport Gizmos" popover. Then all gizmos are drawn for which at least one of
the following is true:
* The gizmo controls an input of the active modifier of the active object.
* The gizmo controls a value in a selected node in an open node editor.
* The gizmo controls a pinned value in an open node editor. Pinning works by
clicking the gizmo icon next to the value.
Pull Request: https://projects.blender.org/blender/blender/pulls/112677
The goal here is to remove the assumption that multi-inputs will always
have a corresponding node that packs all individual values into a `Vector`
(which is what `LazyFunctionForMultiInput` is doing).
This is slight optimization. Number of nodes with additional outputs like `Valid`
flag of other math-related things will be more in future, its better to strict api
and do not evaluate even singles. This also can reduce number of logged values.
Pull Request: https://projects.blender.org/blender/blender/pulls/121799
Move all header file into namespace.
Unnecessary namespaces was removed from implementations file.
Part of forward declarations in header was moved in the top part
of file just to do not have a lot of separate namespaces.
Pull Request: https://projects.blender.org/blender/blender/pulls/121637
This changes the menu switch socket to use the socket-items system
(`NOD_socket_items.hh`) that is already used by the simulation zone, repeat
zone, bake node and index switch node. By using this system, the per-node
boilerplate can be removed significantly. This is especially important as we
plan to have dynamic socket amounts in more nodes in the future.
There are some user visible changes which make the node more consistent with
others:
* Move the menu items list into the properties panel as in 0c585a1b8a.
* Add an extend socket.
* Duplicating a menu item keeps the name of the old one.
There is also a (backward compatible) change in the Python API: It's now
possible to directly access `node.enum_items` and `node.active_index` instead of
having to use `node.enum_definition.enum_items`. This is consistent with the
other nodes. For backward compatibility, `node.enum_definition` still exists,
but simply returns the node itself.
Many API functions from `NodeEnumDefinition` like
`NodeEnumDefinition::remove_item` have been removed. Those are not used anymore
and are unnecessary boilerplate. If ever necessary, they can be implemented back
in terms of the socket-items system.
The socket-items system had to be extended a little bit to support the case for
the menu switch node where each socket item has a name but no type. Previously,
there was the case without name and type in the index switch node, and the case
with both in the bake node and zones. The system was trivial to extend to this
case.
Pull Request: https://projects.blender.org/blender/blender/pulls/121234
Currently for node tools we create and evaluate a temporary depsgraph
with all the selected object data-blocks and all data-blocks referenced
by the node tree.
Needless to say, this can be very slow when those data-blocks contain
arbitrary procedural operations. Re-evaluating all the selected objects
is particularly because it will give a slowdown even in very basic uses
of node tools.
Originally I hoped that geometry nodes could be made to work with
original as well as evaluated data-blocks. But that would require far
too many tricky changes and arguably isn't right design-wise anyway.
Instead of that, this commit makes node tools dependency graph
evaluation more fine-grained in a few ways.
1. Remove the evaluation of selected objects. These are always visible
in the viewport and part of the active depsgraph anyway. To protect
against cyclic dependencies, we now compare `orig_id` instead of the
object pointer itself.
2. Evaluate the node group and its dependencies in a separate depsgraph
used only when necessary. This allows using the original node tree
without any copies when it doesn't reference any data-blocks.
3. Evaluate IDs from node group inputs (from the redo panel) in the extra
depsgraph as well, only when necessary.
Pull Request: https://projects.blender.org/blender/blender/pulls/120723
To know if link is connected to dangling reroute and can be skipped
as value-less, we need to know if reroute is dangling. This requires
graph traversal. Currently this is done by non-recursive iteration.
But this can lead quadratic complexity for some of the cases.
Other way is to make this linear while cache building.
Pull Request: https://projects.blender.org/blender/blender/pulls/120375
This adds a new special purpose container data structure that can be
used to gather many elements into many (potentially small) lists efficiently.
I originally worked on this data structure because I might want to use it
in #118772. However, also it's useful in the geometry nodes logger already.
I'm measuring a 10-20% speed improvement in my many-math-nodes file
when I enable logging for all sockets (not just the ones that are currently visible).
Pull Request: https://projects.blender.org/blender/blender/pulls/118774
Previously, the viewer node would work, but since the simulation output node
was never evaluated, the new simulation state was never stored. Now make
sure that the simulation output node is executed when there is an active viewer
inside the simulation.
Pull Request: https://projects.blender.org/blender/blender/pulls/118804
With this patch, materials are kept intact in simulation zones and bake nodes
without any additional user action.
This implements the design proposed in #108410 to support referencing
data-blocks (only materials for now) in the baked data. The task also describes
why this is not a trivial issue. A previous attempt was implemented in #109703
but it didn't work well-enough.
The solution is to have an explicit `name (+ library name) -> data-block`
mapping that is stored in the modifier for each bake node and simulation zone.
The `library name` is necessary for it to be unique within a .blend file. Note
that this refers to the name of the `Library` data-block and not a file path.
The baked data only contains the names of the used data-blocks. When the baked
data is loaded, the correct material data-block is looked up from the mapping.
### Automatic Mapping Generation
The most tricky aspect of this approach is to make it feel mostly automatic.
From the user point-of-view, it should just work. Therefore, we don't want the
user to have to create the mapping manually in the majority of cases. Creating
the mapping automatically is difficult because the data-blocks that should
become part of the mapping are only known during depsgraph evaluation. So we
somehow have to gather the missing data blocks during evaluation and then write
the new mappings back to the original data.
While writing back to original data is something we do in some cases already,
the situation here is different, because we are actually creating new relations
between data-blocks. This also means that we'll have to do user-counting. Since
user counts in data-blocks are *not* atomic, we can't do that from multiple
threads at the same time. Also, under some circumstances, it may be necessary to
trigger depsgraph evaluation again after the write-back because it actually
affects the result.
To solve this, a small new API is added in `DEG_depsgraph_writeback_sync.hh`. It
allows gathering tasks which write back to original data in a synchronous way
which may also require a reevaluation.
### Accessing the Mapping
A new `BakeDataBlockMap` is passed to geometry nodes evaluation by the modifier.
This map allows getting the `ID` pointer that should be used for a specific
data-block name that is stored in baked data. It's also used to gather all the
missing data mappings during evaluation.
### Weak ID References
The baked/cached geometries may have references to other data-blocks (currently
only materials, but in the future also e.g. instanced objects/collections).
However, the pointers of these data-blocks are not stable over time. That is
especially true when storing/loading the data from disk, but also just when
playing back the animation. Therefore, the used data-blocks have to referenced
in a different way at run-time.
This is solved by adding `std::unique_ptr<bake::BakeMaterialsList>` to the
run-time data of various geometry data-blocks. If the data-block is cached over
a longer period of time (such that material pointers can't be used directly), it
stores the material name (+ library name) used by each material slot. When the
geometry is used again, the material pointers are restored using these weak name
references and the `BakeDataBlockMap`.
### Manual Mapping Management
There is a new `Data-Blocks` panel in the bake settings in the node editor
sidebar that allows inspecting and modifying the data-blocks that are used when
baking. The user can change what data-block a specific name is mapped to.
Pull Request: https://projects.blender.org/blender/blender/pulls/117043
This patch adds support for _Menu Switch_ nodes and enum definitions in
node trees more generally. The design is based on the outcome of the
[2022 Nodes Workshop](https://code.blender.org/2022/11/geometry-nodes-workshop-2022/#menu-switch).
The _Menu Switch_ node is an advanced version of the _Switch_ node which
has a customizable **menu input socket** instead of a simple boolean.
The _items_ of this menu are owned by the node itself. Each item has a
name and description and unique identifier that is used internally. A
menu _socket_ represents a concrete value out of the list of items.
To enable selection of an enum value for unconnected sockets the menu is
presented as a dropdown list like built-in enums. When the socket is
connected a shared pointer to the enum definition is propagated along
links and stored in socket default values. This allows node groups to
expose a menu from an internal menu switch as a parameter. The enum
definition is a runtime copy of the enum items in DNA that allows
sharing.
A menu socket can have multiple connections, which can lead to
ambiguity. If two or more different menu source nodes are connected to a
socket it gets marked as _undefined_. Any connection to an undefined
menu socket is invalid as a hint to users that there is a problem. A
warning/error is also shown on nodes with undefined menu sockets.
At runtime the value of a menu socket is the simple integer identifier.
This can also be a field in geometry nodes. The identifier is unique
within each enum definition, and it is persistent even when items are
added, removed, or changed. Changing the name of an item does not affect
the internal identifier, so users can rename enum items without breaking
existing input values. This also persists if, for example, a linked node
group is temporarily unavailable.
Pull Request: https://projects.blender.org/blender/blender/pulls/113445
Possibly also fixes write past array boundaries.
Happens with the mikassa-shading.blend file, by simply opening
the file in debug mode. The fix is suggested by Jacques.
Pull Request: https://projects.blender.org/blender/blender/pulls/117331
This simplifies the code that creates the zone socket mappings at the cost of
slightly more memory per zone, which is not significant.
Previously, `IndexRange` was used where it now uses `Vector<int>`.
Pull Request: https://projects.blender.org/blender/blender/pulls/116939
`set_default_remaining_node_outputs` didn't work because the mapping
between the original node group sockets and the lazy function outputs
wasn't set up during the construction of the node type, as done by the
bake node and others.
Along with the 4.1 libraries upgrade, we are bumping the clang-format
version from 8-12 to 17. This affects quite a few files.
If not already the case, you may consider pointing your IDE to the
clang-format binary bundled with the Blender precompiled libraries.
