This patch adds a new `BLI_mutex.hh` header which adds `blender::Mutex` as alias
for either `tbb::mutex` or `std::mutex` depending on whether TBB is enabled.
Description copied from the patch:
```
/**
* blender::Mutex should be used as the default mutex in Blender. It implements a subset of the API
* of std::mutex but has overall better guaranteed properties. It can be used with RAII helpers
* like std::lock_guard. However, it is not compatible with e.g. std::condition_variable. So one
* still has to use std::mutex for that case.
*
* The mutex provided by TBB has these properties:
* - It's as fast as a spin-lock in the non-contended case, i.e. when no other thread is trying to
* lock the mutex at the same time.
* - In the contended case, it spins a couple of times but then blocks to avoid draining system
* resources by spinning for a long time.
* - It's only 1 byte large, compared to e.g. 40 bytes when using the std::mutex of GCC. This makes
* it more feasible to have many smaller mutexes which can improve scalability of algorithms
* compared to using fewer larger mutexes. Also it just reduces "memory slop" across Blender.
* - It is *not* a fair mutex, i.e. it's not guaranteed that a thread will ever be able to lock the
* mutex when there are always more than one threads that try to lock it. In the majority of
* cases, using a fair mutex just causes extra overhead without any benefit. std::mutex is not
* guaranteed to be fair either.
*/
```
The performance benchmark suggests that the impact is negilible in almost
all cases. The only benchmarks that show interesting behavior are the once
testing foreach zones in Geometry Nodes. These tests are explicitly testing
overhead, which I still have to reduce over time. So it's not unexpected that
changing the mutex has an impact there. What's interesting is that on macos the
performance improves a lot while on linux it gets worse. Since that overhead
should eventually be removed almost entirely, I don't really consider that
blocking.
Links:
* Documentation of different mutex flavors in TBB:
https://www.intel.com/content/www/us/en/docs/onetbb/developer-guide-api-reference/2021-12/mutex-flavors.html
* Older implementation of a similar mutex by me:
https://archive.blender.org/developer/differential/0016/0016711/index.html
* Interesting read regarding how a mutex can be this small:
https://webkit.org/blog/6161/locking-in-webkit/
Pull Request: https://projects.blender.org/blender/blender/pulls/138370
Use the safer VArraySpan instead of get_internal_span(). And remove
the size check for the output data. If the callback inside `ensure` is
called it means that recomputing the data is necessary, regardless of
the array's current size.
Previously we generally expected CustomData layers to have implicit
sharing info, but we didn't require it. This PR clarifies that we do
require layers with non-null data to have implicit sharing info. This
generally makes code simpler because we don't have to have a separate
code path for non-shared layers. For example, it makes the "totelem"
arguments for layer freeing functions unnecessary, since shared data
knows how to free itself. Those arguments are removed in this PR.
Pull Request: https://projects.blender.org/blender/blender/pulls/134578
Previously, the attribute accessor were defined in the `geometry_component_*.cc`
files. This made sense back in the day, because this attribute API was only used
through `GeometryComponent`. However, nowadays this attribute API is independent
of `GeometryComponent`. E.g. one can use `mesh.attributes()` without ever
creating a component.
The refactor contains the following changes:
* Move attribute accessors to separate files for each geometry type. E.g. from
`geometry_component_mesh.cc` to `mesh_attributes.cc`.
* Move implementations of e.g. `Mesh::attributes()` to `mesh.cc`.
* Provide access to the `AttributeAccessorFunctions` without actually having a
geometry. This will be useful to e.g. implement
`attribute_is_builtin_on_component_type` without dummy components.
Pull Request: https://projects.blender.org/blender/blender/pulls/130516
Joining many geometries was O(n^2) because of deduplication of the same
components was not done using a map. My test file that generates 1000
stars in a repeat zone got 10x faster, but it's possible to create a file for any
speedup.
This introduces the concept of an #AttributeFilter. It's used to tell a geometry
algorithm which attributes it should process/propagate and which can be ignored.
We already had something similar before named
`AnonymousAttributePropagationInfo`. However, as the name implies, this was
specific to anonymous attributes. This had some downsides:
* A lot of code had to be aware of the concept of anonymous attributes even if
it did nothing special with anonymous attributes.
* For non-anonymous attributes we often had a separate `Set<std::string> skip`
parameter. It's not nice to have to pass two kinds of filters around and to
have to construct a `Set<std::string>` in many cases.
`AttributeFilter` solves both of these downsides.
Technically, `AttributeFilter` could also just be a `FunctionRef<bool(StringRef
attribute_name)>`, but that also has some issues:
* The `bool` return value is often ambiguous, i.e. it's not clear if it means
that the attribute should be processed or not. Using an enum works better.
* Passing function refs around and combining them works, but can very easily
lead to dangling references.
* The default value of a `FunctionRef` is "empty", i.e. it can't be called. It's
generally more nice to not have a special case for the default value. Now the
default `AttributeFilter` propagates all attributes without any extra handling
on the call-site.
Pull Request: https://projects.blender.org/blender/blender/pulls/127155
And same for the `copy_layout` function. These functions do not free any
potentially existing data in destination, but expect it to be uninitialized.
Hopefully these new names will make it more clear and avoid bugs like #122160.
We often have the situation where it would be good if we could easily estimate
the memory usage of some value (e.g. a mesh, or volume). Examples of where we
ran into this in the past:
* Undo step size.
* Caching of volume grids.
* Caching of loaded geometries for import geometry nodes.
Generally, most caching systems would benefit from the ability to know how much
memory they currently use to make better decisions about which data to free and
when. The goal of this patch is to introduce a simple general API to count the
memory usage that is independent of any specific caching system. I'm doing this
to "fix" the chicken and egg problem that caches need to know the memory usage,
but we don't really need to count the memory usage without using it for caches.
Implementing caching and memory counting at the same time make both harder than
implementing them one after another.
The main difficulty with counting memory usage is that some memory may be shared
using implicit sharing. We want to avoid double counting such memory. How
exactly shared memory is treated depends a bit on the use case, so no specific
assumptions are made about that in the API. The gathered memory usage is not
expected to be exact. It's expected to be a decent approximation. It's neither a
lower nor an upper bound unless specified by some specific type. Cache systems
generally build on top of heuristics to decide when to free what anyway.
There are two sides to this API:
1. Get the amount of memory used by one or more values. This side is used by
caching systems and/or systems that want to present the used memory to the
user.
2. Tell the caller how much memory is used. This side is used by all kinds of
types that can report their memory usage such as meshes.
```cpp
/* Get how much memory is used by two meshes together. */
MemoryCounter memory;
mesh_a->count_memory(memory);
mesh_b->count_memory(memory);
int64_t bytes_used = memory.counted_bytes();
/* Tell the caller how much memory is used. */
void Mesh::count_memory(blender::MemoryCounter &memory) const
{
memory.add_shared(this->runtime->face_offsets_sharing_info,
this->face_offsets().size_in_bytes());
/* Forward memory counting to lower level types. This should be fairly common. */
CustomData_count_memory(this->vert_data, this->verts_num, memory);
}
void CustomData_count_memory(const CustomData &data,
const int totelem,
blender::MemoryCounter &memory)
{
for (const CustomDataLayer &layer : Span{data.layers, data.totlayer}) {
memory.add_shared(layer.sharing_info, [&](blender::MemoryCounter &shared_memory) {
/* Not quite correct for all types, but this is only a rough approximation anyway. */
const int64_t elem_size = CustomData_get_elem_size(&layer);
shared_memory.add(totelem * elem_size);
});
}
}
```
Pull Request: https://projects.blender.org/blender/blender/pulls/126295
The problem was the automatic instance deduplication. There were two
instance references before baking, both of which were referenced by instances.
When loading the bake, they were deduplicated, so there was only one geometry,
but the instances still referenced two.
The fix is to not do deduplication when loading instances from a bake.
Previously, it was not possible to see detailed information about instances in
the spreadsheet. Only the attributes on the top level instances were shown. Now,
all nested instances can be inspected too.
Combined with #114910 this will make inspecting more complex geometry with the
spreadsheet much more feasible. It's also an important part of integrating
grease pencil into geometry nodes because it makes it more obvious how layers
are converted to curve instances.
The data-selection is split into two separate tree views now. One that selects
the geometry from the instance tree, and one that's used to select the geometry
component and domain within that geometry. We found that this works better than
combining both tree views into one (we tried that in #124186).
Pull Request: https://projects.blender.org/blender/blender/pulls/125293
This adds a new `name` member to the `GeometrySet` class. This name can be set
with the new `Set Geometry Name` node. Currently, the name is only used in the
spreadsheet when displaying instances.
The main purpose of this name is to help debugging in instance trees. However, in the
future it may also be used when exporting instance trees or when creating separate
objects from them.
Note, the name is not expected to be unique, it is fully in user control.
Naming geometries is necessary to make the spreadsheet more useful for instances,
because currently the user has no information for which geometry is used by each instance.
We also want to use this name to improve the integration with grease pencil where
sometimes layers become instances with the same name.
Pull Request: https://projects.blender.org/blender/blender/pulls/114910
- Sort add menu alphabetically
- Use forward declaration for GeometrySet again
- Use `this->` to access class methods
- Use `MEM_cnew`
- Fix typo
- Pass Span by value
- Pass MutableSpan instead of Vector &
- Remove unnecessary whitespace
- Use `BLI_SCOPED_DEFER` for freeing non-RAII objects
- Use `is_empty()` instead of `size() == 0`
- Use `GeometrySet::from_mesh` ability to handle null argument
Similar to 2e6223d90f, but potentially 16 times more effective.
The new attribute is named "instance_transform". It isn't displayed in the
spreadsheet since that wouldn't really be useful. This simplifies a lot of
code since it doesn't have to handle transforms specially anymore. But
complexity is added in the store named attribute node and attribute input
node to keep the old "position" attribute working for compatibility.
Pull Request: https://projects.blender.org/blender/blender/pulls/118531
This means the array can be shared between geometries when unchanged,
reducing memory usage and increasing performance. It also means that
handling the data can be done more generically using the attribute
system. In the future the transforms can become attributes too.
Two other changes are related here:
- The "almost unique ids" are cached with a shared cache so they
are shared too.
- The reference indices are baked as an attribute now, making the process
more generic and potentially easier to optimize in the future.
Pull Request: https://projects.blender.org/blender/blender/pulls/117951
Some common headers were including this. Separating the includes
will ideally lead to better conceptual separation between CustomData
and the attribute API too. Mostly the change is adding the file to
places where it was included indirectly before. But some code is
shuffled around to hopefully better places as well.
Each value is now out of the global namespace, so they can be shorter
and easier to read. Most of this commit just adds the necessary casting
and namespace specification. `enum class` can be forward declared since
it has a specified size. We will make use of that in the next commit.
This was the last use of `CustomDataAttributes`. It's not very useful
nowadays because we have a lower level (`CustomData`) and higher level
(`AttributeAccessor`) API.
As part of this I removed the ability to `reserve` instances which was
useful when adding instances one by one. Generally, such code should
eventually be rewritten to handle more instances at once, instead of
handling them one by one. This will lead to better performance and
is more in line with how other geometry types (like mesh) are handled.
This also adds the ability to move and assign `Instances` just like we can
move and assign `CurvesGeometry`.
A copy has to compare equal to itself and have the same hash
when it is supposed to be used as a reference in a hash table
like `VectorSet`.
Just making the hash not change during a copy by hashing the
geometry component pointers instead of the geometry-set pointer
does not work because of `geometry_set_from_reference` which
assumed that changing the geometry set does not change the
hash of the reference.
For now the solution is to just not use a hash table as this
makes it easier to get corretness right. Instead, just use a
regular `Vector` to store all the references which avoids
the need for a hash function.
This can now lead to some O(n^2) behavior when adding many
references. Fortunately, this is not too common yet, as usually
one has few references but many instances that use those.
It's still something that has to be solved at some point. It's
not clear yet what approach would work best:
* Reintroduce `VectorSet` for the references and properly update
the reference positions in the hash table after the references
have changed.
* Add a separate `Map<Object*/Collection*, int>` for the
deduplication.
* Do deduplication on the call-site of `add_reference` by building
a temporary map there.
Listing the "Blender Foundation" as copyright holder implied the Blender
Foundation holds copyright to files which may include work from many
developers.
While keeping copyright on headers makes sense for isolated libraries,
Blender's own code may be refactored or moved between files in a way
that makes the per file copyright holders less meaningful.
Copyright references to the "Blender Foundation" have been replaced with
"Blender Authors", with the exception of `./extern/` since these this
contains libraries which are more isolated, any changed to license
headers there can be handled on a case-by-case basis.
Some directories in `./intern/` have also been excluded:
- `./intern/cycles/` it's own `AUTHORS` file is planned.
- `./intern/opensubdiv/`.
An "AUTHORS" file has been added, using the chromium projects authors
file as a template.
Design task: #110784
Ref !110783.
Move `GeometrySet` and `GeometryComponent` and subclasses
to the `blender::bke` namespace. This wasn't done earlier since
these were one of the first C++ classes used throughout Blender,
but now it is common.
Also remove the now-unnecessary C-header, since all users of
the geometry set header are now in C++.
Pull Request: https://projects.blender.org/blender/blender/pulls/109020
A lot of files were missing copyright field in the header and
the Blender Foundation contributed to them in a sense of bug
fixing and general maintenance.
This change makes it explicit that those files are at least
partially copyrighted by the Blender Foundation.
Note that this does not make it so the Blender Foundation is
the only holder of the copyright in those files, and developers
who do not have a signed contract with the foundation still
hold the copyright as well.
Another aspect of this change is using SPDX format for the
header. We already used it for the license specification,
and now we state it for the copyright as well, following the
FAQ:
https://reuse.software/faq/
Goals of this refactor:
* Reduce memory consumption of `IndexMask`. The old `IndexMask` uses an
`int64_t` for each index which is more than necessary in pretty much all
practical cases currently. Using `int32_t` might still become limiting
in the future in case we use this to index e.g. byte buffers larger than
a few gigabytes. We also don't want to template `IndexMask`, because
that would cause a split in the "ecosystem", or everything would have to
be implemented twice or templated.
* Allow for more multi-threading. The old `IndexMask` contains a single
array. This is generally good but has the problem that it is hard to fill
from multiple-threads when the final size is not known from the beginning.
This is commonly the case when e.g. converting an array of bool to an
index mask. Currently, this kind of code only runs on a single thread.
* Allow for efficient set operations like join, intersect and difference.
It should be possible to multi-thread those operations.
* It should be possible to iterate over an `IndexMask` very efficiently.
The most important part of that is to avoid all memory access when iterating
over continuous ranges. For some core nodes (e.g. math nodes), we generate
optimized code for the cases of irregular index masks and simple index ranges.
To achieve these goals, a few compromises had to made:
* Slicing of the mask (at specific indices) and random element access is
`O(log #indices)` now, but with a low constant factor. It should be possible
to split a mask into n approximately equally sized parts in `O(n)` though,
making the time per split `O(1)`.
* Using range-based for loops does not work well when iterating over a nested
data structure like the new `IndexMask`. Therefor, `foreach_*` functions with
callbacks have to be used. To avoid extra code complexity at the call site,
the `foreach_*` methods support multi-threading out of the box.
The new data structure splits an `IndexMask` into an arbitrary number of ordered
`IndexMaskSegment`. Each segment can contain at most `2^14 = 16384` indices. The
indices within a segment are stored as `int16_t`. Each segment has an additional
`int64_t` offset which allows storing arbitrary `int64_t` indices. This approach
has the main benefits that segments can be processed/constructed individually on
multiple threads without a serial bottleneck. Also it reduces the memory
requirements significantly.
For more details see comments in `BLI_index_mask.hh`.
I did a few tests to verify that the data structure generally improves
performance and does not cause regressions:
* Our field evaluation benchmarks take about as much as before. This is to be
expected because we already made sure that e.g. add node evaluation is
vectorized. The important thing here is to check that changes to the way we
iterate over the indices still allows for auto-vectorization.
* Memory usage by a mask is about 1/4 of what it was before in the average case.
That's mainly caused by the switch from `int64_t` to `int16_t` for indices.
In the worst case, the memory requirements can be larger when there are many
indices that are very far away. However, when they are far away from each other,
that indicates that there aren't many indices in total. In common cases, memory
usage can be way lower than 1/4 of before, because sub-ranges use static memory.
* For some more specific numbers I benchmarked `IndexMask::from_bools` in
`index_mask_from_selection` on 10.000.000 elements at various probabilities for
`true` at every index:
```
Probability Old New
0 4.6 ms 0.8 ms
0.001 5.1 ms 1.3 ms
0.2 8.4 ms 1.8 ms
0.5 15.3 ms 3.0 ms
0.8 20.1 ms 3.0 ms
0.999 25.1 ms 1.7 ms
1 13.5 ms 1.1 ms
```
Pull Request: https://projects.blender.org/blender/blender/pulls/104629
Previously, the lifetimes of anonymous attributes were determined by
reference counts which were non-deterministic when multiple threads
are used. Now the lifetimes of anonymous attributes are handled
more explicitly and deterministically. This is a prerequisite for any kind
of caching, because caching the output of nodes that do things
non-deterministically and have "invisible inputs" (reference counts)
doesn't really work.
For more details for how deterministic lifetimes are achieved, see D16858.
No functional changes are expected. Small performance changes are expected
as well (within few percent, anything larger regressions should be reported as
bugs).
Differential Revision: https://developer.blender.org/D16858
* Support bidirectional type lookups. E.g. finding the base type of a
field was supported, but not the other way around. This also removes
the todo in `get_vector_type`. To achieve this, types have to be
registered up-front.
* Separate `CPPType` from other "type traits". For example, previously
`ValueOrFieldCPPType` adds additional behavior on top of `CPPType`.
Previously, it was a subclass, now it just contains a reference to the
`CPPType` it corresponds to. This follows the composition-over-inheritance
idea. This makes it easier to have self-contained "type traits" without
having to put everything into `CPPType`.
Differential Revision: https://developer.blender.org/D16479
Attribute copying often uses identical logic for copying selected
elements or copying with an index map. Instead of reimplementing
this in each file, use the common implementation in the array_utils
namespace. This makes the commonality more obvious, gives improved
performance (this implementation is multithreaded), reduces binary
size (I observed a 173KB reduction), and probably reduces compile time.
This makes instance handling more consistent with all the other geometry
component types. For example, `MeshComponent` contains a `Mesh *` and
now `InstancesComponent` has a `Instances *`.
Differential Revision: https://developer.blender.org/D16137
