Files
test/source/blender/blenkernel/intern/geometry_fields.cc
Jacques Lucke 2cfcb8b0b8 BLI: refactor IndexMask for better performance and memory usage
Goals of this refactor:
* Reduce memory consumption of `IndexMask`. The old `IndexMask` uses an
  `int64_t` for each index which is more than necessary in pretty much all
  practical cases currently. Using `int32_t` might still become limiting
  in the future in case we use this to index e.g. byte buffers larger than
  a few gigabytes. We also don't want to template `IndexMask`, because
  that would cause a split in the "ecosystem", or everything would have to
  be implemented twice or templated.
* Allow for more multi-threading. The old `IndexMask` contains a single
  array. This is generally good but has the problem that it is hard to fill
  from multiple-threads when the final size is not known from the beginning.
  This is commonly the case when e.g. converting an array of bool to an
  index mask. Currently, this kind of code only runs on a single thread.
* Allow for efficient set operations like join, intersect and difference.
  It should be possible to multi-thread those operations.
* It should be possible to iterate over an `IndexMask` very efficiently.
  The most important part of that is to avoid all memory access when iterating
  over continuous ranges. For some core nodes (e.g. math nodes), we generate
  optimized code for the cases of irregular index masks and simple index ranges.

To achieve these goals, a few compromises had to made:
* Slicing of the mask (at specific indices) and random element access is
  `O(log #indices)` now, but with a low constant factor. It should be possible
  to split a mask into n approximately equally sized parts in `O(n)` though,
  making the time per split `O(1)`.
* Using range-based for loops does not work well when iterating over a nested
  data structure like the new `IndexMask`. Therefor, `foreach_*` functions with
  callbacks have to be used. To avoid extra code complexity at the call site,
  the `foreach_*` methods support multi-threading out of the box.

The new data structure splits an `IndexMask` into an arbitrary number of ordered
`IndexMaskSegment`. Each segment can contain at most `2^14 = 16384` indices. The
indices within a segment are stored as `int16_t`. Each segment has an additional
`int64_t` offset which allows storing arbitrary `int64_t` indices. This approach
has the main benefits that segments can be processed/constructed individually on
multiple threads without a serial bottleneck. Also it reduces the memory
requirements significantly.

For more details see comments in `BLI_index_mask.hh`.

I did a few tests to verify that the data structure generally improves
performance and does not cause regressions:
* Our field evaluation benchmarks take about as much as before. This is to be
  expected because we already made sure that e.g. add node evaluation is
  vectorized. The important thing here is to check that changes to the way we
  iterate over the indices still allows for auto-vectorization.
* Memory usage by a mask is about 1/4 of what it was before in the average case.
  That's mainly caused by the switch from `int64_t` to `int16_t` for indices.
  In the worst case, the memory requirements can be larger when there are many
  indices that are very far away. However, when they are far away from each other,
  that indicates that there aren't many indices in total. In common cases, memory
  usage can be way lower than 1/4 of before, because sub-ranges use static memory.
* For some more specific numbers I benchmarked `IndexMask::from_bools` in
  `index_mask_from_selection` on 10.000.000 elements at various probabilities for
  `true` at every index:
  ```
  Probability      Old        New
  0              4.6 ms     0.8 ms
  0.001          5.1 ms     1.3 ms
  0.2            8.4 ms     1.8 ms
  0.5           15.3 ms     3.0 ms
  0.8           20.1 ms     3.0 ms
  0.999         25.1 ms     1.7 ms
  1             13.5 ms     1.1 ms
  ```

Pull Request: https://projects.blender.org/blender/blender/pulls/104629
2023-05-24 18:11:41 +02:00

645 lines
22 KiB
C++

/* SPDX-License-Identifier: GPL-2.0-or-later */
#include "BLI_array_utils.hh"
#include "BKE_attribute.hh"
#include "BKE_curves.hh"
#include "BKE_geometry_fields.hh"
#include "BKE_geometry_set.hh"
#include "BKE_instances.hh"
#include "BKE_mesh.hh"
#include "BKE_pointcloud.h"
#include "BKE_type_conversions.hh"
#include "DNA_mesh_types.h"
#include "DNA_pointcloud_types.h"
#include "BLT_translation.h"
namespace blender::bke {
MeshFieldContext::MeshFieldContext(const Mesh &mesh, const eAttrDomain domain)
: mesh_(mesh), domain_(domain)
{
BLI_assert(mesh.attributes().domain_supported(domain_));
}
CurvesFieldContext::CurvesFieldContext(const CurvesGeometry &curves, const eAttrDomain domain)
: curves_(curves), domain_(domain)
{
BLI_assert(curves.attributes().domain_supported(domain));
}
GeometryFieldContext::GeometryFieldContext(const void *geometry,
const GeometryComponentType type,
const eAttrDomain domain)
: geometry_(geometry), type_(type), domain_(domain)
{
BLI_assert(ELEM(type,
GEO_COMPONENT_TYPE_MESH,
GEO_COMPONENT_TYPE_CURVE,
GEO_COMPONENT_TYPE_POINT_CLOUD,
GEO_COMPONENT_TYPE_INSTANCES));
}
GeometryFieldContext::GeometryFieldContext(const GeometryComponent &component,
const eAttrDomain domain)
: type_(component.type()), domain_(domain)
{
switch (component.type()) {
case GEO_COMPONENT_TYPE_MESH: {
const MeshComponent &mesh_component = static_cast<const MeshComponent &>(component);
geometry_ = mesh_component.get_for_read();
break;
}
case GEO_COMPONENT_TYPE_CURVE: {
const CurveComponent &curve_component = static_cast<const CurveComponent &>(component);
const Curves *curves = curve_component.get_for_read();
geometry_ = curves ? &curves->geometry.wrap() : nullptr;
break;
}
case GEO_COMPONENT_TYPE_POINT_CLOUD: {
const PointCloudComponent &pointcloud_component = static_cast<const PointCloudComponent &>(
component);
geometry_ = pointcloud_component.get_for_read();
break;
}
case GEO_COMPONENT_TYPE_INSTANCES: {
const InstancesComponent &instances_component = static_cast<const InstancesComponent &>(
component);
geometry_ = instances_component.get_for_read();
break;
}
case GEO_COMPONENT_TYPE_VOLUME:
case GEO_COMPONENT_TYPE_EDIT:
BLI_assert_unreachable();
break;
}
}
GeometryFieldContext::GeometryFieldContext(const Mesh &mesh, eAttrDomain domain)
: geometry_(&mesh), type_(GEO_COMPONENT_TYPE_MESH), domain_(domain)
{
}
GeometryFieldContext::GeometryFieldContext(const CurvesGeometry &curves, eAttrDomain domain)
: geometry_(&curves), type_(GEO_COMPONENT_TYPE_CURVE), domain_(domain)
{
}
GeometryFieldContext::GeometryFieldContext(const PointCloud &points)
: geometry_(&points), type_(GEO_COMPONENT_TYPE_POINT_CLOUD), domain_(ATTR_DOMAIN_POINT)
{
}
GeometryFieldContext::GeometryFieldContext(const Instances &instances)
: geometry_(&instances), type_(GEO_COMPONENT_TYPE_INSTANCES), domain_(ATTR_DOMAIN_INSTANCE)
{
}
std::optional<AttributeAccessor> GeometryFieldContext::attributes() const
{
if (const Mesh *mesh = this->mesh()) {
return mesh->attributes();
}
if (const CurvesGeometry *curves = this->curves()) {
return curves->attributes();
}
if (const PointCloud *pointcloud = this->pointcloud()) {
return pointcloud->attributes();
}
if (const Instances *instances = this->instances()) {
return instances->attributes();
}
return {};
}
const Mesh *GeometryFieldContext::mesh() const
{
return this->type() == GEO_COMPONENT_TYPE_MESH ? static_cast<const Mesh *>(geometry_) : nullptr;
}
const CurvesGeometry *GeometryFieldContext::curves() const
{
return this->type() == GEO_COMPONENT_TYPE_CURVE ?
static_cast<const CurvesGeometry *>(geometry_) :
nullptr;
}
const PointCloud *GeometryFieldContext::pointcloud() const
{
return this->type() == GEO_COMPONENT_TYPE_POINT_CLOUD ?
static_cast<const PointCloud *>(geometry_) :
nullptr;
}
const Instances *GeometryFieldContext::instances() const
{
return this->type() == GEO_COMPONENT_TYPE_INSTANCES ? static_cast<const Instances *>(geometry_) :
nullptr;
}
GVArray GeometryFieldInput::get_varray_for_context(const fn::FieldContext &context,
const IndexMask &mask,
ResourceScope & /*scope*/) const
{
if (const GeometryFieldContext *geometry_context = dynamic_cast<const GeometryFieldContext *>(
&context))
{
return this->get_varray_for_context(*geometry_context, mask);
}
if (const MeshFieldContext *mesh_context = dynamic_cast<const MeshFieldContext *>(&context)) {
return this->get_varray_for_context({mesh_context->mesh(), mesh_context->domain()}, mask);
}
if (const CurvesFieldContext *curve_context = dynamic_cast<const CurvesFieldContext *>(&context))
{
return this->get_varray_for_context({curve_context->curves(), curve_context->domain()}, mask);
}
if (const PointCloudFieldContext *point_context = dynamic_cast<const PointCloudFieldContext *>(
&context))
{
return this->get_varray_for_context({point_context->pointcloud()}, mask);
}
if (const InstancesFieldContext *instances_context = dynamic_cast<const InstancesFieldContext *>(
&context))
{
return this->get_varray_for_context({instances_context->instances()}, mask);
}
return {};
}
std::optional<eAttrDomain> GeometryFieldInput::preferred_domain(
const GeometryComponent & /*component*/) const
{
return std::nullopt;
}
GVArray MeshFieldInput::get_varray_for_context(const fn::FieldContext &context,
const IndexMask &mask,
ResourceScope & /*scope*/) const
{
if (const GeometryFieldContext *geometry_context = dynamic_cast<const GeometryFieldContext *>(
&context))
{
if (const Mesh *mesh = geometry_context->mesh()) {
return this->get_varray_for_context(*mesh, geometry_context->domain(), mask);
}
}
if (const MeshFieldContext *mesh_context = dynamic_cast<const MeshFieldContext *>(&context)) {
return this->get_varray_for_context(mesh_context->mesh(), mesh_context->domain(), mask);
}
return {};
}
std::optional<eAttrDomain> MeshFieldInput::preferred_domain(const Mesh & /*mesh*/) const
{
return std::nullopt;
}
GVArray CurvesFieldInput::get_varray_for_context(const fn::FieldContext &context,
const IndexMask &mask,
ResourceScope & /*scope*/) const
{
if (const GeometryFieldContext *geometry_context = dynamic_cast<const GeometryFieldContext *>(
&context))
{
if (const CurvesGeometry *curves = geometry_context->curves()) {
return this->get_varray_for_context(*curves, geometry_context->domain(), mask);
}
}
if (const CurvesFieldContext *curves_context = dynamic_cast<const CurvesFieldContext *>(
&context)) {
return this->get_varray_for_context(curves_context->curves(), curves_context->domain(), mask);
}
return {};
}
std::optional<eAttrDomain> CurvesFieldInput::preferred_domain(
const CurvesGeometry & /*curves*/) const
{
return std::nullopt;
}
GVArray PointCloudFieldInput::get_varray_for_context(const fn::FieldContext &context,
const IndexMask &mask,
ResourceScope & /*scope*/) const
{
if (const GeometryFieldContext *geometry_context = dynamic_cast<const GeometryFieldContext *>(
&context))
{
if (const PointCloud *pointcloud = geometry_context->pointcloud()) {
return this->get_varray_for_context(*pointcloud, mask);
}
}
if (const PointCloudFieldContext *point_context = dynamic_cast<const PointCloudFieldContext *>(
&context))
{
return this->get_varray_for_context(point_context->pointcloud(), mask);
}
return {};
}
GVArray InstancesFieldInput::get_varray_for_context(const fn::FieldContext &context,
const IndexMask &mask,
ResourceScope & /*scope*/) const
{
if (const GeometryFieldContext *geometry_context = dynamic_cast<const GeometryFieldContext *>(
&context))
{
if (const Instances *instances = geometry_context->instances()) {
return this->get_varray_for_context(*instances, mask);
}
}
if (const InstancesFieldContext *instances_context = dynamic_cast<const InstancesFieldContext *>(
&context))
{
return this->get_varray_for_context(instances_context->instances(), mask);
}
return {};
}
GVArray AttributeFieldInput::get_varray_for_context(const GeometryFieldContext &context,
const IndexMask & /*mask*/) const
{
const eCustomDataType data_type = cpp_type_to_custom_data_type(*type_);
if (auto attributes = context.attributes()) {
return *attributes->lookup(name_, context.domain(), data_type);
}
return {};
}
std::string AttributeFieldInput::socket_inspection_name() const
{
std::stringstream ss;
ss << '"' << name_ << '"' << TIP_(" attribute from geometry");
return ss.str();
}
uint64_t AttributeFieldInput::hash() const
{
return get_default_hash_2(name_, type_);
}
bool AttributeFieldInput::is_equal_to(const fn::FieldNode &other) const
{
if (const AttributeFieldInput *other_typed = dynamic_cast<const AttributeFieldInput *>(&other)) {
return name_ == other_typed->name_ && type_ == other_typed->type_;
}
return false;
}
std::optional<eAttrDomain> AttributeFieldInput::preferred_domain(
const GeometryComponent &component) const
{
const std::optional<AttributeAccessor> attributes = component.attributes();
if (!attributes.has_value()) {
return std::nullopt;
}
const std::optional<AttributeMetaData> meta_data = attributes->lookup_meta_data(name_);
if (!meta_data.has_value()) {
return std::nullopt;
}
return meta_data->domain;
}
static StringRef get_random_id_attribute_name(const eAttrDomain domain)
{
switch (domain) {
case ATTR_DOMAIN_POINT:
case ATTR_DOMAIN_INSTANCE:
return "id";
default:
return "";
}
}
GVArray IDAttributeFieldInput::get_varray_for_context(const GeometryFieldContext &context,
const IndexMask &mask) const
{
const StringRef name = get_random_id_attribute_name(context.domain());
if (auto attributes = context.attributes()) {
if (GVArray attribute = *attributes->lookup(name, context.domain(), CD_PROP_INT32)) {
return attribute;
}
}
/* Use the index as the fallback if no random ID attribute exists. */
return fn::IndexFieldInput::get_index_varray(mask);
}
std::string IDAttributeFieldInput::socket_inspection_name() const
{
return TIP_("ID / Index");
}
uint64_t IDAttributeFieldInput::hash() const
{
/* All random ID attribute inputs are the same within the same evaluation context. */
return 92386459827;
}
bool IDAttributeFieldInput::is_equal_to(const fn::FieldNode &other) const
{
/* All random ID attribute inputs are the same within the same evaluation context. */
return dynamic_cast<const IDAttributeFieldInput *>(&other) != nullptr;
}
GVArray AnonymousAttributeFieldInput::get_varray_for_context(const GeometryFieldContext &context,
const IndexMask & /*mask*/) const
{
const eCustomDataType data_type = cpp_type_to_custom_data_type(*type_);
return *context.attributes()->lookup(*anonymous_id_, context.domain(), data_type);
}
std::string AnonymousAttributeFieldInput::socket_inspection_name() const
{
std::stringstream ss;
ss << '"' << debug_name_ << '"' << TIP_(" from ") << producer_name_;
return ss.str();
}
uint64_t AnonymousAttributeFieldInput::hash() const
{
return get_default_hash_2(anonymous_id_.get(), type_);
}
bool AnonymousAttributeFieldInput::is_equal_to(const fn::FieldNode &other) const
{
if (const AnonymousAttributeFieldInput *other_typed =
dynamic_cast<const AnonymousAttributeFieldInput *>(&other))
{
return anonymous_id_.get() == other_typed->anonymous_id_.get() && type_ == other_typed->type_;
}
return false;
}
std::optional<eAttrDomain> AnonymousAttributeFieldInput::preferred_domain(
const GeometryComponent &component) const
{
const std::optional<AttributeAccessor> attributes = component.attributes();
if (!attributes.has_value()) {
return std::nullopt;
}
const std::optional<AttributeMetaData> meta_data = attributes->lookup_meta_data(*anonymous_id_);
if (!meta_data.has_value()) {
return std::nullopt;
}
return meta_data->domain;
}
} // namespace blender::bke
/* -------------------------------------------------------------------- */
/** \name Mesh and Curve Normals Field Input
* \{ */
namespace blender::bke {
GVArray NormalFieldInput::get_varray_for_context(const GeometryFieldContext &context,
const IndexMask &mask) const
{
if (const Mesh *mesh = context.mesh()) {
return mesh_normals_varray(*mesh, mask, context.domain());
}
if (const CurvesGeometry *curves = context.curves()) {
return curve_normals_varray(*curves, context.domain());
}
return {};
}
std::string NormalFieldInput::socket_inspection_name() const
{
return TIP_("Normal");
}
uint64_t NormalFieldInput::hash() const
{
return 213980475983;
}
bool NormalFieldInput::is_equal_to(const fn::FieldNode &other) const
{
return dynamic_cast<const NormalFieldInput *>(&other) != nullptr;
}
static std::optional<AttributeIDRef> try_get_field_direct_attribute_id(const fn::GField &any_field)
{
if (const auto *field = dynamic_cast<const AttributeFieldInput *>(&any_field.node())) {
return field->attribute_name();
}
if (const auto *field = dynamic_cast<const AnonymousAttributeFieldInput *>(&any_field.node())) {
return *field->anonymous_id();
}
return {};
}
static bool attribute_kind_matches(const AttributeMetaData meta_data,
const eAttrDomain domain,
const eCustomDataType data_type)
{
return meta_data.domain == domain && meta_data.data_type == data_type;
}
/**
* Some fields reference attributes directly. When the referenced attribute has the requested type
* and domain, use implicit sharing to avoid duplication when creating the captured attribute.
*/
static bool try_add_shared_field_attribute(MutableAttributeAccessor attributes,
const AttributeIDRef &id_to_create,
const eAttrDomain domain,
const fn::GField &field)
{
const std::optional<AttributeIDRef> field_id = try_get_field_direct_attribute_id(field);
if (!field_id) {
return false;
}
const std::optional<AttributeMetaData> meta_data = attributes.lookup_meta_data(*field_id);
if (!meta_data) {
return false;
}
const eCustomDataType data_type = bke::cpp_type_to_custom_data_type(field.cpp_type());
if (!attribute_kind_matches(*meta_data, domain, data_type)) {
/* Avoid costly domain and type interpolation, which would make sharing impossible. */
return false;
}
const GAttributeReader attribute = attributes.lookup(*field_id, domain, data_type);
if (!attribute.sharing_info || !attribute.varray.is_span()) {
return false;
}
const AttributeInitShared init(attribute.varray.get_internal_span().data(),
*attribute.sharing_info);
return attributes.add(id_to_create, domain, data_type, init);
}
bool try_capture_field_on_geometry(GeometryComponent &component,
const AttributeIDRef &attribute_id,
const eAttrDomain domain,
const fn::Field<bool> &selection,
const fn::GField &field)
{
MutableAttributeAccessor attributes = *component.attributes_for_write();
const int domain_size = attributes.domain_size(domain);
const CPPType &type = field.cpp_type();
const eCustomDataType data_type = bke::cpp_type_to_custom_data_type(type);
if (domain_size == 0) {
return attributes.add(attribute_id, domain, data_type, AttributeInitConstruct{});
}
const bke::GeometryFieldContext field_context{component, domain};
const IndexMask mask{IndexMask(domain_size)};
const bke::AttributeValidator validator = attributes.lookup_validator(attribute_id);
const std::optional<AttributeMetaData> meta_data = attributes.lookup_meta_data(attribute_id);
const bool attribute_matches = meta_data &&
attribute_kind_matches(*meta_data, domain, data_type);
/* We are writing to an attribute that exists already with the correct domain and type. */
if (attribute_matches) {
if (GSpanAttributeWriter dst_attribute = attributes.lookup_for_write_span(attribute_id)) {
const IndexMask mask{IndexMask(domain_size)};
const bke::GeometryFieldContext field_context{component, domain};
fn::FieldEvaluator evaluator{field_context, &mask};
evaluator.add(validator.validate_field_if_necessary(field));
evaluator.set_selection(selection);
evaluator.evaluate();
const IndexMask selection = evaluator.get_evaluated_selection_as_mask();
array_utils::copy(evaluator.get_evaluated(0), selection, dst_attribute.span);
dst_attribute.finish();
return true;
}
}
const bool selection_is_full = !selection.node().depends_on_input() &&
fn::evaluate_constant_field(selection);
if (!validator && selection_is_full) {
if (try_add_shared_field_attribute(attributes, attribute_id, domain, field)) {
return true;
}
}
/* Could avoid allocating a new buffer if:
* - The field does not depend on that attribute (we can't easily check for that yet). */
void *buffer = MEM_mallocN_aligned(type.size() * domain_size, type.alignment(), __func__);
if (!selection_is_full) {
type.value_initialize_n(buffer, domain_size);
}
fn::FieldEvaluator evaluator{field_context, &mask};
evaluator.add_with_destination(validator.validate_field_if_necessary(field),
GMutableSpan{type, buffer, domain_size});
evaluator.set_selection(selection);
evaluator.evaluate();
if (attribute_matches) {
if (GAttributeWriter attribute = attributes.lookup_for_write(attribute_id)) {
attribute.varray.set_all(buffer);
attribute.finish();
type.destruct_n(buffer, domain_size);
MEM_freeN(buffer);
return true;
}
}
attributes.remove(attribute_id);
if (attributes.add(attribute_id, domain, data_type, bke::AttributeInitMoveArray(buffer))) {
return true;
}
/* If the name corresponds to a builtin attribute, removing the attribute might fail if
* it's required, and adding the attribute might fail if the domain or type is incorrect. */
type.destruct_n(buffer, domain_size);
MEM_freeN(buffer);
return false;
}
bool try_capture_field_on_geometry(GeometryComponent &component,
const AttributeIDRef &attribute_id,
const eAttrDomain domain,
const fn::GField &field)
{
const fn::Field<bool> selection = fn::make_constant_field<bool>(true);
return try_capture_field_on_geometry(component, attribute_id, domain, selection, field);
}
std::optional<eAttrDomain> try_detect_field_domain(const GeometryComponent &component,
const fn::GField &field)
{
const GeometryComponentType component_type = component.type();
if (component_type == GEO_COMPONENT_TYPE_POINT_CLOUD) {
return ATTR_DOMAIN_POINT;
}
if (component_type == GEO_COMPONENT_TYPE_INSTANCES) {
return ATTR_DOMAIN_INSTANCE;
}
const std::shared_ptr<const fn::FieldInputs> &field_inputs = field.node().field_inputs();
if (!field_inputs) {
return std::nullopt;
}
std::optional<eAttrDomain> output_domain;
auto handle_domain = [&](const std::optional<eAttrDomain> domain) {
if (!domain.has_value()) {
return false;
}
if (output_domain.has_value()) {
if (*output_domain != *domain) {
return false;
}
return true;
}
output_domain = domain;
return true;
};
if (component_type == GEO_COMPONENT_TYPE_MESH) {
const MeshComponent &mesh_component = static_cast<const MeshComponent &>(component);
const Mesh *mesh = mesh_component.get_for_read();
if (mesh == nullptr) {
return std::nullopt;
}
for (const fn::FieldInput &field_input : field_inputs->deduplicated_nodes) {
if (const auto *geometry_field_input = dynamic_cast<const GeometryFieldInput *>(
&field_input)) {
if (!handle_domain(geometry_field_input->preferred_domain(component))) {
return std::nullopt;
}
}
else if (const auto *mesh_field_input = dynamic_cast<const MeshFieldInput *>(&field_input)) {
if (!handle_domain(mesh_field_input->preferred_domain(*mesh))) {
return std::nullopt;
}
}
else {
return std::nullopt;
}
}
}
if (component_type == GEO_COMPONENT_TYPE_CURVE) {
const CurveComponent &curve_component = static_cast<const CurveComponent &>(component);
const Curves *curves = curve_component.get_for_read();
if (curves == nullptr) {
return std::nullopt;
}
for (const fn::FieldInput &field_input : field_inputs->deduplicated_nodes) {
if (const auto *geometry_field_input = dynamic_cast<const GeometryFieldInput *>(
&field_input)) {
if (!handle_domain(geometry_field_input->preferred_domain(component))) {
return std::nullopt;
}
}
else if (const auto *curves_field_input = dynamic_cast<const CurvesFieldInput *>(
&field_input)) {
if (!handle_domain(curves_field_input->preferred_domain(curves->geometry.wrap()))) {
return std::nullopt;
}
}
else {
return std::nullopt;
}
}
}
return output_domain;
}
} // namespace blender::bke
/** \} */