Fixes issues in importers written in C++ (T100737):
- Materials had one reference count too much. Affected Collada,
Alembic, USD, OBJ importers, looks like "since forever".
- Active material index was not properly set on imported meshes.
Regression since 3.3 (D15145). Affected Alembic, USD, OBJ. Note:
now it sets the first material as the active one, whereas
previously the last one was set as active. First one sounds more
"intuitive" to me.
Reviewed By: Bastien Montagne
Differential Revision: https://developer.blender.org/D15831
This patch moves material indices from the mesh `MPoly` struct to a
generic integer attribute. The builtin material index was already
exposed in geometry nodes, but this makes it a "proper" attribute
accessible with Python and visible in the "Attributes" panel.
The goals of the refactor are code simplification and memory and
performance improvements, mainly because the attribute doesn't have
to be stored and processed if there are no materials. However, until
4.0, material indices will still be read and written in the old
format, meaning there may be a temporary increase in memory usage.
Further notes:
* Completely removing the `MPoly.mat_nr` after 4.0 may require
changes to DNA or introducing a new `MPoly` type.
* Geometry nodes regression tests didn't look at material indices,
so the change reveals a bug in the realize instances node that I fixed.
* Access to material indices from the RNA `MeshPolygon` type is slower
with this patch. The `material_index` attribute can be used instead.
* Cycles is changed to read from the attribute instead.
* BMesh isn't changed in this patch. Theoretically it could be though,
to save 2 bytes per face when less than two materials are used.
* Eventually we could use a 16 bit integer attribute type instead.
Ref T95967
Differential Revision: https://developer.blender.org/D15675
When allocating new `CustomData` layers, often we do redundant
initialization of arrays. For example, it's common that values are
allocated, set to their default value, and then set to some other
value. This is wasteful, and it negates the benefits of optimizations
to the allocator like D15082. There are two reasons for this. The
first is array-of-structs storage that makes it annoying to initialize
values manually, and the second is confusing options in the Custom Data
API. This patch addresses the latter.
The `CustomData` "alloc type" options are rearranged. Now, besides
the options that use existing layers, there are two remaining:
* `CD_SET_DEFAULT` sets the default value.
* Usually zeroes, but for colors this is white (how it was before).
* Should be used when you add the layer but don't set all values.
* `CD_CONSTRUCT` refers to the "default construct" C++ term.
* Only necessary or defined for non-trivial types like vertex groups.
* Doesn't do anything for trivial types like `int` or `float3`.
* Should be used every other time, when all values will be set.
The attribute API's `AttributeInit` types are updated as well.
To update code, replace `CD_CALLOC` with `CD_SET_DEFAULT` and
`CD_DEFAULT` with `CD_CONSTRUCT`. This doesn't cause any functional
changes yet. Follow-up commits will change to avoid initializing
new layers where the correctness is clear.
Differential Revision: https://developer.blender.org/D15617
Added logic to the USD Preview Surface importer to
convert UDIM textures.
Reviewed by: Sybren and Jesse
Differential Revision: https://developer.blender.org/D15379
This is a partial fix for T90535.
USD allows binding materials generically as well as for a
specific purpose. I.e., purpose may be generic (unspecified)
or one of
- Full: truest representation of the scene
- Preview: lightweight material for preview
Curently, only generically bound materials, with unspecified
purpose (allPurpose), are imported. This issue is preventing
preview materials from being imported in the Alab scene.
This patch adds logic to attempt to fall back on importing
preview or full materials, in that order, if there is no
generic material bound to the mesh.
Reviewed by: Sybren
Differential Revision: https://developer.blender.org/D15352
Previously, when creating "very large" (tens-hundreds of thousands)
amounts of objects, the Blender code that was ensuring name
uniqueness was the bottleneck. That got recently addressed (D14162),
however now sorting of IDs by their names is the remaining bottleneck.
Name sorting code in Blender is optimized for the pattern where names
are inserted in already sorted order (i.e. objects expect to get added
near the end of the list). By doing this pre-sorting of objects
intended to get created by an importer (USD and OBJ, in this patch),
this sorting bottleneck can be largely removed, especially with very
high object counts.
Windows, Ryzen 5950X, import times:
- OBJ, splash screen scene (26k objects): 22.0s -> 20.7s
- USD, Disney Moana scene (250k objects): 585s -> 82.2s (10 minutes -> 1.5 minutes)
Reviewed By: Michael Kowalski, Howard Trickey
Differential Revision: https://developer.blender.org/D15506
The importer parts that were doing assignment of materials to the
imported objects/meshes were essentially having a quadratic complexity
in terms of scene object count. For each material assigned to each
object, they were scanning the whole scene, checking which other
Objects use the same Mesh data, in order to resize their material
arrays to match the size.
Performance details (Windows, Ryzen 5950X):
- Import OBJ Blender 3.0 splash scene (24k objects): 43.0s -> 32.9s
- Import USD Disney Moana scene (260k objects): saves two hours
(~7400s). Note that later on this crashes when trying to render the
imported result; crashes in the same way/place both in master and
this patch.
Implementation details:
The importers were doing "scan the world" basically twice for each
object, for each material: once when creating a new material slot
(assigns an empty material), and then again when assigning the
material.
However, all these importers (USD, Alembic, OBJ) always create one
Object for one Mesh. So that whole quadratic complexity resulting
from "scan the world for possible other users of this obdata" is
completely not needed; it just never finds anything. So add a new
dedicated function BKE_object_material_assign_single_obdata that skips
the expensive part, but should only be used when the caller knows that
the obdata has exactly one user (the passed object).
Reviewed By: Bastien Montagne, Michael Kowalski
Differential Revision: https://developer.blender.org/D15145
Many existing importers/exporters do log the time it takes to system
console (some others log more information too). In particular, OBJ
(C++ & python), STL (C++ & python), PLY, glTF2 all log the time it
takes. However, neither USD nor Alembic do. And also it's harder to
know the time it takes there from a profiler, since all the work
normally is done on a background job and is split between several
threads (so you can't just find some top-level function and see how
much time it took).
This change:
- Adds import/export time logging to USD & Alembic importer/exporter,
- In the time utility class (also used by OBJ & STL), improve the
output formatting: 1) print only one decimal digit, 2) for long
times, print seconds and also produce a hours:minutes:seconds form.
Reviewed By: Michael Kowalski, Kévin Dietrich
Differential Revision: https://developer.blender.org/D15170
Previous code was rebuilding "name to material" map for each object
being imported. Which means O(N*M) complexity (N=object count,
M=material count). There was already a TODO comment suggesting that
a single map that's maintained for the whole import would be enough.
This commit does exactly that.
While importing Moana USD scene (260k objects, 18k materials) this
saves about 6 minutes of import time.
Reviewed By: Bastien Montagne
Differential Revision: https://developer.blender.org/D15222
Previous code was doing N collection syncs when importing N objects
(essentially quadratic complexity in terms of object count). New
code avoids all the intermediate syncs by using
BKE_layer_collection_resync_forbid and
BKE_layer_collection_resync_allow, and then does one
BKE_main_collection_sync + BKE_main_collection_sync_remap for the
whole operation. The things done on the importer objects that are
dependent on the sync happening (marking them selected) are done in a
separate loop after the sync.
Timings: importing Moana USD scene (480k objects) on Windows, VS2022
Release build, AMD Ryzen 5950X: 12344sec -> 10979sec (saves 22 minutes).
Reviewed By: Bastien Montagne
Differential Revision: https://developer.blender.org/D15215
This is a partial fix for T90535.
Added Material Name Collision USD import menu option, to specify
the behavior when USD materials in different namespaces have the
same name.
The Material Name Collision menu options are
- Make Unique: Import each USD material as a unique Blender material.
- Reference Existing: If a material with the same name already
exists, reference that instead of importing.
Previously, the default behavior was to always keep the existing
material. This was causing an issue in the ALab scene, where
dozens of different USD materials all have the same name,
usdpreviewsurface1, so that only one instance of these materials
would be imported.
Reviewed by: Sybren
Differential Revision: https://developer.blender.org/D14869
No longer happens on the buildbot, but for users building with an older
Xcode, still need to avoid using value().
Differential Revision: https://developer.blender.org/D14883
Add support for volume (OpenVDB) USD export:
- Allows to export both static and animated volumes.
- Supports volumes that have OpenVDB data from files or are generated in
Blender with 'Mesh to Volume' modifier.
- For volumes that have generated data in Blender it also exports
corresponding .vdb files. Those files are saved in a new folder named
"volumes".
- Slightly changes the USD export UI panel. "Relative Texture Paths"
becomes "Relative Paths" (and has separate UI box) as the
functionality will now apply to both textures and volumes. Disabling
of this option due to "Materials" checkbox being turned off has been
removed.
Reviewed By: sybren, makowalski
Differential Revision: https://developer.blender.org/D14193
Manifest Task: T95407
The "PROP" in the name reflects its generic status, and removing
"LOOP" makes sense because it is no longer associated with just
mesh face corners. In general the goal is to remove extra semantic
meaning from the custom data types.
- Missing star prefix.
- Unnecessary indentation.
- Blank line after dot-points
(otherwise doxygen merges with the previous dot-point).
- Use back-slash for doxygen commands.
- Correct spelling.
Both the Alembic and USD libraries use double precision floating
point numbers internally to store time. However the Alembic I/O
code defaulted to floats even though Blender's Scene FPS, which is
generally used for look ups, is stored using a double type. Such
downcasts could lead to imprecise lookups, and would cause
compilation warnings (at least on MSVC).
This modifies the Alembic exporter and importer to make use of
doubles for the current scene time, and only downcasting to float
at the very last steps (e.g. for vertex interpolation). For the
importer, doubles are also used for computing interpolation weights,
as it is based on a time offset.
Although the USD code already used doubles internally, floats were used
at the C API level. Those were replaced as well.
Differential Revision: https://developer.blender.org/D13855
For 3.2 USD will be bumped to a newer version with some
slight API changes, however since we cannot simultaneously
land the libs for all platforms as well as these code changes,
we'll have to support both 21.02 and 21.11+ for at least a
short period of time making the code slightly more messy than
it could have been.
Differential Revision: https://developer.blender.org/D14184
Reviewed by: sybren
Fix provided by Piotr Makal in patch D14204.
This patch fixes volume grid duplication which was occurring during
importing USD files. This was caused by calling BKE_volume_grid_add
twice per grid (excluding 'density' grid) for the same Volume
object: (1) in USDVolumeReader::read_object_data and (2) later in
BKE_volume_load.
Differential Revision: https://developer.blender.org/D14204
This adds a const qualifier to some code path in the Alembic and USD
importers. More could be added elsewhere. This change is done as it will
be required when GeometrySets are supported and helps keeping diff noise
in the patch to a bare minimum.
Also fixes missing code to read/write/free/copy color management settings
in various places. This can't be set through the UI currently, but still
should be handled consistently.
Added call to ensure that the USD plugins are registered
when opening a USD cache archive. This is to avoid USD
load errors due to missing USD file format plugins when
opening blender files that contain USD transform cache
constraints and mesh sequence cache modifilers.
Fixes T94396
This commit renames enums related the "Curve" object type and ID type
to add `_LEGACY` to the end. The idea is to make our aspirations clearer
in the code and to avoid ambiguities between `CURVE` and `CURVES`.
Ref T95355
To summarize for the record, the plans are:
- In the short/medium term, replace the `Curve` object data type with
`Curves`
- In the longer term (no immediate plans), use a proper data block for
3D text and surfaces.
Differential Revision: https://developer.blender.org/D14114
Use a shorter/simpler license convention, stops the header taking so
much space.
Follow the SPDX license specification: https://spdx.org/licenses
- C/C++/objc/objc++
- Python
- Shell Scripts
- CMake, GNUmakefile
While most of the source tree has been included
- `./extern/` was left out.
- `./intern/cycles` & `./intern/atomic` are also excluded because they
use different header conventions.
doc/license/SPDX-license-identifiers.txt has been added to list SPDX all
used identifiers.
See P2788 for the script that automated these edits.
Reviewed By: brecht, mont29, sergey
Ref D14069
Changes to recent addition: c85c52f2ce.
Having both BLI_paths_equal and BLI_path_cmp made it ambiguous
which should be used, as `BLI_paths_equal` wasn't the equivalent to
`BLI_path_cmp(..) == 0` as it is for string equals macro `STREQ(..)`.
It's also a more specialized function which is not used for path
comparison throughout Blender's internal path handling logic.
Instead rename this `BLI_path_cmp_normalized` and return the result of
`BLI_path_cmp` to make it clear paths are modified before comparison.
Also add comments about the conventions for Blender's path comparison
as well as a possible equivalent to Python's `os.path.samefile`
for checking if two paths point to the same location on the file-system.
Add `USD Preview Surface From Nodes` export option, to convert a
Principled BSDF material node network to an approximate USD Preview
Surface shader representation. If this option is disabled, the original
material export behavior is maintained, where viewport setting are saved
to the Preview Surface shader.
Also added the following options for texture export.
- `Export Textures`: If converting Preview Surface, export textures
referenced by shader nodes to a 'textures' directory which is a
sibling of the USD file.
- `Overwrite Textures`: Allow overwriting existing texture files when
exporting textures (this option is off by default).
- `Relative Texture Paths`: Make texture asset paths relative to the
USD.
The entry point for the new functionality is
`create_usd_preview_surface_material()`, called from
`USDAbstractWriter::ensure_usd_material()`. The material conversion
currently handles a small subset of Blender shading nodes,
`BSDF_DIFFUSE`, `BSDF_PRINCIPLED`, `TEX_IMAGE` and `UVMAP`.
Texture export is handled by copying texture files from their original
location to a `textures` folder in the same directory as the USD.
In-memory and packed textures are saved directly to the textures folder.
This patch is based, in part, on code in Tangent Animation's USD
exporter branch.
Reviewed By: sybren, HooglyBoogly
Differential Revision: https://developer.blender.org/D13647
This adds vertex creasing support for OpenSubDiv for modeling, rendering,
Alembic and USD I/O.
For modeling, vertex creasing follows the edge creasing implementation with an
operator accessible through the Vertex menu in Edit Mode, and some parameter in
the properties panel. The option in the Subsurf and Multires to use edge
creasing also affects vertex creasing.
The vertex crease data is stored as a CustomData layer, unlike edge creases
which for now are stored in `MEdge`, but will in the future also be moved to
a `CustomData` layer. See comments for details on the difference in behavior
for the `CD_CREASE` layer between egdes and vertices.
For Cycles this adds sockets on the Mesh node to hold data about which vertices
are creased (one socket for the indices, one for the weigths).
Viewport rendering of vertex creasing reuses the same color scheme as for edges
and creased vertices are drawn bigger than uncreased vertices.
For Alembic and USD, vertex crease support follows the edge crease
implementation, they are always read, but only exported if a `Subsurf` modifier
is present on the Mesh.
Reviewed By: brecht, fclem, sergey, sybren, campbellbarton
Differential Revision: https://developer.blender.org/D10145
As described in T91186, this commit moves mesh vertex normals into a
contiguous array of float vectors in a custom data layer, how face
normals are currently stored.
The main interface is documented in `BKE_mesh.h`. Vertex and face
normals are now calculated on-demand and cached, retrieved with an
"ensure" function. Since the logical state of a mesh is now "has
normals when necessary", they can be retrieved from a `const` mesh.
The goal is to use on-demand calculation for all derived data, but
leave room for eager calculation for performance purposes (modifier
evaluation is threaded, but viewport data generation is not).
**Benefits**
This moves us closer to a SoA approach rather than the current AoS
paradigm. Accessing a contiguous `float3` is much more efficient than
retrieving data from a larger struct. The memory requirements for
accessing only normals or vertex locations are smaller, and at the
cost of more memory usage for just normals, they now don't have to
be converted between float and short, which also simplifies code
In the future, the remaining items can be removed from `MVert`,
leaving only `float3`, which has similar benefits (see T93602).
Removing the combination of derived and original data makes it
conceptually simpler to only calculate normals when necessary.
This is especially important now that we have more opportunities
for temporary meshes in geometry nodes.
**Performance**
In addition to the theoretical future performance improvements by
making `MVert == float3`, I've done some basic performance testing
on this patch directly. The data is fairly rough, but it gives an idea
about where things stand generally.
- Mesh line primitive 4m Verts: 1.16x faster (36 -> 31 ms),
showing that accessing just `MVert` is now more efficient.
- Spring Splash Screen: 1.03-1.06 -> 1.06-1.11 FPS, a very slight
change that at least shows there is no regression.
- Sprite Fright Snail Smoosh: 3.30-3.40 -> 3.42-3.50 FPS, a small
but observable speedup.
- Set Position Node with Scaled Normal: 1.36x faster (53 -> 39 ms),
shows that using normals in geometry nodes is faster.
- Normal Calculation 1.6m Vert Cube: 1.19x faster (25 -> 21 ms),
shows that calculating normals is slightly faster now.
- File Size of 1.6m Vert Cube: 1.03x smaller (214.7 -> 208.4 MB),
Normals are not saved in files, which can help with large meshes.
As for memory usage, it may be slightly more in some cases, but
I didn't observe any difference in the production files I tested.
**Tests**
Some modifiers and cycles test results need to be updated with this
commit, for two reasons:
- The subdivision surface modifier is not responsible for calculating
normals anymore. In master, the modifier creates different normals
than the result of the `Mesh` normal calculation, so this is a bug
fix.
- There are small differences in the results of some modifiers that
use normals because they are not converted to and from `short`
anymore.
**Future improvements**
- Remove `ModifierTypeInfo::dependsOnNormals`. Code in each modifier
already retrieves normals if they are needed anyway.
- Copy normals as part of a better CoW system for attributes.
- Make more areas use lazy instead of eager normal calculation.
- Remove `BKE_mesh_normals_tag_dirty` in more places since that is
now the default state of a new mesh.
- Possibly apply a similar change to derived face corner normals.
Differential Revision: https://developer.blender.org/D12770
Remove `const` from pass-by-value parameters in function declarations.
The variables passed as parameters can never be modified by the function
anyway, so declaring them as `const` is meaningless. Having the
declaration there could confuse, especially as it suggests it does have
a meaning, training people to write meaningless code.
Goals of this refactor:
* More unified approach to updating everything that needs to be updated
after a change in a node tree.
* The updates should happen in the correct order and quadratic or worse
algorithms should be avoided.
* Improve detection of changes to the output to avoid tagging the depsgraph
when it's not necessary.
* Move towards a more declarative style of defining nodes by having a
more centralized update procedure.
The refactor consists of two main parts:
* Node tree tagging and update refactor.
* Generally, when changes are done to a node tree, it is tagged dirty
until a global update function is called that updates everything in
the correct order.
* The tagging is more fine-grained compared to before, to allow for more
precise depsgraph update tagging.
* Depsgraph changes.
* The shading specific depsgraph node for node trees as been removed.
* Instead, there is a new `NTREE_OUTPUT` depsgrap node, which is only
tagged when the output of the node tree changed (e.g. the Group Output
or Material Output node).
* The copy-on-write relation from node trees to the data block they are
embedded in is now non-flushing. This avoids e.g. triggering a material
update after the shader node tree changed in unrelated ways. Instead
the material has a flushing relation to the new `NTREE_OUTPUT` node now.
* The depsgraph no longer reports data block changes through to cycles
through `Depsgraph.updates` when only the node tree changed in ways
that do not affect the output.
Avoiding unnecessary updates seems to work well for geometry nodes and cycles.
The situation is a bit worse when there are drivers on the node tree, but that
could potentially be improved separately in the future.
Avoiding updates in eevee and the compositor is more tricky, but also less urgent.
* Eevee updates are triggered by calling `DRW_notify_view_update` in
`ED_render_view3d_update` indirectly from `DEG_editors_update`.
* Compositor updates are triggered by `ED_node_composite_job` in `node_area_refresh`.
This is triggered by calling `ED_area_tag_refresh` in `node_area_listener`.
Removing updates always has the risk of breaking some dependency that no
one was aware of. It's not unlikely that this will happen here as well. Adding
back missing updates should be quite a bit easier than getting rid of
unnecessary updates though.
Differential Revision: https://developer.blender.org/D13246
