On Windows/MSVC this gives a minor (~20%) speedup presumably due to a faster float/int formatter. On macOS (Xcode13), this gives a massive speedup, since snprintf that is in system libraries ends up spending almost all the time inside some locale-related mutex lock.
The actual exporter code becomes quite a bit smaller too, since it does not have to do any juggling to support std::string arguments, and the buffer handling code is smaller as well.
Windows (VS2022 release build, Ryzen 5950X 32 threads) timings:
- Blender 3.0 splash scene (2.4GB obj): 4.57s -> 3.86s
- Monkey subdivided level 6 (330MB obj): 1.10s -> 0.99s
macOS (Xcode 13 release build, Apple M1Max) timings:
- Blender 3.0 splash scene (2.4GB obj): 21.03s -> 5.52s
- Monkey subdivided level 6 (330MB obj): 3.28s -> 1.20s
Linux (ThreadRipper 3960X 48 threads) timings:
- Blender 3.0 splash scene (2.4GB obj): 10.10s -> 4.40s
- Monkey subdivided level 6 (330MB obj): 2.16s -> 1.37s
The produced obj/mtl files are identical to before.
Reviewed By: Howard Trickey, Dalai Felinto
Differential Revision: https://developer.blender.org/D13998
This change makes it possible to add implementation of common
C++ methods for DNA structures which helps ensuring unsafe
operations like shallow copy are done explicitly.
For example, creating a shallow copy used to be:
Object temp_object = *input_object;
In the C++ context it was seen like the temp_object is
properly decoupled from the input object, while in the
reality is it not. Now this code becomes:
Object temp_object = blender:🧬:shallow_copy(*input_object);
The copy and move constructor and assignment operators are
now explicitly disabled.
Other than a more explicit resource management this change
also solves a lot of warnings generated by the implicitly
defined copy constructors w.r.t dealing with deprecated fields.
These warnings were generated by Apple Clang when a shallow
object copy was created via implicitly defined copy constructor.
In order to enable C++ methods for DNA structures a newly added
macro `DNA_DEFINE_CXX_METHODS()` is to be used:
tpyedef struct Object {
DNA_DEFINE_CXX_METHODS(Object)
...
} Object;
For the shallow copy use `blender:🧬:shallow_copy()`.
The implementation of the memcpy is hidden via an internal DNA
function to avoid pulling `string.h` into every DNA header.
This means that the solution does not affect on the headers
dependencies.
---
Ideally `DNA_shallow_copy` would be defined in a more explicit
header, but don;t think we have a suitable one already. Maybe
we can introduce `DNA_access.h` ?
Differential Revision: https://developer.blender.org/D14427
This reverts commit 8c44793228.
Apparently, this generated a lot of warnings in GCC.
Didn't find a quick solution and is it not something I want to be
trading between (more quiet Clang in an expense of less quiet GCC).
Will re-iterate on the patch are re-commit it.
This change makes it possible to add implementation of common
C++ methods for DNA structures which helps ensuring unsafe
operations like shallow copy are done explicitly.
For example, creating a shallow copy used to be:
Object temp_object = *input_object;
In the C++ context it was seen like the temp_object is
properly decoupled from the input object, while in the
reality is it not. Now this code becomes:
Object temp_object = blender:🧬:shallow_copy(*input_object);
The copy and move constructor and assignment operators are
now explicitly disabled.
Other than a more explicit resource management this change
also solves a lot of warnings generated by the implicitly
defined copy constructors w.r.t dealing with deprecated fields.
These warnings were generated by Apple Clang when a shallow
object copy was created via implicitly defined copy constructor.
In order to enable C++ methods for DNA structures a newly added
macro `DNA_DEFINE_CXX_METHODS()` is to be used:
tpyedef struct Object {
DNA_DEFINE_CXX_METHODS(Object)
...
} Object;
For the shallow copy use `blender:🧬:shallow_copy()`.
The implementation of the memcpy is hidden via an internal DNA
function to avoid pulling `string.h` into every DNA header.
This means that the solution does not affect on the headers
dependencies.
---
Ideally `DNA_shallow_copy` would be defined in a more explicit
header, but don;t think we have a suitable one already. Maybe
we can introduce `DNA_access.h` ?
Differential Revision: https://developer.blender.org/D14427
Currently there is a "calc_face_normal" argument to mesh to bmesh
conversion, but vertex normals had always implicitly inherited whatever
dirty state the mesh input's vertex normals were in. Probably they were
most often assumed to not be dirty, but this was never really correct in
the general case.
Ever since the refactor to move vertex normals out of mesh vertices,
cfa53e0fbe, the copying logic has been explicit: copy the
normals when they are not dirty. But it turns out that more control is
needed, and sometimes normals should be calculated for the resulting
BMesh.
This commit adds an option to the conversion to calculate vertex
normals, true by default. In almost all places except the decimate
and edge split modifiers, I just copied the value of the
"calc_face_normals" argument.
Differential Revision: https://developer.blender.org/D14406
Old python exporter in 3.0 and earlier ordered faces by material,
but the new C++ exporter in 3.1+ did not, and was just writing them
in whatever is the order of the mesh data structure.
This mostly does not cause problems, except in some apps e.g.
Procreate -- for large enough meshes, this lack of
"order by material" (which ends up having more usemtl lines)
ends up creating more mesh subsets than necessary inside Procreate.
The change is not computationally heavy, e.g. exporting 6-level
subdivided Monkey mesh goes 1085ms -> 1105ms on my machine.
Reviewed By: @howardt
Differential Revision: https://developer.blender.org/D14368
This is patch D14349 from Aras Pranckevicius.
The logic in the code was _completely different_ from the documentation
and what the python exporter in 3.0 did. The new code assumed that
"export material groups" meant "append material name to the object name",
and was only ever kicking in when the "export object groups" option was
also checked. But the proper behavior (as in 3.0 exporter & the online docs),
is to emit g objectname_materialname before each usemtl line. Which is something entirely else.
This is patch D14347 from Aras Pranckevicius.
Instead of scaling "the scene" (i.e. transform vertices by object matrix,
then multiply by scale factor), it was instead first applying the scale
factor in local space, and then transforming by the object matrix.
This applies patch D14343 from Aras Pranckevicius, with a description:
The new 3.1+ OBJ exporter did not have correct logic when faced with
non-uniform & mirrored (negative on odd number of axes) object scale:
- Normals were not transformed correctly (should use inverse transpose of the matrix),
and were not normalized,
- Face order was not "flipped" when transform has negative scale on odd number of axes
(visible when using "face orientation" viewport overlay).
This commit renames enums related the "Curve" object type and ID type
to add `_LEGACY` to the end. The idea is to make our aspirations clearer
in the code and to avoid ambiguities between `CURVE` and `CURVES`.
Ref T95355
To summarize for the record, the plans are:
- In the short/medium term, replace the `Curve` object data type with
`Curves`
- In the longer term (no immediate plans), use a proper data block for
3D text and surfaces.
Differential Revision: https://developer.blender.org/D14114
This patch reverses the dependency between `BLI_math_vec_types.hh` and
`BLI_math_vector.hh`. Now the higher level `blender::math` functions
depend on the header that defines the types they work with, rather than
the other way around.
The initial goal was to allow defining an `enable_if` in the types header
and using it in the math header. But I also think this operations to types
dependency is more natural anyway.
This required changing the includes some files used from the type
header to the math implementation header. I took that change a bit
further removing the C vector math header from the C++ header;
I think that helps to make the transition between the two systems
clearer.
Differential Revision: https://developer.blender.org/D14112
Use a shorter/simpler license convention, stops the header taking so
much space.
Follow the SPDX license specification: https://spdx.org/licenses
- C/C++/objc/objc++
- Python
- Shell Scripts
- CMake, GNUmakefile
While most of the source tree has been included
- `./extern/` was left out.
- `./intern/cycles` & `./intern/atomic` are also excluded because they
use different header conventions.
doc/license/SPDX-license-identifiers.txt has been added to list SPDX all
used identifiers.
See P2788 for the script that automated these edits.
Reviewed By: brecht, mont29, sergey
Ref D14069
Fixes T95384. New exporter was missing a fix for T94516 that recently got applied to the python exporter.
Also changed the obj export tests code so that when save_failing_test_output is requested and MTL result is different from the golden expectation, it is saved as well, similar to how it's done for the OBJ file result.
This change from Aras further parallelizes wihin large meshes (the previous one
just parallelized over objects).
Some stats: on A Windows machine, AMD Ryzen (32 threads):
(one mesh) Monkey subdivided to level 6: 4.9s -> 1.2s (blender 3.1 was 6.3s; 3.0 was 49.4s).
(one mesh) "Rungholt" minecraft level: 8.5s -> 2.9s (3.1 was 10.5s; 3.0 was 73.7s).
(lots of meshes) Blender 3 splash: 6.2s -> 5.2s (3.1 was 48.9s; 3.0 was 392.3s).
On a Linux machine (Threadripper, 48 threads, writing to SSD):
Monkey - 5.08s -> 1.18s (4.2x speedup)
Rungholt - 9.52s -> 3.22s (2.95x speedup)
Blender 3 splash - 5.91s -> 4.61s (1.28x speedup)
For details see patch D14028.
Also fixed conflicts due to the change in file writing in the new obj exporter
in master, and fixed one of the tests that was added in master but not 3.1.
The new wavefront .obj exporter in 3.1 was producing slightly invalid parm line syntax (missing u), and was not setting first/last N params to zeroes and ones for curves with "endpoint" flag properly.
This is a patch from Aras Pranckevicius, D13927. See that patch for full
details. On Windows, the many small fprintfs were taking up a large amount
of time and significant speedup comes from using snprintf into chained buffers,
and writing them all out later.
On both Windows and Linux, parallelizing the processing by Object can also lead
to a significant increase in speed.
The 3.0 splash screen scene exports 8 times faster than the current C++ exporter
on a Windows machine with 32 threads, and 5.8 times faster on a Linux machine
with 48 threads.
There is admittedly more memory usage for this, but it is still using 25 times
less memory than the old python exporter on the 3.0 splash screen scene, so
this seems an acceptable tradeoff. If use cases come up for exporting obj files
that exceed the memory size of users, a flag could be added to not parallelize
and write the buffers out every so often.
Previously, the new obj exporter was only exporting per-vertex normals for faces
marked as "smooth". But a face can have custom normals, as soon as the normals
data layer exists. This change makes it follow the behavior of USD & Collada
exporters and the old Python one, which also export per-vertex normals as soon
as the layer is there. (From Patch D13957.)
The new OBJ exporter did not handle object instances.
The fix is to use a dependency graph iterator, asking for instances.
Unfortunately that iterator makes a temporary copy of instance objects
that does not persist past the iteration, but we need to save all the
objects and meshes to write later, so the Object has to be copied now.
This changed some unit tests. Even though the tests don't have instancing,
the iterator also picks up some Text objects as Mesh ones (which is a good
thing), resulting in two more objects in the all_objects.obj file output.
Some new obj exporter tests were disabled because the normals were different
in the last decimal place on different platforms.
The old python exporter deduped normals with their coordinates rounded to
four decimal places. This change does the same in the new exporter.
On one test, this produced a file 25% smaller and even ran 10% faster.
The switch to how normals are kept has led to tiny differences in
the normal output values on different platforms. Disabling the failing
tests while working on a solution to this problem.
As described in T91186, this commit moves mesh vertex normals into a
contiguous array of float vectors in a custom data layer, how face
normals are currently stored.
The main interface is documented in `BKE_mesh.h`. Vertex and face
normals are now calculated on-demand and cached, retrieved with an
"ensure" function. Since the logical state of a mesh is now "has
normals when necessary", they can be retrieved from a `const` mesh.
The goal is to use on-demand calculation for all derived data, but
leave room for eager calculation for performance purposes (modifier
evaluation is threaded, but viewport data generation is not).
**Benefits**
This moves us closer to a SoA approach rather than the current AoS
paradigm. Accessing a contiguous `float3` is much more efficient than
retrieving data from a larger struct. The memory requirements for
accessing only normals or vertex locations are smaller, and at the
cost of more memory usage for just normals, they now don't have to
be converted between float and short, which also simplifies code
In the future, the remaining items can be removed from `MVert`,
leaving only `float3`, which has similar benefits (see T93602).
Removing the combination of derived and original data makes it
conceptually simpler to only calculate normals when necessary.
This is especially important now that we have more opportunities
for temporary meshes in geometry nodes.
**Performance**
In addition to the theoretical future performance improvements by
making `MVert == float3`, I've done some basic performance testing
on this patch directly. The data is fairly rough, but it gives an idea
about where things stand generally.
- Mesh line primitive 4m Verts: 1.16x faster (36 -> 31 ms),
showing that accessing just `MVert` is now more efficient.
- Spring Splash Screen: 1.03-1.06 -> 1.06-1.11 FPS, a very slight
change that at least shows there is no regression.
- Sprite Fright Snail Smoosh: 3.30-3.40 -> 3.42-3.50 FPS, a small
but observable speedup.
- Set Position Node with Scaled Normal: 1.36x faster (53 -> 39 ms),
shows that using normals in geometry nodes is faster.
- Normal Calculation 1.6m Vert Cube: 1.19x faster (25 -> 21 ms),
shows that calculating normals is slightly faster now.
- File Size of 1.6m Vert Cube: 1.03x smaller (214.7 -> 208.4 MB),
Normals are not saved in files, which can help with large meshes.
As for memory usage, it may be slightly more in some cases, but
I didn't observe any difference in the production files I tested.
**Tests**
Some modifiers and cycles test results need to be updated with this
commit, for two reasons:
- The subdivision surface modifier is not responsible for calculating
normals anymore. In master, the modifier creates different normals
than the result of the `Mesh` normal calculation, so this is a bug
fix.
- There are small differences in the results of some modifiers that
use normals because they are not converted to and from `short`
anymore.
**Future improvements**
- Remove `ModifierTypeInfo::dependsOnNormals`. Code in each modifier
already retrieves normals if they are needed anyway.
- Copy normals as part of a better CoW system for attributes.
- Make more areas use lazy instead of eager normal calculation.
- Remove `BKE_mesh_normals_tag_dirty` in more places since that is
now the default state of a new mesh.
- Possibly apply a similar change to derived face corner normals.
Differential Revision: https://developer.blender.org/D12770
This patch implements the vector types (i.e:`float2`) by making heavy
usage of templating. All vector functions are now outside of the vector
classes (inside the `blender::math` namespace) and are not vector size
dependent for the most part.
In the ongoing effort to make shaders less GL centric, we are aiming
to share more code between GLSL and C++ to avoid code duplication.
####Motivations:
- We are aiming to share UBO and SSBO structures between GLSL and C++.
This means we will use many of the existing vector types and others
we currently don't have (uintX, intX). All these variations were
asking for many more code duplication.
- Deduplicate existing code which is duplicated for each vector size.
- We also want to share small functions. Which means that vector
functions should be static and not in the class namespace.
- Reduce friction to use these types in new projects due to their
incompleteness.
- The current state of the `BLI_(float|double|mpq)(2|3|4).hh` is a
bit of a let down. Most clases are incomplete, out of sync with each
others with different codestyles, and some functions that should be
static are not (i.e: `float3::reflect()`).
####Upsides:
- Still support `.x, .y, .z, .w` for readability.
- Compact, readable and easilly extendable.
- All of the vector functions are available for all the vectors types
and can be restricted to certain types. Also template specialization
let us define exception for special class (like mpq).
- With optimization ON, the compiler unroll the loops and performance
is the same.
####Downsides:
- Might impact debugability. Though I would arge that the bugs are
rarelly caused by the vector class itself (since the operations are
quite trivial) but by the type conversions.
- Might impact compile time. I did not saw a significant impact since
the usage is not really widespread.
- Functions needs to be rewritten to support arbitrary vector length.
For instance, one can't call `len_squared_v3v3` in
`math::length_squared()` and call it a day.
- Type cast does not work with the template version of the `math::`
vector functions. Meaning you need to manually cast `float *` and
`(float *)[3]` to `float3` for the function calls.
i.e: `math::distance_squared(float3(nearest.co), positions[i]);`
- Some parts might loose in readability:
`float3::dot(v1.normalized(), v2.normalized())`
becoming
`math::dot(math::normalize(v1), math::normalize(v2))`
But I propose, when appropriate, to use
`using namespace blender::math;` on function local or file scope to
increase readability.
`dot(normalize(v1), normalize(v2))`
####Consideration:
- Include back `.length()` method. It is quite handy and is more C++
oriented.
- I considered the GLM library as a candidate for replacement. It felt
like too much for what we need and would be difficult to extend / modify
to our needs.
- I used Macros to reduce code in operators declaration and potential
copy paste bugs. This could reduce debugability and could be reverted.
- This touches `delaunay_2d.cc` and the intersection code. I would like
to know @howardt opinion on the matter.
- The `noexcept` on the copy constructor of `mpq(2|3)` is being removed.
But according to @JacquesLucke it is not a real problem for now.
I would like to give a huge thanks to @JacquesLucke who helped during this
and pushed me to reduce the duplication further.
Reviewed By: brecht, sergey, JacquesLucke
Differential Revision: https://developer.blender.org/D13791
This patch implements the vector types (i.e:`float2`) by making heavy
usage of templating. All vector functions are now outside of the vector
classes (inside the `blender::math` namespace) and are not vector size
dependent for the most part.
In the ongoing effort to make shaders less GL centric, we are aiming
to share more code between GLSL and C++ to avoid code duplication.
####Motivations:
- We are aiming to share UBO and SSBO structures between GLSL and C++.
This means we will use many of the existing vector types and others
we currently don't have (uintX, intX). All these variations were
asking for many more code duplication.
- Deduplicate existing code which is duplicated for each vector size.
- We also want to share small functions. Which means that vector
functions should be static and not in the class namespace.
- Reduce friction to use these types in new projects due to their
incompleteness.
- The current state of the `BLI_(float|double|mpq)(2|3|4).hh` is a
bit of a let down. Most clases are incomplete, out of sync with each
others with different codestyles, and some functions that should be
static are not (i.e: `float3::reflect()`).
####Upsides:
- Still support `.x, .y, .z, .w` for readability.
- Compact, readable and easilly extendable.
- All of the vector functions are available for all the vectors types
and can be restricted to certain types. Also template specialization
let us define exception for special class (like mpq).
- With optimization ON, the compiler unroll the loops and performance
is the same.
####Downsides:
- Might impact debugability. Though I would arge that the bugs are
rarelly caused by the vector class itself (since the operations are
quite trivial) but by the type conversions.
- Might impact compile time. I did not saw a significant impact since
the usage is not really widespread.
- Functions needs to be rewritten to support arbitrary vector length.
For instance, one can't call `len_squared_v3v3` in
`math::length_squared()` and call it a day.
- Type cast does not work with the template version of the `math::`
vector functions. Meaning you need to manually cast `float *` and
`(float *)[3]` to `float3` for the function calls.
i.e: `math::distance_squared(float3(nearest.co), positions[i]);`
- Some parts might loose in readability:
`float3::dot(v1.normalized(), v2.normalized())`
becoming
`math::dot(math::normalize(v1), math::normalize(v2))`
But I propose, when appropriate, to use
`using namespace blender::math;` on function local or file scope to
increase readability.
`dot(normalize(v1), normalize(v2))`
####Consideration:
- Include back `.length()` method. It is quite handy and is more C++
oriented.
- I considered the GLM library as a candidate for replacement. It felt
like too much for what we need and would be difficult to extend / modify
to our needs.
- I used Macros to reduce code in operators declaration and potential
copy paste bugs. This could reduce debugability and could be reverted.
- This touches `delaunay_2d.cc` and the intersection code. I would like
to know @howardt opinion on the matter.
- The `noexcept` on the copy constructor of `mpq(2|3)` is being removed.
But according to @JacquesLucke it is not a real problem for now.
I would like to give a huge thanks to @JacquesLucke who helped during this
and pushed me to reduce the duplication further.
Reviewed By: brecht, sergey, JacquesLucke
Differential Revision: https://developer.blender.org/D13791
This patch implements the vector types (i.e:float2) by making heavy
usage of templating. All vector functions are now outside of the vector
classes (inside the blender::math namespace) and are not vector size
dependent for the most part.
In the ongoing effort to make shaders less GL centric, we are aiming
to share more code between GLSL and C++ to avoid code duplication.
Motivations:
- We are aiming to share UBO and SSBO structures between GLSL and C++.
This means we will use many of the existing vector types and others we
currently don't have (uintX, intX). All these variations were asking
for many more code duplication.
- Deduplicate existing code which is duplicated for each vector size.
- We also want to share small functions. Which means that vector functions
should be static and not in the class namespace.
- Reduce friction to use these types in new projects due to their
incompleteness.
- The current state of the BLI_(float|double|mpq)(2|3|4).hh is a bit of a
let down. Most clases are incomplete, out of sync with each others with
different codestyles, and some functions that should be static are not
(i.e: float3::reflect()).
Upsides:
- Still support .x, .y, .z, .w for readability.
- Compact, readable and easilly extendable.
- All of the vector functions are available for all the vectors types and
can be restricted to certain types. Also template specialization let us
define exception for special class (like mpq).
- With optimization ON, the compiler unroll the loops and performance is
the same.
Downsides:
- Might impact debugability. Though I would arge that the bugs are rarelly
caused by the vector class itself (since the operations are quite trivial)
but by the type conversions.
- Might impact compile time. I did not saw a significant impact since the
usage is not really widespread.
- Functions needs to be rewritten to support arbitrary vector length. For
instance, one can't call len_squared_v3v3 in math::length_squared() and
call it a day.
- Type cast does not work with the template version of the math:: vector
functions. Meaning you need to manually cast float * and (float *)[3] to
float3 for the function calls.
i.e: math::distance_squared(float3(nearest.co), positions[i]);
- Some parts might loose in readability:
float3::dot(v1.normalized(), v2.normalized())
becoming
math::dot(math::normalize(v1), math::normalize(v2))
But I propose, when appropriate, to use
using namespace blender::math; on function local or file scope to
increase readability. dot(normalize(v1), normalize(v2))
Consideration:
- Include back .length() method. It is quite handy and is more C++
oriented.
- I considered the GLM library as a candidate for replacement.
It felt like too much for what we need and would be difficult to
extend / modify to our needs.
- I used Macros to reduce code in operators declaration and potential
copy paste bugs. This could reduce debugability and could be reverted.
- This touches delaunay_2d.cc and the intersection code. I would like to
know @Howard Trickey (howardt) opinion on the matter.
- The noexcept on the copy constructor of mpq(2|3) is being removed.
But according to @Jacques Lucke (JacquesLucke) it is not a real problem
for now.
I would like to give a huge thanks to @Jacques Lucke (JacquesLucke) who
helped during this and pushed me to reduce the duplication further.
Reviewed By: brecht, sergey, JacquesLucke
Differential Revision: http://developer.blender.org/D13791
