This checkin will use OIIO to replace the image save/load code for BMP,
DDS, DPX, HDR, PNG, TGA, and TIFF.
This simplifies our build environment, reduces binary duplication,
removes large amounts of hard to maintain code, and fixes some bugs
along the way.
It should also help reduce rare differences between Blender and Cycles
which already uses OIIO for most situations. Or potentially makes them
easier to solve once discovered.
This is a continuation of the work for #101413
Pull Request: https://projects.blender.org/blender/blender/pulls/105785
Implements #95967.
Currently the `MPoly` struct is 12 bytes, and stores the index of a
face's first corner and the number of corners/verts/edges. Polygons
and corners are always created in order by Blender, meaning each
face's corners will be after the previous face's corners. We can take
advantage of this fact and eliminate the redundancy in mesh face
storage by only storing a single integer corner offset for each face.
The size of the face is then encoded by the offset of the next face.
The size of a single integer is 4 bytes, so this reduces memory
usage by 3 times.
The same method is used for `CurvesGeometry`, so Blender already has
an abstraction to simplify using these offsets called `OffsetIndices`.
This class is used to easily retrieve a range of corner indices for
each face. This also gives the opportunity for sharing some logic with
curves.
Another benefit of the change is that the offsets and sizes stored in
`MPoly` can no longer disagree with each other. Storing faces in the
order of their corners can simplify some code too.
Face/polygon variables now use the `IndexRange` type, which comes with
quite a few utilities that can simplify code.
Some:
- The offset integer array has to be one longer than the face count to
avoid a branch for every face, which means the data is no longer part
of the mesh's `CustomData`.
- We lose the ability to "reference" an original mesh's offset array
until more reusable CoW from #104478 is committed. That will be added
in a separate commit.
- Since they aren't part of `CustomData`, poly offsets often have to be
copied manually.
- To simplify using `OffsetIndices` in many places, some functions and
structs in headers were moved to only compile in C++.
- All meshes created by Blender use the same order for faces and face
corners, but just in case, meshes with mismatched order are fixed by
versioning code.
- `MeshPolygon.totloop` is no longer editable in RNA. This API break is
necessary here unfortunately. It should be worth it in 3.6, since
that's the best way to allow loading meshes from 4.0, which is
important for an LTS version.
Pull Request: https://projects.blender.org/blender/blender/pulls/105938
This change aimed to solve the following issues:
- Possible threading issue of two tests writing to the same
file, depending on how the ctest is invoked
- Test using the release directory, and potentially leaving
temp file behind on test failure, breaking code sign on
macOS.
Pull Request: https://projects.blender.org/blender/blender/pulls/106311
The goal is to solve confusion of the "All rights reserved" for licensing
code under an open-source license.
The phrase "All rights reserved" comes from a historical convention that
required this phrase for the copyright protection to apply. This convention
is no longer relevant.
However, even though the phrase has no meaning in establishing the copyright
it has not lost meaning in terms of licensing.
This change makes it so code under the Blender Foundation copyright does
not use "all rights reserved". This is also how the GPL license itself
states how to apply it to the source code:
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software ...
This change does not change copyright notice in cases when the copyright
is dual (BF and an author), or just an author of the code. It also does
mot change copyright which is inherited from NaN Holding BV as it needs
some further investigation about what is the proper way to handle it.
Similar to recent OBJ UV values merging commit (05a63e3705) - build
mapping of (vertex, UV) by going over the face loops directly, instead
of using BKE_mesh_uv_vert_map_get_vert and then having an additional
map on top. Provide the mapping into ready to use flat arrays, instead
of building a map and then building arrays out of that in a separate
pass.
While at it, avoid the extra cost of building all this complicated
mapping when we don't have or are not exporting UVs.
Timing tests on exporting several models into binary PLY file
(Win10, Ryzen 5950X):
- Suzanne subdivided to level 6 (2.1M verts): 0.93s -> 0.68s
- Rungholt Minecraft level (9.7M verts): 3.3s -> 2.3s
- Stanford Lucy 3D scan (14.0M verts, no UVs): 5.2s -> 1.5s
For example
```
OIIOOutputDriver::~OIIOOutputDriver()
{
}
```
becomes
```
OIIOOutputDriver::~OIIOOutputDriver() {}
```
Saves quite some vertical space, which is especially handy for
constructors.
Pull Request: https://projects.blender.org/blender/blender/pulls/105594
Previous code was using BKE_mesh_uv_vert_map_get_vert to somewhat
detect identical UV values on mesh vertices. But this was not
handling the case when separate mesh vertices still use the same UV
values (e.g. cube with all six faces mapped to whole image).
Replace usage of BKE_mesh_uv_vert_map_get_vert with a simple "unique UV
value" map, very similar to how OBJ normals are calculated for export.
Measurements on a somewhat extreme case: exporting "Rungholt" Minecraft
level from https://casual-effects.com/data time (Win10, Ryzen 5950X)
goes 2.9sec -> 2.2sec, and resulting file size 486MB -> 231MB. This
particular model has a lot of mesh faces mapping to the same places
in UV map.
On less extreme cases the timings are similar between old and new
code, with a tiny speedup in most tests I've tried.
UV attribute refactor in 6c774feb has changed the logic from "UV data does not exist" to "there's no active UV layer". The repro mesh has a UV layer, but not UV data due to the mesh being only a point cloud.
Pull Request: https://projects.blender.org/blender/blender/pulls/106185
Fixed a bug where the active color wasn't being set
on imported meshes, resulting in no colors displaying
in the viewport.
This bug has been in the code for a long time. However,
the colors have been displaying correctly until recently,
so this issue wasn't previously apparent.
Also, changed custom color data name from "displayColors"
to "displayColor", to match the actual USD primvar name.
(This was a typo in the original code.)
Note that pull request
https://projects.blender.org/blender/blender/pulls/104542
addresses other issues in the color import code (e.g.,
converting all color primvars and not just "displayColor",
avoiding hard-coding of attribute names, handling all
iterpolation types, etc.).
However, the current commit is meant as a short term fix
to a regression, where the "displayColor" attribute does
not render in the viewport at all, until the above pull
can be merged.
Implements #102359.
Split the `MLoop` struct into two separate integer arrays called
`corner_verts` and `corner_edges`, referring to the vertex each corner
is attached to and the next edge around the face at each corner. These
arrays can be sliced to give access to the edges or vertices in a face.
Then they are often referred to as "poly_verts" or "poly_edges".
The main benefits are halving the necessary memory bandwidth when only
one array is used and simplifications from using regular integer indices
instead of a special-purpose struct.
The commit also starts a renaming from "loop" to "corner" in mesh code.
Like the other mesh struct of array refactors, forward compatibility is
kept by writing files with the older format. This will be done until 4.0
to ease the transition process.
Looking at a small portion of the patch should give a good impression
for the rest of the changes. I tried to make the changes as small as
possible so it's easy to tell the correctness from the diff. Though I
found Blender developers have been very inventive over the last decade
when finding different ways to loop over the corners in a face.
For performance, nearly every piece of code that deals with `Mesh` is
slightly impacted. Any algorithm that is memory bottle-necked should
see an improvement. For example, here is a comparison of interpolating
a vertex float attribute to face corners (Ryzen 3700x):
**Before** (Average: 3.7 ms, Min: 3.4 ms)
```
threading::parallel_for(loops.index_range(), 4096, [&](IndexRange range) {
for (const int64_t i : range) {
dst[i] = src[loops[i].v];
}
});
```
**After** (Average: 2.9 ms, Min: 2.6 ms)
```
array_utils::gather(src, corner_verts, dst);
```
That's an improvement of 28% to the average timings, and it's also a
simplification, since an index-based routine can be used instead.
For more examples using the new arrays, see the design task.
Pull Request: https://projects.blender.org/blender/blender/pulls/104424
The input file contains a partial cube, with two regular faces, two
loose edges and one loose vertex. This checks whether the loose vertex
for example is included into the output when exporting with UVs
(loose verts do not have UVs in Blender, so they should get (0,0) UV
in PLY).
Follow connections when reading the varname attribute of a primvar
reader, and support both string and TfToken types for the varname.
A unit test is also provided.
Authored by Apple: Matt McLin
Pull Request: https://projects.blender.org/blender/blender/pulls/105508
1) There was a logic error in FileBuffer where when it was trying to
add a new 64kb chunk to hold output, it was adding an empty chunk,
but making sure we have space capacity to hold 64 thousand chunks.
So that was a bit of pointless juggling to get nothing good.
2) In UV_vertex_key, instead of trying to combine three members into
a hash value, badly, by doing some ad-hoc shifts and xors, use
get_default_hash_3 instead, which combines them way more properly.
Also avoid copying the whole hash map object.
On my windows box (Ryzen 5950X, VS2022), exporting Stanford Lucy 3D
scan:
- Binary: 13.4 -> 5.4 sec
- ASCII: 29.3 -> 14.6 sec
So basically 2x faster for two tiny changes.
Fixes:
* General:
- Was assuming that positions/normals/UVs are always `float`, and colors
are always `uchar`. But e.g. positions are sometimes doubles, or colors
are floats etc.
- Was assuming that `face` element is always just a single vertex indices
property. But some files have more arbitrary properties per-face.
* ASCII importer:
- Was assuming that file elements are always in this order: `vertex`,
`face`, `edge`. That is not necessarily the case.
- Was only handling space character as separator, but not other
whitespace e.g. tab.
- Was not checking for partially present properties (e.g. if just `nx` is
present but not `ny` and `nz`, it was still assuming whole normal is there).
* Binary importer:
- Was assuming that faces are always 1-byte list size and 4-byte indices.
- Was assuming that edge vertex indices are always 4-byte.
For the assumptions above, it often lead to either crashes, garbage data
or not detecting presence of a vertex component. E.g. binary importer was
just always reading 4 bytes for vertex indices, but if a file has 2 byte
indices then that would read too much, and then later on would start reading
garbage. ASCII importer was doing out-of-bounds reads into properties array
if some assumptions were not correct, etc.
Improvements:
- Some ply files use different property type names, e.g. `float32`,
`uint16` or `int8` instead of `float`, `ushort`, `char`; now that is
supported too.
- Handles `tristrips` element. Some files out there do not have `face` but
rather has a triangle strip, internally using -1 indices for strip restarts.
Performance:
While doing the above fixes/improvements, in order to fix some things it was
more convenient to also make them more performant :) Now it avoids splitting
each ASCII line into a vector of `std::string` objects, for example, or
reading a binary file in tiny chunks.
Generally this is now 2x-4x faster than the previous state of C++ importer.
The code has changed quite a bit to achieve both the fixes and performance:
- Instead of separate files for ASCII and Binary reading, they are now both
in `ply_import_data.cc/hh`. Reason is that all of the "find correct property
indices" logic is the same between both (and that bit was mostly missing
previously).
- Instead of using raw C++ `fstream` object and reading line by line (which
in turn reads one byte at a time) or reading field by field, now there's a
`ply_import_buffer.cc/hh` that reads in 64KB chunks and does any needed
logic for the ASCII/header part to properly split into lines. This avoids
all the mutex locking, locale lookups, C++ abstraction layers overhead
that C++ iostreams have when doing a ton of tiny reads.
Tests:
Extended test coverage with new files (comitted in svn revision 63274).
11 new "situations", in both ascii and binary forms. Additionally, now also
has a Big Endian binary file to test; BE codepath was completely untested
before.
Overall reworked the tests so that they don't do the whole "load dummy
blend file, import ply file on top, go over meshes, check them", but rather
do a simpler "run ply importer logic that fills in PlyData structure, check
that". The followup conversion to blender meshes is fairly trivial and the
tests were not really covering that in a useful way.
Pull Request: https://projects.blender.org/blender/blender/pulls/105842
This simplifies the usage of the API and is preparation for #104478.
The `CustomData_add_layer` and `CustomData_add_layer_named` now have corresponding
`*_with_data` functions that should be used when creating the layer from existing data.
Pull Request: https://projects.blender.org/blender/blender/pulls/105708
Refactoring mesh code, it has become clear that local cleanups and
simplifications are limited by the need to keep a C public API for
mesh functions. This change makes code more obvious and makes further
refactoring much easier.
- Add a new `BKE_mesh.hh` header for a C++ only mesh API
- Introduce a new `blender::bke::mesh` namespace, documented here:
https://wiki.blender.org/wiki/Source/Objects/Mesh#Namespaces
- Move some functions to the new namespace, cleaning up their arguments
- Move code to `Array` and `float3` where necessary to use the new API
- Define existing inline mesh data access functions to the new header
- Keep some C API functions where necessary because of RNA
- Move all C++ files to use the new header, which includes the old one
In the future it may make sense to split up `BKE_mesh.hh` more, but for
now keeping the same name as the existing header keeps things simple.
Pull Request: https://projects.blender.org/blender/blender/pulls/105416
The C4100 warning is related to unused formal parameters in functions.
Enabling it better aligns with "-Wunused-parameter" option in other
compilers.
While suppressing it with `__pragma(warning(suppress:4100))` is not the
same as using `__attribute__((__unused__))` in GCC or Clang, it is
still preferable to use it over completely hiding the warning.
This ensures consistent warning behavior across compilers and improves
code quality by addressing unused function parameters.
(Note that some warnings in Windows-specific code have already been
silenced in 7fcb262dfd)
Pull Request: https://projects.blender.org/blender/blender/pulls/105534
Currently the shade smooth status for mesh faces is stored as part of
`MPoly::flag`. As described in #95967, this moves that information
to a separate boolean attribute. It also flips its status, so the
attribute is now called `sharp_face`, which mirrors the existing
`sharp_edge` attribute. The attribute doesn't need to be allocated
when all faces are smooth. Forward compatibility is kept until
4.0 like the other mesh refactors.
This will reduce memory bandwidth requirements for some operations,
since the array of booleans uses 12 times less memory than `MPoly`.
It also allows faces to be stored more efficiently in the future, since
the flag is now unused. It's also possible to use generic functions to
process the values. For example, finding whether there is a sharp face
is just `sharp_faces.contains(true)`.
The `shade_smooth` attribute is no longer accessible with geometry nodes.
Since there were dedicated accessor nodes for that data, that shouldn't
be a problem. That's difficult to version automatically since the named
attribute nodes could be used in arbitrary combinations.
**Implementation notes:**
- The attribute and array variables in the code use the `sharp_faces`
term, to be consistent with the user-facing "sharp faces" wording,
and to avoid requiring many renames when #101689 is implemented.
- Cycles now accesses smooth face status with the generic attribute,
to avoid overhead.
- Changing the zero-value from "smooth" to "flat" takes some care to
make sure defaults are the same.
- Versioning for the edge mode extrude node is particularly complex.
New nodes are added by versioning to propagate the attribute in its
old inverted state.
- A lot of access is still done through the `CustomData` API rather
than the attribute API because of a few functions. That can be
cleaned up easily in the future.
- In the future we would benefit from a way to store attributes as a
single value for when all faces are sharp.
Pull Request: https://projects.blender.org/blender/blender/pulls/104422
At a random the bf_io_ply_tests would fail in one of the fixtures.
The root of the issue was that the exporter parameters were used
uninitialized, causing the mesh to be triangulated in some of the
runs and not be triangulated in other runs.
This change makes it so PLYExportParams is always zero-initialized,
so that this solves this issue, and that adding fields to it in the
future will not re-introduce the issue.
Pull Request: https://projects.blender.org/blender/blender/pulls/105537
Doing such writes leaves dangling file in the installation directory which
then is packaged as well. Not only this makes it so a random file gets
packaged for installation, it also makes notarization process to fail because
of not-so-clear reason.
The `ply_exporter_ply_data_test.SuzanneLoadPLYDataUV` fixture seems to be
unreliable and fails at random, even before this change. This makes it
hard to reliably get green light on all tests.
Pull Request #105504
The up_axis_update/forward_axis_update was the same logic between
the two, so factor that out.
Also use the same time reporting logic in PLY as in OBJ/USD/Alembic.
If the texture image path in the MTL is a "quoted" absolute path, the importer will fail to find the
file. It was only attempting to un-quote the path for the relative case. Now we attempt to un-quote
in all cases.
Pull Request #105478
If the texture image path in the MTL is a "quoted" absolute path, the importer will fail to find the
file. It was only attempting to un-quote the path for the relative case. Now we attempt to un-quote
in all cases.
Pull Request #105478
- Add missing braces for if statements
- Tweak variable naming to use snake case
- Use more common name for `MLoop`s of a face
- Use `std::move` when appending an array
- Use const for a few variable declarations
