The Python based importer had a special case handling of "no faces in
the whole file at all", where it ended up treating the whole file
as essentially a point-cloud-like object (just loose vertices, no
faces or edges). The new importer code was missing this special case.
Fixes T100017. Added gtest coverage that was failing without the fix.
Previously, when creating "very large" (tens-hundreds of thousands)
amounts of objects, the Blender code that was ensuring name
uniqueness was the bottleneck. That got recently addressed (D14162),
however now sorting of IDs by their names is the remaining bottleneck.
Name sorting code in Blender is optimized for the pattern where names
are inserted in already sorted order (i.e. objects expect to get added
near the end of the list). By doing this pre-sorting of objects
intended to get created by an importer (USD and OBJ, in this patch),
this sorting bottleneck can be largely removed, especially with very
high object counts.
Windows, Ryzen 5950X, import times:
- OBJ, splash screen scene (26k objects): 22.0s -> 20.7s
- USD, Disney Moana scene (250k objects): 585s -> 82.2s (10 minutes -> 1.5 minutes)
Reviewed By: Michael Kowalski, Howard Trickey
Differential Revision: https://developer.blender.org/D15506
The importer code was written under incorrect assumption that vertex
data (v, vn, vt commands etc.) are grouped by object, i.e. follow the
o command, and that each object has its own vertex data commands. This
is not the case -- all the vertex data in the whole OBJ file is
"global", with no relation to any objects/groups; it's just that the
faces belong to the object, and then they pull in any vertices they
like.
This patch fixes this incorrect assumption in the importer:
- Vertex data is now properly global; no need to track some sort of
"offsets" per object like it was doing before.
- For each object, face definitions track the minimum & maximum vertex
indices referenced by the object, and then all that vertex range is
created in the final Blender object. Note: it might be (unusual, but
possible) that an object does not reference a sequential range of
vertices, e.g. just a single face with vertex indices 1, 10, 100 --
the resulting Blender mesh will have all the 100 vertices (some
"loose" without belonging to a face). It should be possible to track
the used vertices exactly (e.g. with a vector set), but I haven't
done that for performance reasons.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D15410
Address the issue by re-working line continuation handling: stop
trying to parse sequences like "backslash, newline" (which is the
bug: it should also handle "backslash, possible whitespace, newline")
during parsing. Instead, fixup line continuations after reading chunks
of input file data - turn backslash and the following newline into
spaces. The rest of parsing code does not have to be aware of them
at all then.
Makes the file attached to T99536 load correctly now. Also will extend
one of the test files in subversion tests repo to contain backslashes
followed by newlines.
Currently, there are two attribute API. The first, defined in `BKE_attribute.h` is
accessible from RNA and C code. The second is implemented with `GeometryComponent`
and is only accessible in C++ code. The second is widely used, but only being
accessible through the `GeometrySet` API makes it awkward to use, and even impossible
for types that don't correspond directly to a geometry component like `CurvesGeometry`.
This patch adds a new attribute API, designed to replace the `GeometryComponent`
attribute API now, and to eventually replace or be the basis of the other one.
The basic idea is that there is an `AttributeAccessor` class that allows code to
interact with a set of attributes owned by some geometry. The accessor itself has
no ownership. `AttributeAccessor` is a simple type that can be passed around by
value. That makes it easy to return it from functions and to store it in containers.
For const-correctness, there is also a `MutableAttributeAccessor` that allows
changing individual and can add or remove attributes.
Currently, `AttributeAccessor` is composed of two pointers. The first is a pointer
to the owner of the attribute data. The second is a pointer to a struct with
function pointers, that is similar to a virtual function table. The functions
know how to access attributes on the owner.
The actual attribute access for geometries is still implemented with the `AttributeProvider`
pattern, which makes it easy to support different sources of attributes on a
geometry and simplifies dealing with built-in attributes.
There are different ways to get an attribute accessor for a geometry:
* `GeometryComponent.attributes()`
* `CurvesGeometry.attributes()`
* `bke::mesh_attributes(const Mesh &)`
* `bke::pointcloud_attributes(const PointCloud &)`
All of these also have a `_for_write` variant that returns a `MutabelAttributeAccessor`.
Differential Revision: https://developer.blender.org/D15280
As pointed out in a comment on T89421, if a MTL file contained
something like: `map_Ka -o 1 2.png` then it was parsed as having
offset `1 2` and the texture filename just a `.png`. Make it so that
mtl option numbers are parsed in a way where the number is only
accepted only if it's followed by whitespace.
Differential Revision: https://developer.blender.org/D15385
Fixes T97743: the import code was setting EEVEE blending mode whenever
a transparency texture was present (map_d), or when the materials
illum was saying "yo, transparency!". But if only the material's d
was below 1.0, it was not setting the blend mode, which is different
to user expectations.
Differential Revision: https://developer.blender.org/D15383
The importer parts that were doing assignment of materials to the
imported objects/meshes were essentially having a quadratic complexity
in terms of scene object count. For each material assigned to each
object, they were scanning the whole scene, checking which other
Objects use the same Mesh data, in order to resize their material
arrays to match the size.
Performance details (Windows, Ryzen 5950X):
- Import OBJ Blender 3.0 splash scene (24k objects): 43.0s -> 32.9s
- Import USD Disney Moana scene (260k objects): saves two hours
(~7400s). Note that later on this crashes when trying to render the
imported result; crashes in the same way/place both in master and
this patch.
Implementation details:
The importers were doing "scan the world" basically twice for each
object, for each material: once when creating a new material slot
(assigns an empty material), and then again when assigning the
material.
However, all these importers (USD, Alembic, OBJ) always create one
Object for one Mesh. So that whole quadratic complexity resulting
from "scan the world for possible other users of this obdata" is
completely not needed; it just never finds anything. So add a new
dedicated function BKE_object_material_assign_single_obdata that skips
the expensive part, but should only be used when the caller knows that
the obdata has exactly one user (the passed object).
Reviewed By: Bastien Montagne, Michael Kowalski
Differential Revision: https://developer.blender.org/D15145
The new OBJ/MTL importer was already handling case T89421
correctly, but there was no test coverage to prove it. Extend
the tests to parse various forms of "-o" and "-s" (one, two, three
numbers).
Removes the following macros for scene/render frame values:
- `CFRA`
- `SUBFRA`
- `SFRA`
- `EFRA`
These macros don't add much, other than saving a few characters when typing.
It's not immediately clear what they refer to, they just hide what they
actually access. Just be explicit and clear about that.
Plus these macros gave read and write access to the variables, so eyesores like
this would be done (eyesore because it looks like assigning to a constant):
```
CFRA = some_frame_nbr;
```
Reviewed By: sergey
Differential Revision: https://developer.blender.org/D15311
The old Python OBJ importer had a (somewhat confusingly named) "Keep
Vertex Order -> Poly Groups" option, that imported OBJ groups as
"vertex groups" on the resulting mesh. All vertices of any face were
assigned the vertex group, with a 1.0 weight.
The new C++ importer did not have this option. It was trying to do
something with vertex groups, but failing to actually achieve
anything :) -- the vertex groups were created on the wrong object
(later on overwritten by "nomain mesh to main mesh" operation);
vertex weights were set to 1.0/vertex_count, and each vertex was only
set to be in one group, even when it belongs to multiple faces from
different groups. End result was that to the user, vertex groups were
not visible/present at all (see T98874).
This patch adds the import option (named "Vertex Groups"), which is
off by default, and fixes the import code logic to actually do the
right thing. Tested on file from T98874; vertex groups are imported
just like with the Python importer.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D15200
The new OBJ importer is producing "sharp" edges on some meshes that
should be completely smooth. Only observed on UV-Sphere type meshes
so far (see T97820).
I'm not 100% sure what is the root cause, but my theory was that
maybe due to limited number of float digits that are printed for
vertex normals in the file, the normals that are read in are not
always exactly 1.0 length. And then the Blender's "set custom loop
normals" function (which expects normalized inputs) wrongly marks
some edges as sharp.
Adding explicit normalization for the normals that are read from the
file fixes the wrongly-sharp edges in test cases from T97820. I
have not observed measurable performance impact in importing large
models (e.g. 6-level subdivided Monkey) that contain vertex normals.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D15202
Some I/O code paths (Collada, OBJ) were using mat3_from_axis_conversion
followed by transpose_m3, instead of swapping the axis arguments
which achieves exactly the same result.
Reviewed By: Aras Pranckevicius
Differential Revision: https://developer.blender.org/D15158
OBJ vertex color related tests were not producing identical results
across various platforms, primarily due to sRGB<->Linear color space
conversions.
While D15193 has just made the color space conversion accuracy match
much closer between platforms, it's still not 100% the same.
This change reduces the amount of decimal places used for exporting
vertex colors, to 4 digits (down from 6). Vertex normals were
already always printed with 4 digits, and colors are conceptually
similar (usually 0..1 range etc.).
This makes the vertex color tests pass again, so re-enable them
after adjusting to 4 decimals expectations.
Some OBJ files out there (see T98782) have face definitions that
contain vertex normal indices, but the files themselves don't
contain any vertex normals. The code was doing a "hey, that's an
invalid index" and skipping these faces. But the old python importer
was silently ignoring these normal indices, so do the same here.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D15177
Adds support for vertex colors to OBJ I/O.
Importer:
- Supports both "xyzrgb" and "MRGB" vertex color formats.
- Whenever vertex color is present in the file for a model, it is
imported and a Color attribute is created (per-vertex, full float
color data type). Color coming from the file is assumed to be sRGB,
and is converted to linear upon import.
Exporter:
- Option to export the vertex colors. Defaults to "off", since not
all 3rd party software supports vertex colors.
- When the option is "on", if a mesh has a color attribute layer,
the active one is exported in "xyzrgb" form. If the mesh has
per-face-corner colors, they are averaged on the vertices.
Colors are converted from linear to sRGB upon export.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D15159
To match the existing Python .obj importer, and to make it easier for
the user to determine which object is which, use the filename for the
default object name instead of "New object".
Differential Revision: https://developer.blender.org/D15133
A new experimentatl STL importer, written in C++. Roughly 7-9x faster than the
Python based one.
Reviewed By: Aras Pranckevicius, Hans Goudey.
Differential Revision: https://developer.blender.org/D14941
The importer was not doing a notification that the scene has changed, so
the bottom status bar scene stats info was not updated right after the
new OBJ import.
Reviewed By: Julian Eisel
Differential Revision: https://developer.blender.org/D15015
The OBJ parser was primarily using StringRef for convenience, with
functions like "skip whitespace" or "parse a number" taking an input
stringref, representing an input line, and returning a new stringref,
representing the remainder of the line. This is convenient, but does
more work than strictly needed -- while parsing, only the "beginning"
of the line ever changes by moving forward; the end of the line
always stays the same. We can change the code to take a pair of
pointers (begin of line, end of line) as input, and make the
functions return the new begin of line pointer. This makes the return
value neatly fit into a processor register, which StringRef did not.
On Windows, this does result in non-trivial speedups in the actual
OBJ file parsing part, due to Windows calling convention where return
values larger than 64 bits are returned via memory. Does not
measurably affect performance on Mac/Linux, because the calling
convention there uses a pair of 64-bit registers to return a
StringRef.
End-to-end times of importing several test files, on Windows
(VS2022 build, Ryzen 5950X):
- Monkey subdivided to level 6, no normals (220MB file): 1.25s -> 0.85s
- Rungholt minecraft level (270MB file): 7.0s -> 5.8s
- Blender 3 splash scene (2.4GB file): 49.1s -> 45.5s
The full import process has a lot of other overhead besides actual
OBJ file parsing (mostly creating actual blender objects out of
parsed data). In pure parsing, in the monkey test scene above, the
parsing part goes 1.0s -> 0.6s.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14936
New OBJ exporter is missing "Path Mode" setting for exporting .mtl
files. The options that used to be available were: Auto, Absolute,
Relative, Match, Strip Path, Copy. All of them are important. The new
behavior (without any UI option to control it) curiously does not match
any of the previous setting. New behavior is like "Relative, but to the
source blender file, and not the destination export file".
Most of the previous logic was only present in Python based code
(bpy_extras.io_utils.path_reference and friends). The bulk of this
commit is porting that to C++.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14906
While possible extra whitespace after all OBJ/MTL keywords was properly
skipped, it was not done for the "f" (face definition) keyword.
While at it, also support indented keywords, i.e. extra whitespace at
the beginning of the line.
There's a tiny bit of performance drop while importing (e.g. importing
blender 3.0 splash scene: 53.38sec -> 54.21sec on my machine). But
correctness is more important.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14854
Fixes T97794 (which is a reintroduction of an older issue T67266 that
has been fixed in the python importer, but the fix was not in the C++
one). Some software produces OBJ files with mtllib statements like
mtllib "file name in quotes.mtl", and the new importer was not stripping
the quotes away.
While at it, I noticed that MTLParser constructor was taking a StringRef
and treating it as a zero-terminated string, which is not necessarily
the case. Fixed that by explicitly using a StringRefNull type.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14838
- Fix T97793: when a UV coordinate after vt is missing, use zero. While
at it, also use zeroes for positions & normals, since "maximum
possible float" is very likely to cause issues in imported meshes.
- Fix T97795: use 1.0 default if -bm value is missing, instead of zero.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14826
Fix several correctness issues where the new OBJ/MTL importer was not
producing the same results as the old one, mostly because the code for
some reason had slightly different logic. Fixes T97757:
- When .obj file tries to use a material that does not exist, the code
was continuing to use the previous material, instead of creating new
default one, as the previous importer did.
- Previous importer was always searching/parsing "foo.mtl" for a
"foo.obj" file, even if the file itself does not contain
"mtllib foo.mtl" statement. One file from T97757 repros happens to
depend on that, so resurrect that behavior.
- When IOR (Ni) or Alpha (d) are not specified in .mtl file, do not
wrongly set -1 values to the blender material.
- When base (Kd) or emissive (Ke) colors are not specified in the .mtl
file, do not set them on the blender material.
- Roughness and metallic values used by viewport shading were not set
onto blender material.
- The logic for when metallic was set to zero was incorrect; it should
be set to zero when "not using reflection", not when "mtl file does
not contain metallic".
- Do not produce a warning when illum value is not spelled out in .mtl
file, treat as default (1).
- Parse illum as a float just like python importer does, as to not
reintroduce part of T60135.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14822
The old python importer had a "if do_transparency, set blend_method to
BLEND" type of logic. This bit was missing in the new importer; it was
only setting the eevee blend method when a transparency texture was
present, but not in other cases of transparency (as driven by MTL
"illum" mode).
Reviewd By: Howard Trickey
Differential Revision: https://developer.blender.org/D14783
Even if available OBJ/MTL format documentations don't explicitly specify
which characters can possibly separate keywords & arguments, turns out
some files out there in the wild use TAB character after the line
keywords. Which is something the new 3.2 importer was not quite
expecting (T97417).
Fix this by factoring out a utility function that checks if line starts
with a keyword followed by any whitespace, and using that across the
importer. Also fix some other "possible whitespace around name-like
parts" of obj/mtl parser as pointed out by the repro files in T97417.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14782
Continued improvements to the new C++ based OBJ importer.
Performance: about 2x faster.
- Rungholt.obj (several meshes, 263MB file): Windows 12.7s -> 5.9s, Mac 7.7s -> 3.1s.
- Blender 3.0 splash (24k meshes, 2.4GB file): Windows 97.3s -> 53.6s, Mac 137.3s -> 80.0s.
- "Windows" is VS2022, AMD Ryzen 5950X (32 threads), "Mac" is Xcode/clang 13, M1Max (10 threads).
- Slightly reduced memory usage during import as well.
The performance gains are a combination of several things:
- Replacing `std::stof` / `std::stoi` with C++17 `from_chars`.
- Stop reading input file char-by-char using `std::getline`, and instead read in 64kb chunks, and parse from there (taking care of possibly handling lines split mid-way due to chunk boundaries).
- Removing abstractions for splitting a line by some char,
- Avoid tiny memory allocations: instead of storing a vector of polygon corners in each face, store all the corners in one big array, and per-face only store indices "where do corners start, and how many". Likewise, don't store full string names of material/group names for each face; only store indices into overall material/group names arrays.
- Stop always doing mesh validation, which is slow. Do it just like the Alembic importer does: only do validation if found some invalid faces during import, or if requested by the user via an import setting checkbox (which defaults to off).
- Stop doing "collection sync" for each object being added; instead do the collection sync right after creating all the objects.
Cleanup / Robustness:
This reworking of parser (see "removing abstractions" point above) means that all the functions that were in `parser_string_utils` file are gone, and replaced with different set of functions. However they are not OBJ specific, so as pointed out during review of the previous differential, they are now in `source/blender/io/common` library.
Added gtest coverage for said functions as well; something that was only indirectly covered by obj tests previously.
Rework of some bits of parsing made the parser actually better able to deal with invalid syntax. E.g. previously, if a face corner were a `/123` string, it would have incorrectly treated that as a vertex index (since it would get "hey that's one number" after splitting a string by a slash), instead of properly marking it as invalid syntax.
Added gtest coverage for .mtl parsing; something that was not covered by any tests at all previously.
Reviewed By: Howard Trickey
Differential Revision: https://developer.blender.org/D14586
