This adds a new `DNA_sdna_type_ids.hh` header:
```cpp
namespace blender::dna {
/**
* Each DNA struct has an integer identifier which is unique within a specific
* Blender build, but not necessarily across different builds. The identifier
* can be used to index into `SDNA.structs`.
*/
template<typename T> int sdna_struct_id_get();
/**
* The maximum identifier that will be returned by #sdna_struct_id_get in this
* Blender build.
*/
int sdna_struct_id_get_max();
} // namespace blender::dna
```
The `sdna_struct_id_get` function is used as replacement of
`SDNA_TYPE_FROM_STRUCT` in all places except the DNA defaults system. The
defaults system is C code and therefore can't use the template. There is ongoing
work to replace the defaults system as well though: #134531.
Using this templated function has some benefits over the old approach:
* No need to rely on macros.
* Can use type inferencing in functions like `BLO_write_struct` which avoids
redundancy on the call site. E.g. `BLO_write_struct(writer, ActionStrip,
strip);` can become `BLO_write_struct(writer, strip);` which could even become
`writer.write_struct(strip);`. None of that is implemented as part of this
patch though.
* No need to include the generated `dna_type_offsets.h` file which contains a
huge enum.
Implementation wise, this is done using explicit template instantiations in a
new file generated by `makesdna.cc`: `dna_struct_ids.cc`. The generated file
looks like so:
```cpp
namespace blender::dna {
template<typename T> int sdna_struct_id_get();
int sdna_struct_id_get_max();
int sdna_struct_id_get_max() { return 951; }
}
struct IDPropertyUIData;
template<> int blender:🧬:sdna_struct_id_get<IDPropertyUIData>() { return 1; }
struct IDPropertyUIDataEnumItem;
template<> int blender:🧬:sdna_struct_id_get<IDPropertyUIDataEnumItem>() { return 2; }
```
I tried using static variables instead of separate functions, but I didn't
manage to link it properly. Not quite sure yet if that's an actual limitation or
if I was just missing something.
Pull Request: https://projects.blender.org/blender/blender/pulls/138706
We recently started using blenloader code in blendthumb to avoid having to
reimplement some parts of .blend file parsing. While this works, it has the side
effect that on Windows referencing blenloader code increased the binary size of
blendthumb from < 1MB to ~75 MB. That happens because this dependency drags
along lots of other code which effectively is unused, but the compiler is unable
to remove it.
There didn't seem to be a simple solution to make msvc optimize the unused code
away. This patch solves the issue by extracting the shared code into a separate
`blenloader_core` module which does not depend on the rest of Blender (except
blenlib). Therefore, using this new module in blendthumb does not drag along all
the other dependencies, bring its file size back down.
In the future, more code may be moved from blenloader to blenloader_core, but
for now I extracted these two headers:
* `BLO_core_bhead.hh`: Various `BHead` types and related parsing functions.
* `BLO_core_blend_header.hh`: Parsing of the header at the beginning of .blend
files.
Pull Request: https://projects.blender.org/blender/blender/pulls/138371
This is implements option 1 of #129309. It contains a few changes:
* Split `BHead8` into `SmallBHead8` and `LargeBHead8`. The latter is the new one
and uses `int64_t` for array sizes instead of just `int`. That applies to to
buffer size in bytes (`len`) and the array size (`nr`).
* The first .blend file header (the first few bytes of the file) are updated
according to #129309.
* Support reading files with that use `BHead4`, `SmallBHead8` and `LargeBHead8`.
* New option in the preferences that controls whether new files are written with
the older `SmallBHead8` or the new `LargeBHead8`. The new file format is
disabled by default. Potential unofficial 32 bit builds (#67184) always write
`BHead4`, but can read all types (in theory anyway, can't test it).
Note that there are other places in Blender that don't fully support arrays this
large. E.g. I noticed that the spreadsheet currently can't scroll all the way
down.
The experimental option can be removed once we are in the 5.0 branch, at which
point only 4.5 will be able to open the files saved with 5.0.
Co-authored-by: Bastien Montagne <bastien@blender.org>
Pull Request: https://projects.blender.org/blender/blender/pulls/129751
Implements a crash dialog for Windows.
The crash popup provides the following actions:
- Restart: reopen Blender from the last saved or auto-saved time
- Report a Bug: forward to Blender bug tracker
- View Crash Log: open the .txt file with the crash log
- Close: Closes without any further action
Pull Request: https://projects.blender.org/blender/blender/pulls/129974
The main issue of 'type-less' standard C allocations is that there is no check on
allocated type possible.
This is a serious source of annoyance (and crashes) when making some
low-level structs non-trivial, as tracking down all usages of these
structs in higher-level other structs and their allocation is... really
painful.
MEM_[cm]allocN<T> templates on the other hand do check that the
given type is trivial, at build time (static assert), which makes such issue...
trivial to catch.
NOTE: New code should strive to use MEM_new (i.e. allocation and
construction) as much as possible, even for trivial PoD types.
Pull Request: https://projects.blender.org/blender/blender/pulls/136121
Note: This commit is essentially non-behavioral change, expect in some
fairly rare edge cases.
This commit does a few things:
* Move the whole BKE_main_namemap code to modern C++.
* Split API calls to work with the global namemap, or the local ones.
* Simplify and make the code easier to follow and understand.
* Reduce 'default' memory usage by using growing BitVector for numeric
suffix management, instead of a fixed 1K items.
* Fix inconsistent handling of 'same base name and numeric suffix,
different name' issues (e.g. 'Foo.1' and 'Foo.001'), see
`re_create_equivalent_numeric_suffixes` new unittest.
* Fix completely broken handling of `global` namemaps. This was
(probably!) OK so far because of their currently very limited
use-cases.
It also adds a few minor improvements to existing behavior (essentially
in exotic rare edge cases):
* Names that get too long are now only shortened by one char at a time,
trying to modify the requested base name as little as possible.
* Names that are short, but for which all the manageable numeric suffixes
are already in use, are extended with an (increasing) number, instead
of being shortened.
This work also allowed to detect a few (apparently harmless?) bugs in
existing code, which have been fixed already in 4.4 and main, or in this
commit as well when they depend on changes in namemap code itself.
About performances: This commit introduces a minor slow-down. Some tests
heavily relying on this code (like `bl_id_management` and `blendkernel`
e.g.) get slightly slower (resp. about 1% and 5%). This seems to come
mostly from the added complexity to handle correctly multiple different
names with the same base and numeric suffix value ('Foo.1' and
'Foo.001', but also in the global namemap context where IDs from
different libraries can have the same name).
Pull Request: https://projects.blender.org/blender/blender/pulls/135199
Move `Library.runtime` to be a pointer, move the related
`LibraryRuntime` struct to `BKE_library.hh`. Similar to e.g.
Mesh.runtime, that pointer is expected to always be valid, and is
allocated at readtime or when creating a new Library ID.
Related smaller changes:
* Write code now uses standard ID writing codepath for Library IDs too.
* Runtime pointer is reset to nullptr before writing.
* Looking up a library by its absolute path is now handled through a
dedicated utils, `search_filepath_abs`, instead of using
`BLI_findstring`.
Pull Request: https://projects.blender.org/blender/blender/pulls/134188
This adds a debugging utility for developers that makes it easier to what's
stored in a .blend file. Some possible use cases:
* Figure out if some specific data has been written to the .blend file.
* Check the order of data in the .blend file.
* Work towards having less runtime dependent changes in .blend files (it's
easier to diff the textual output).
It is **not** a goal to provide a general human and machine readable form of
.blend files (like xml/json files). That would be way more involved.
The way to use this is to set `GENERATE_DEBUG_BLEND_FILE` to `1` at the top of
`writefile.cc`. Then, whenever Blender saves a .blend file, it will generate a
`.debug.txt` file next to it.
There is already `blend2json.py` which serves a similar purpose but is a
separate program that can be executed on the .blend file afterwards. With the
`--full-data` flag it outputs comparable data, but the output is a bit more
verbose and it needs an extra step that can be avoided by generating the
`.debug.txt` file immediately. For certain use cases,
`GENERATE_DEBUG_BLEND_FILE` can be more convenient. It's also much simpler to
add log additional data in that file that is produced during the blend-write
process.
Pull Request: https://projects.blender.org/blender/blender/pulls/133063
Avoids some binary noise in saved files, and will some day allow us to use runtime data on read without having to explicitly clean it up (unless we refactor this into an allocated pointer instead!).
Pull Request: https://projects.blender.org/blender/blender/pulls/132190
Previously, calling `clear()` on `Map`, `Set` or `VectorSet` would remove all
elements but did not free the already allocated capacity. This is fine in most
cases, but has very bad and non-obvious worst-case behavior as can be seen in
#131793. The issue is that having a huge hash table with only very few elements
is inefficient when having to iterate over it (e.g. when clearing).
There used to be a `clear_and_shrink()` method to avoid this worst-case
behavior. However, it's not obvious that this should be used to improve
performance.
This patch changes the behavior of `clear` to what `clear_and_shrink` did before
to avoid accidentally running in worst-case behavior. The old behavior is still
available with the name `clear_and_keep_capacity`. This is more efficient if
it's known that the hash-table is filled with approximately the same number of
elements or more again.
The main annoying aspect from an API perspective is that for `Vector`, the
default behavior of `clear` is and should stay to not free the memory. `Vector`
does not have the same worst-case behavior when there is a lot of unused
capacity (besides taking up memory), because the extra memory is never looked
at. `std::vector::clear` also does not free the memory, so that's the expected
behavior. While this patch introduces an inconsistency between `Vector` and
`Map/Set/VectorSet` with regards to freeing memory, it makes them more
consistent in that `clear` is the better default when reusing the data-structure
repeatedly.
I went over existing uses of `clear` to see if any of them should be changed to
`clear_and_keep_capacity`. None of them seemed to really benefit from that or
showed that it was impossible to get into the worst-case scenario. Therefore,
this patch slightly changes the behavior of these calls (only performance wise,
semantics are exactly the same).
Pull Request: https://projects.blender.org/blender/blender/pulls/131852
Give the `IDWALK_CB_…` enum an explicit name:
`LibraryForeachIDCallbackFlag`. This way the flags are type-safe, and
it's known where values come from. This is much preferred (at least by
me) to just having `int flags`.
Uses of `0` have been replaced with `IDWALK_CB_NOP` as that has the same
value and is of the right type.
One invalid use of `IDWALK_NOP` was detected by this change, and is
replaced by `IDWALK_CB_NOP`.
This change might be incomplete; I gave the enum a name, fixed the
compiler errors, and then also updated assignments like `int cb_flag =
cb_data->cb_flag`. I might have missed some assignments to `int` though.
No functional changes.
Pull Request: https://projects.blender.org/blender/blender/pulls/131865
Two commits that basically do the same thing for two `enum`s: give
them a name.
- the `IDWALK_…` enum → `LibraryForeachIDFlag`.
- the `IDWALK_CB_…` enum → `LibraryForeachIDCallbackFlag`.
This way the flags are type-safe, and it's known where values come
from. This is much preferred (at least by me) to just having `int
flags`.
Uses of `0` have been replaced with `IDWALK_NOP` and `IDWALK_CB_NOP`,
as those have the same value and are of the right type.
One invalid use of `IDWALK_NOP` was detected by this change, and is
replaced by `IDWALK_CB_NOP`. And another one in the opposite
direction.
This change might be incomplete; I gave the enum a name, fixed the
compiler errors, and then also updated assignments like `int cb_flag =
cb_data->cb_flag`. I might have missed some assignments to `int`
though.
No functional changes.
----------
I intend to land this PR as its two separate commits. I just put them in the same PR so the buildbot can handle them in one go, and we don't have a stack of highly relatled PRs.
In the future this could also apply to the `IDWALK_RET_…` enum. This one I left out, though, because a proper cleanup there would also have to include their ambiguity on whether they are bitflags (like the enums in this PR) or not. Their values and the code in `BKE_lib_query_foreachid_process()` implies they are bitflags, but in practice they are never or'ed together and just used as discrete values.
Pull Request: https://projects.blender.org/blender/blender/pulls/131803
This improves the `write_libraries` function in a couple of ways:
* No need to split the `bmain` which I find somewhat hard to reason about. I
think it's good if `bmain` is as read-only as possible in this write-context.
Instead a more explicit C++ data structure (`MultiValueMap<Library *, ID *>`)
is used. I think this was the only place in write-code that used
`blo_split_main`.
* Deduplication of the check that determines whether a placeholder should be
written for a specific ID.
* General cleanup to avoid unnecessarily deep nesting when iterating over IDs.
No functional changes are expected.
Pull Request: https://projects.blender.org/blender/blender/pulls/131385
Splitting the write-loop into stages makes it easier to understand, debug and time.
All the complex filtering in `gather_local_ids_to_write` is the same as before.
With this and some previous refactors, it's also much easier to have a clean `write_id`
function that should also work for embedded IDs later on.
Pull Request: https://projects.blender.org/blender/blender/pulls/130983
This is a leftover from an earlier version of overrides, but is effectively dead
code currently. It was used when there were proportional diffing capabilities.
However, Bastien mentioned that this would be implemented differently nowadays
anyway if it becomes necessary again.
This simplifies the write ID loop quite significantly.
Pull Request: https://projects.blender.org/blender/blender/pulls/130928
This replaces the existing C API of `BLO_Write_IDBuffer` with a C++ API. This
helps because:
* No need for explicit freeing.
* Can use more generic `DynamicStackBuffer` utility instead of having a separate
implementation of that.
Additionally, the API is changed so that a new `BLO_Write_IDBuffer` is created
for each `ID` instead of reusing the same for multiple IDs. This simplifies the
code quite a bit and allows for better use of the RAII pattern.
I expect the performance to be the same as before. In theory, there could be a
small speedup, because the `BLO_Write_IDBuffer` is not allocated separately
anymore, but that should be negligible.
No functional changes are expected.
Pull Request: https://projects.blender.org/blender/blender/pulls/130452
This adds a `write_bhead` utility function and also reduces the scope of
some `BHead` variables. The separate `write_bhead` function was quite
useful for #129751 and will likely be useful in a potential separate
implementation too.
Pull Request: https://projects.blender.org/blender/blender/pulls/130457
`writedata` used to align the written buffer size to a multiple of 4. This
causes multiple issues:
* Writes uninitialized data.
* Crash with ASAN due to a heap buffer overflow if the buffer is not any longer
than what is passed in.
* Modifies the length of the buffer which can't be undone when reading the
buffer again.
I don't know of any reason for this alignment here. I'd think that it doesn't
matter when writing to a file. If it would matter, then we should probably align
to at least 8 nowadays because that's the alignment of pointers. The original
reason for this alignment seems to be lost to history. It was already part of
the initial commit.
Pull Request: https://projects.blender.org/blender/blender/pulls/129821
Before efb511a76d this was not necessary, because the G_FILE_COMPRESS option
was not disabled when writing memfile undo steps. However, compression is
generally disabled when writing quit.blend or autosave files.
Previously, values for `ID.flag` and `ID.tag` used the prefixes `LIB_` and
`LIB_TAG` respectively. This was somewhat confusing because it's not really
related to libraries in general. This patch changes the prefix to `ID_FLAG_` and
`ID_TAG_`. This makes it more obvious what they correspond to, simplifying code.
Pull Request: https://projects.blender.org/blender/blender/pulls/125811
Previous namings in makesdna code was very confusing and inconsistent.
This commit unifies names accross the codebase as such:
- `struct` for the structs definitions data.
- `type` for the types definitions data.
- `member` for the struct members definitions data.
Structs and types definitions are not in synced for two reasons:
- types contains also definitions for basic types (int, float, etc.).
- types can be discovered before their struct definition (as members of
other structs).
This commit also groups all members of `SDNA` struct more logically (all
'structs' ones together, then all 'types' ones, then all 'struct
members' ones).
This commit should have no behavioral change at all.
Pull Request: https://projects.blender.org/blender/blender/pulls/125605
If a data-block referenced the same shared data more than once, it used to be
written multiple times. This worked out so far, because the written data is
always the same. So when reading the file, it doesn't matter which written
buffer Blender decides to read on. However, this makes the .blend file look
corrupt and leads to memory leaks on load (#125001).
Pull Request: https://projects.blender.org/blender/blender/pulls/125489
Blendfile uses the 'old (memory) address' of its data as 'uid' in the
blendfile. There should only be one block written for a given address
and a same ID (each ID define its own 'virtual address space').
This commit checks that this condition is met at wrtite time (except for
undo steps, for performances reasons).
Tooling part of the investigations on #125001.
File from #124777 (from 2.79.1) cause a temp bScreen to not be properly
tagged as such, which prevents its deletion when related window is
closed, and leaves a zero-user bScreen in Main.
While this should not happen, this does not seem trivial to track down,
and not an important enough issue to spend more time on it.
Also reported as #124857.
This is a workaround to allow user to keep working without loss of data
when an issue like #124049 happens.
This commit also expose again the `use_all_linked_data_direct` debug
option, no idea why that one was removed.
Regression from 435b6743fd, no usage of unlinkable ID (mainly
shapekeys...) should make them directly linked.
Note that this had no serious consequences, it was mainly printing annoying
error messages in release builds, and asserting in debug ones.
This was a consequence of the work done in #106321, where this specific
'active in UI' case was not identified and properly handled.
Now, consider most ID usages from UI (editors) as 'weak links', i.e.
keep a reference to these IDs even if they are only indirectly used.
Note that missing weak links will not create placeholders if the source
data is not found in the library anymore on load. they are just silently
dropped.
Pull Request: https://projects.blender.org/blender/blender/pulls/122207
A few ID types are considered as 'never unused' in Blender (UI related
ones, the Libraries and the Scenes). Local IDs of this type are always
considered as used, even if no other ID links to them.
This was previously fairly weekly defined and implemented (mainly in the
writefile code and the 'tag unused' libquery functions).
This commit formalize this characteristic of ID types by adding a new
`IDTYPE_FLAGS_NEVER_UNUSED` flag, and using this in the few places in
the code that handle unused IDs.
