It is unknown why/how this can happen, but there are some files out there
that have e.g. Objects flagged as embedded data... See e.g. the
`(Anim) Hero p23 for 2.blend` file from our cloud gallery
(https://cloud.blender.org/p/gallery/5b642e25bf419c1042056fc6).
Not much to be done, but add another checking pass at the end of
readfile process to fix these.
One of the consequences of the mistake in 3fcf535d2e (fixed in previous
commit), was that the more recent `LIB_EMBEDDED_DATA_LIB_OVERRIDE` ID
flag could be wrongly set in some nodetrees from pre-2.76 blendfiles.
This commit adds a check that embedded IDs flagged with
`LIB_EMBEDDED_DATA_LIB_OVERRIDE` are actually embedded IDs of a
liboverride.
Pull Request: https://projects.blender.org/blender/blender/pulls/118921
This adds implicit sharing support for the `MemFile` undo-step. This decreases memory
usage and increases performance.
Implicit sharing allows the undo system to take (shared) ownership of some data.
Previously, the data would always be serialized and compared to the previous undo-step.
So this turns an O(n) operation into O(1) (in terms of memory usage and time).
Read/write code that wants to make use of this has to use the new `BLO_read_shared`
and `BLO_write_shared` functions respectively. Those either make use of implicit-sharing
internally or do the "full" read/write based on a passed-in function. It seems possible to
use the same API in the future to store shared data to .blend files.
Improvements:
* Much faster undo step creation in many cases by avoiding the majority data copies
and equality checks. This fixes#98574. I found undo step creation and undo step
decoding to be 2-5 times faster in some demo files from the blender website and in
some production files from the Heist project.
* Reduced memory usage when there is large data in `bmain`. For example, when
loading the same highly subdivided mesh that I used in #106228 the memory usage
is 1.03 GB now (compared to 1.62 GB in `main` currently). The main remaining copy
of the data now is done by rendering code.
* Some significant performance improvements were also measured for the new grease
pencil type (#105540).
There is one main downside of using implicit-sharing as implemented here: `MemFile`
undo steps can't be written as .blend files anymore. This has a few consequences:
* Auto-save becomes slower (up to 3x), because it can't just save the previous undo step
anymore and does a normal save instead. This has been discussed in more detail here:
https://devtalk.blender.org/t/remove-support-for-saving-memfile-undo-steps-as-blend-files-proposal/33544
It would be nice to work towards asynchronous auto-save to alleviate this problem.
Some previous work has been done to reduce the impact of this change in 41b10424c7
and f0f304e240. This has been committed separately in efb511a76d.
* Writing `quit.blend` has to do a normal file save now. So it's a bit slower too, but it's
less of a problem in practice.
* The `USE_WRITE_CRASH_BLEND` functionality does not work anymore. It doesn't seem
to be used by anyone (removed in e90f5d03c4)
There are also benefits to not writing `MemFile` from undo steps to disk. It allows us to
more safely do undo-specific optimizations without risking corrupted .blend files. This
is especially useful when we want to preserve forward compatibility in some cases.
This requires converting data before writing the .blend files, but this conversion is not
necessary for undo steps. Trying to implement this kind of optimization in the past has
often lead to bugs (e.g. 43b37fbc93).
Another new problem is that it is harder to know the size of each undo step. Currently, a
heuristic is used to approximate the memory usage, but better solutions could be found
if necessary.
Pull Request: https://projects.blender.org/blender/blender/pulls/106903
When an asset is heavily modified, all production files having
liboverrides of it will still try to link all their known linked
reference IDs, leading to potentially thousands of not-really-useful
warnings about missing IDs in the console.
Now that liboverrides resync cleans up better these left-over data, it's
better to report missing linked data _after_ the liboverride resync
process.
Note that the original place can still report all effectively missing
linked data if needed, but this is now a logging info, so it won't be
displayed anywhere unless explicitely requested.
Pull Request: https://projects.blender.org/blender/blender/pulls/118577
The depsgraph CoW mechanism is a bit of a misnomer. It creates an
evaluated copy for data-blocks regardless of whether the copy will
actually be written to. The point is to have physical separation between
original and evaluated data. This is in contrast to the commonly used
performance improvement of keeping a user count and copying data
implicitly when it needs to be changed. In Blender code we call this
"implicit sharing" instead. Importantly, the dependency graph has no
idea about the _actual_ CoW behavior in Blender.
Renaming this functionality in the despgraph removes some of the
confusion that comes up when talking about this, and will hopefully
make the depsgraph less confusing to understand initially too. Wording
like "the evaluated copy" (as opposed to the original data-block) has
also become common anyway.
Pull Request: https://projects.blender.org/blender/blender/pulls/118338
Avoids having yet another `switch` case over all ID types.
Note that we could also add a string to the IDTypeInfo, but imho this is
a bit too limited in scope/interest to expose this outside of the readfile
code.
Pull Request: https://projects.blender.org/blender/blender/pulls/117958
I added a new BLO_userdef_default.h header to contain declarations of
two global variables that are still defined in C files. Use of designated
initializers for large structs make those files harder to change.
Arguably this is a better header for them anyway.
Pull Request: https://projects.blender.org/blender/blender/pulls/118015
`UUID` generally stands for "universally unique identifier". The session identifier that
we use is neither universally unique, nor does it follow the standard. Therefor, the term
"session uuid" is confusing and should be replaced.
In #116888 we briefly talked about a better name and ended up with "session uid".
The reason for "uid" instead of "id" is that the latter is a very overloaded term in Blender
already.
This patch changes all uses of "uuid" to "uid" where it's used in the context of a
"session uid". It's not always trivial to see whether a specific mention of "uuid" refers
to an actual uuid or something else. Therefore, I might have missed some renames.
I can't think of an automated way to differentiate the case.
BMesh also uses the term "uuid" sometimes in a the wrong context (e.g. `UUIDFaceStepItem`)
but there it also does not mean "session uid", so it's *not* changed by this patch.
Pull Request: https://projects.blender.org/blender/blender/pulls/117350
The term `PIL` stands for "platform independent library." It exists since the `Initial Revision`
commit from 2002. Nowadays, we generally just use the `BLI` (blenlib) prefix for such code
and the `PIL` prefix feels more confusing then useful. Therefore, this patch renames the
`PIL` to `BLI`.
Pull Request: https://projects.blender.org/blender/blender/pulls/117325
The previous commit introduced a new `RPT_()` macro to translate
strings which are not tooltips or regular interface elements, but
longer reports or statuses.
This commit uses the new macro to translate many strings all over the
UI.
Most of it is a simple replace from `TIP_()` or `IFACE_()` to
`RPT_()`, but there are some additional changes:
- A few translations inside `BKE_report()` are removed altogether
because they are already handled by the translation system.
- Messages inside `UI_but_disable()` are no longer translated
manually, but they are handled by a new regex in the translation
system.
Pull Request: https://projects.blender.org/blender/blender/pulls/116804
Pull Request: https://projects.blender.org/blender/blender/pulls/116804
Along with the 4.1 libraries upgrade, we are bumping the clang-format
version from 8-12 to 17. This affects quite a few files.
If not already the case, you may consider pointing your IDE to the
clang-format binary bundled with the Blender precompiled libraries.
This is a smaller rewrite/refactor of the VSE copy paste code to use the file copy buffer logic that is used in other places of Blender.
This makes Blender able to copy paste between Blender processes.
It can also paste successfully after closing and reopening Blender.
Other than that, the functionally should remain the exact same as the current copy paste operator with one exception: Scene strips does not retain their scene pointer when pasting into the same file.
This is to make it consistent with how it was before when copy pasting between .blend files.
(The scene data for the scene strips were never copied in when doing that)
Logic for pulling in or linking scenes into new or current files are purposely left for later when a proper proposal of how this would work in a nice fashion is done.
Now the strip data gets copied either fully or partially besides for scene strips. There only the script data gets copied and not the scene data.
If there is a audio/video data block of the same name as in the paste data, it will be reused to reduce potential duplication of data.
Pull Request: https://projects.blender.org/blender/blender/pulls/114703
While it may have been working from a practical PoV (not certain though,
since some bug would prevent clearing runtime data when writing embedded
Scene collection in current code), this is semantically wrong.
The owner of an embedded ID is a critical piece of information in
Blender data structure and ID management code. Having it written in
.blend files is also a potential good source of data for investigating
issues.
Further more, this handling of `owner` ID data is somewhat generic now
in ID management, so if this data should be considered runtime, then the
change should also be made in NodeTree and Key IDs.
This commit partially reverts 44dd3308a5, in the future I'd like to
be involved in the review of changes affecting ID management.
NOTE: fix for embedded collection runtime data not being cleared on
write will be committed separately.
Some left-over code from old refactoring/improvements of old addresses
remapping in library reading code was causing some wrong remapping
between different IDs.
The code was actually not doing what its comment was describing, since
it would cause a random remapping to a different new address than the
expected one:
Initial state, before removed buggy line:
`old_addr_1 -> new_addr_1`
`old_addr_2 -> new_addr_2`
End state, if `old_addr_2` == `new_addr_1`:
`old_addr_1 -> new_addr_2`
`old_addr_2 -> new_addr_2`
From the description of the removed line, that behavior was actually
already covered by adding the remapping rule in the line above, since
adding a remapping overwrite an existing one if needed, so the line
above would do (in case a placeholder ID existed before the actual ID
got read):
Initial state, before adding new remapping rule:
`old_addr_1 -> new_addr_placeholder_1`
End state, after adding remapping to newly read ID:
`old_addr_1 -> new_addr_1`
NOTE: Noticed thanks to rare failures of the liboverride tests on
buildbots. The issue was less than 1%-reproducible with a debug build
and ASAN, but had about 2-4% repducibility with release builds.
Pull Request: https://projects.blender.org/blender/blender/pulls/114784
Avoid confusion when checking for DNA members by using the names
in DNA headers ("Light" instead of "Lamp" for e.g.)
Internally SDNA stores names without aliases for compatibility.
The intention with aliasing DNA names is for `dna_rename_defs.h`
to be the only place where the non-aliased names needed to be referenced
however versioning checks also needed to reference the old names.
This wasn't obvious, causing mistakes in versioning checks (now fixed):
- SpaceOutliner::filter
- BrushGpencilSettings::hardness
- bGPDstroke::hardness
Details:
- Alias lookup tables are now ensured for BlendFileData::filesdna.
- DNA_struct_exists & DNA_struct_member_exists now use aliased names
in versioning code.
