Before efb511a76d this was not necessary, because the G_FILE_COMPRESS option
was not disabled when writing memfile undo steps. However, compression is
generally disabled when writing quit.blend or autosave files.
Previously, values for `ID.flag` and `ID.tag` used the prefixes `LIB_` and
`LIB_TAG` respectively. This was somewhat confusing because it's not really
related to libraries in general. This patch changes the prefix to `ID_FLAG_` and
`ID_TAG_`. This makes it more obvious what they correspond to, simplifying code.
Pull Request: https://projects.blender.org/blender/blender/pulls/125811
Previous namings in makesdna code was very confusing and inconsistent.
This commit unifies names accross the codebase as such:
- `struct` for the structs definitions data.
- `type` for the types definitions data.
- `member` for the struct members definitions data.
Structs and types definitions are not in synced for two reasons:
- types contains also definitions for basic types (int, float, etc.).
- types can be discovered before their struct definition (as members of
other structs).
This commit also groups all members of `SDNA` struct more logically (all
'structs' ones together, then all 'types' ones, then all 'struct
members' ones).
This commit should have no behavioral change at all.
Pull Request: https://projects.blender.org/blender/blender/pulls/125605
If a data-block referenced the same shared data more than once, it used to be
written multiple times. This worked out so far, because the written data is
always the same. So when reading the file, it doesn't matter which written
buffer Blender decides to read on. However, this makes the .blend file look
corrupt and leads to memory leaks on load (#125001).
Pull Request: https://projects.blender.org/blender/blender/pulls/125489
Blendfile uses the 'old (memory) address' of its data as 'uid' in the
blendfile. There should only be one block written for a given address
and a same ID (each ID define its own 'virtual address space').
This commit checks that this condition is met at wrtite time (except for
undo steps, for performances reasons).
Tooling part of the investigations on #125001.
File from #124777 (from 2.79.1) cause a temp bScreen to not be properly
tagged as such, which prevents its deletion when related window is
closed, and leaves a zero-user bScreen in Main.
While this should not happen, this does not seem trivial to track down,
and not an important enough issue to spend more time on it.
Also reported as #124857.
This is a workaround to allow user to keep working without loss of data
when an issue like #124049 happens.
This commit also expose again the `use_all_linked_data_direct` debug
option, no idea why that one was removed.
Regression from 435b6743fd, no usage of unlinkable ID (mainly
shapekeys...) should make them directly linked.
Note that this had no serious consequences, it was mainly printing annoying
error messages in release builds, and asserting in debug ones.
This was a consequence of the work done in #106321, where this specific
'active in UI' case was not identified and properly handled.
Now, consider most ID usages from UI (editors) as 'weak links', i.e.
keep a reference to these IDs even if they are only indirectly used.
Note that missing weak links will not create placeholders if the source
data is not found in the library anymore on load. they are just silently
dropped.
Pull Request: https://projects.blender.org/blender/blender/pulls/122207
A few ID types are considered as 'never unused' in Blender (UI related
ones, the Libraries and the Scenes). Local IDs of this type are always
considered as used, even if no other ID links to them.
This was previously fairly weekly defined and implemented (mainly in the
writefile code and the 'tag unused' libquery functions).
This commit formalize this characteristic of ID types by adding a new
`IDTYPE_FLAGS_NEVER_UNUSED` flag, and using this in the few places in
the code that handle unused IDs.
This introduce a new "secret" per-repository property of type password
as described by #121856.
A token or secret may be used by some non blender.org repositories.
This only shows for remote repositories and is shown in the
"Add Remote Repository" popup.
This commit doesn't implement sending to token to the server which will
be implemented separately.
Ref !121886
Co-authored-by: Dalai Felinto <dalai@blender.org>
Adds a new asset shelf option (`STORE_ENABLED_CATALOGS_IN_PREFERENCES`
option in RNA) to use the Preferences for storing the enabled catalogs.
This way asset shelf types can decide if for their use-case, they want
to synchronize the enabled catalogs over Blender sessions and files, or
keep the stored locally in the file.
This is important because for example on one hand, it would be annoying
if for brush assets you'd have to enable the visible catalog tabs for
every 3D View and every file, while on the other hand you need that
level of control for the pose library where the catalogs the rigger/
animator cares about varies from project to project, character to
character and shot to shot.
Conceptually this also makes some sense: The new brush assets workflow
synchronizes brush assets and their catalogs across Blender sessions
and files, basically making them globally accessible independent of
the current file/project, so treating the enabled catalogs the same
is consistent.
Previously reviewed in #120264
Pull Request: https://projects.blender.org/blender/blender/pulls/121363
This makes the read and write API functions match more closely, and adds
asserts to check that the data size is as expected.
There are still a few places remaining that use BLO_read_data_address
and similar generic functions, these should eventually be replaced as well.
Pull Request: https://projects.blender.org/blender/blender/pulls/120994
This adds implicit sharing support for the `MemFile` undo-step. This decreases memory
usage and increases performance.
Implicit sharing allows the undo system to take (shared) ownership of some data.
Previously, the data would always be serialized and compared to the previous undo-step.
So this turns an O(n) operation into O(1) (in terms of memory usage and time).
Read/write code that wants to make use of this has to use the new `BLO_read_shared`
and `BLO_write_shared` functions respectively. Those either make use of implicit-sharing
internally or do the "full" read/write based on a passed-in function. It seems possible to
use the same API in the future to store shared data to .blend files.
Improvements:
* Much faster undo step creation in many cases by avoiding the majority data copies
and equality checks. This fixes#98574. I found undo step creation and undo step
decoding to be 2-5 times faster in some demo files from the blender website and in
some production files from the Heist project.
* Reduced memory usage when there is large data in `bmain`. For example, when
loading the same highly subdivided mesh that I used in #106228 the memory usage
is 1.03 GB now (compared to 1.62 GB in `main` currently). The main remaining copy
of the data now is done by rendering code.
* Some significant performance improvements were also measured for the new grease
pencil type (#105540).
There is one main downside of using implicit-sharing as implemented here: `MemFile`
undo steps can't be written as .blend files anymore. This has a few consequences:
* Auto-save becomes slower (up to 3x), because it can't just save the previous undo step
anymore and does a normal save instead. This has been discussed in more detail here:
https://devtalk.blender.org/t/remove-support-for-saving-memfile-undo-steps-as-blend-files-proposal/33544
It would be nice to work towards asynchronous auto-save to alleviate this problem.
Some previous work has been done to reduce the impact of this change in 41b10424c7
and f0f304e240. This has been committed separately in efb511a76d.
* Writing `quit.blend` has to do a normal file save now. So it's a bit slower too, but it's
less of a problem in practice.
* The `USE_WRITE_CRASH_BLEND` functionality does not work anymore. It doesn't seem
to be used by anyone (removed in e90f5d03c4)
There are also benefits to not writing `MemFile` from undo steps to disk. It allows us to
more safely do undo-specific optimizations without risking corrupted .blend files. This
is especially useful when we want to preserve forward compatibility in some cases.
This requires converting data before writing the .blend files, but this conversion is not
necessary for undo steps. Trying to implement this kind of optimization in the past has
often lead to bugs (e.g. 43b37fbc93).
Another new problem is that it is harder to know the size of each undo step. Currently, a
heuristic is used to approximate the memory usage, but better solutions could be found
if necessary.
Pull Request: https://projects.blender.org/blender/blender/pulls/106903
I added a new BLO_userdef_default.h header to contain declarations of
two global variables that are still defined in C files. Use of designated
initializers for large structs make those files harder to change.
Arguably this is a better header for them anyway.
Pull Request: https://projects.blender.org/blender/blender/pulls/118015
`UUID` generally stands for "universally unique identifier". The session identifier that
we use is neither universally unique, nor does it follow the standard. Therefor, the term
"session uuid" is confusing and should be replaced.
In #116888 we briefly talked about a better name and ended up with "session uid".
The reason for "uid" instead of "id" is that the latter is a very overloaded term in Blender
already.
This patch changes all uses of "uuid" to "uid" where it's used in the context of a
"session uid". It's not always trivial to see whether a specific mention of "uuid" refers
to an actual uuid or something else. Therefore, I might have missed some renames.
I can't think of an automated way to differentiate the case.
BMesh also uses the term "uuid" sometimes in a the wrong context (e.g. `UUIDFaceStepItem`)
but there it also does not mean "session uid", so it's *not* changed by this patch.
Pull Request: https://projects.blender.org/blender/blender/pulls/117350
Caused by 133dde41bb, it is expected behavior now for linked ID in
general, see also #105786 and #106321.
However, Scenes are a special case here, since they are almost never
(and should not be) indirectly linked by other data, and have (almost)
never any real user, unless they are active in one of the open main
windows.
So this commit essentially reverts the new behavior implemented
in #106321, for linked scenes only.
Should be backported to a potential bugfix release of 4.0.
The range check was applied before padding up, in the unlikely case
sizes (INT_MAX - 2) or (INT_MAX - 1) were passed in,
the value overflow into a negative integer.
- Decouple the compressing and non-compressing wrappers.
- Turn the open(), close() and write() callbacks into proper virtual methods.
- Change the write() interface to take a void pointer and return
a bool to simplify its use.
- Eliminate the need for explicit initialization of the wrappers' fields.
Co-authored-by: Ivan Kosarev <mail@ivankosarev.com>
Pull Request: https://projects.blender.org/blender/blender/pulls/113071
