This adds implicit sharing support for the `MemFile` undo-step. This decreases memory
usage and increases performance.
Implicit sharing allows the undo system to take (shared) ownership of some data.
Previously, the data would always be serialized and compared to the previous undo-step.
So this turns an O(n) operation into O(1) (in terms of memory usage and time).
Read/write code that wants to make use of this has to use the new `BLO_read_shared`
and `BLO_write_shared` functions respectively. Those either make use of implicit-sharing
internally or do the "full" read/write based on a passed-in function. It seems possible to
use the same API in the future to store shared data to .blend files.
Improvements:
* Much faster undo step creation in many cases by avoiding the majority data copies
and equality checks. This fixes#98574. I found undo step creation and undo step
decoding to be 2-5 times faster in some demo files from the blender website and in
some production files from the Heist project.
* Reduced memory usage when there is large data in `bmain`. For example, when
loading the same highly subdivided mesh that I used in #106228 the memory usage
is 1.03 GB now (compared to 1.62 GB in `main` currently). The main remaining copy
of the data now is done by rendering code.
* Some significant performance improvements were also measured for the new grease
pencil type (#105540).
There is one main downside of using implicit-sharing as implemented here: `MemFile`
undo steps can't be written as .blend files anymore. This has a few consequences:
* Auto-save becomes slower (up to 3x), because it can't just save the previous undo step
anymore and does a normal save instead. This has been discussed in more detail here:
https://devtalk.blender.org/t/remove-support-for-saving-memfile-undo-steps-as-blend-files-proposal/33544
It would be nice to work towards asynchronous auto-save to alleviate this problem.
Some previous work has been done to reduce the impact of this change in 41b10424c7
and f0f304e240. This has been committed separately in efb511a76d.
* Writing `quit.blend` has to do a normal file save now. So it's a bit slower too, but it's
less of a problem in practice.
* The `USE_WRITE_CRASH_BLEND` functionality does not work anymore. It doesn't seem
to be used by anyone (removed in e90f5d03c4)
There are also benefits to not writing `MemFile` from undo steps to disk. It allows us to
more safely do undo-specific optimizations without risking corrupted .blend files. This
is especially useful when we want to preserve forward compatibility in some cases.
This requires converting data before writing the .blend files, but this conversion is not
necessary for undo steps. Trying to implement this kind of optimization in the past has
often lead to bugs (e.g. 43b37fbc93).
Another new problem is that it is harder to know the size of each undo step. Currently, a
heuristic is used to approximate the memory usage, but better solutions could be found
if necessary.
Pull Request: https://projects.blender.org/blender/blender/pulls/106903
I added a new BLO_userdef_default.h header to contain declarations of
two global variables that are still defined in C files. Use of designated
initializers for large structs make those files harder to change.
Arguably this is a better header for them anyway.
Pull Request: https://projects.blender.org/blender/blender/pulls/118015
`UUID` generally stands for "universally unique identifier". The session identifier that
we use is neither universally unique, nor does it follow the standard. Therefor, the term
"session uuid" is confusing and should be replaced.
In #116888 we briefly talked about a better name and ended up with "session uid".
The reason for "uid" instead of "id" is that the latter is a very overloaded term in Blender
already.
This patch changes all uses of "uuid" to "uid" where it's used in the context of a
"session uid". It's not always trivial to see whether a specific mention of "uuid" refers
to an actual uuid or something else. Therefore, I might have missed some renames.
I can't think of an automated way to differentiate the case.
BMesh also uses the term "uuid" sometimes in a the wrong context (e.g. `UUIDFaceStepItem`)
but there it also does not mean "session uid", so it's *not* changed by this patch.
Pull Request: https://projects.blender.org/blender/blender/pulls/117350
Caused by 133dde41bb, it is expected behavior now for linked ID in
general, see also #105786 and #106321.
However, Scenes are a special case here, since they are almost never
(and should not be) indirectly linked by other data, and have (almost)
never any real user, unless they are active in one of the open main
windows.
So this commit essentially reverts the new behavior implemented
in #106321, for linked scenes only.
Should be backported to a potential bugfix release of 4.0.
The range check was applied before padding up, in the unlikely case
sizes (INT_MAX - 2) or (INT_MAX - 1) were passed in,
the value overflow into a negative integer.
- Decouple the compressing and non-compressing wrappers.
- Turn the open(), close() and write() callbacks into proper virtual methods.
- Change the write() interface to take a void pointer and return
a bool to simplify its use.
- Eliminate the need for explicit initialization of the wrappers' fields.
Co-authored-by: Ivan Kosarev <mail@ivankosarev.com>
Pull Request: https://projects.blender.org/blender/blender/pulls/113071
Use with_alias / without_alias suffix for functions
so it's clear which is intended (no functional changes).
Add macros for versioning checks to avoid noisy changes,
currently they use the non-aliased versions of these functions
but should eventually be moved to use the aliased versions because
it's confusing to use the original names when these should only need
to be referenced from `dna_rename_defs.h`.
Rename for clarity in preparation for further refactoring.
Remove the need for `_nr` in function names.
- Rename *_find() functions to *_exists() since they returned a boolean.
- Rename *_find_nr() functions to *_find().
- Rename *_struct_elem_* to *_struct_member_*.
- Rename DNA_elem_size_nr -> DNA_struct_member_size.
- Rename DNA_elem_offset -> DNA_struct_member_offset_by_name.
This is more of a temp hack than a proper fix, proper solution would be
to make shapekeys actual embedded IDs (which they are, in all aspects
but actual implementation), and to address long-standing design tasks
about handling of unused data on file save (see #61209 and #87490).
But for now, simply do not write ShapeKeys IDs if they have no owner, or
their owner has no user (and is therefore also not written to disk).
These checks (shape keys and ID names) cost less than 1ms in Pets
production files, so think they are OK to run systematically.
The library consistency one is way more expensive (~200ms), so keeping
it behind the G_DEBUG_IO debug option for now.
NOTE: This code remains only executed in 'unlikely' case `G_DEBUG_IO` is
enabled. Think this should be systematically done, even though it can
have a non-neglectable cost... Will submit design task first though.
Using this flag from linked data is always a double-edge sword, in one
end some user have been relying on it to keep around data that is not
really used as ID (like e.g. text data-blocks, node trees, see
e.g. #103687, #105687). On the other end, it often causes over-keeping
of linked data reference in production files.
From now on, when an unused linked data is to be kept around, users
should create an ID property to reference it.
Implements #106321.
Pull Request: https://projects.blender.org/blender/blender/pulls/111042
Support name-spaced add-ons, exposed via user configurable extension
repositories.
Directories for add-ons can be added at run-time and are name-spaced to
avoid name-collisions with Python modules or add-ons from other
repositories.
This is exposed as an experimental feature "Extension Repositories".
Details:
- A `bUserExtensionRepo` type which represents a repository which is
listed in the add-ons repository.
- `JunctionModuleHandle` class to manage a package with sub-modules
which can point to arbitrary locations.
- `bpy.app.handlers._extension_repos_update_{pre/post}` internal
callbacks run before/after changes to extension repositories,
callbacks are used to sync the changes to the Python package that
exposes these to add-ons.
- The size of an add-on name has been increased so a user-defined package
prefix can be included without enforcing shorter add-on names.
- Functionality relating to package management has been left out of this
change and will be developed separately.
Further work:
- While a repository can be renamed, enabled add-ons aren't renamed.
Eventually we might want to support this although we could also
disallow renaming repositories with add-ons enabled as the name isn't
all that significant.
- Removing a repository should remove all the add-ons located in this
repository.
- Sub-module names are currently restricted to `[A-Za-z]+[A-Za-z0-9_]*`
we might want to relax this to allow unicode characters (we might
still want to disallow `-` or any characters that would prevent
attribute access in code).
Ref !110869.
Reviewed By: brecht
Replace `typedef struct X {} X;` with `struct X {};`
In some cases the first and last name didn't match although this
is rarely useful, even a typo in some cases, e.g. TrachPathPoint.
A lot of files were missing copyright field in the header and
the Blender Foundation contributed to them in a sense of bug
fixing and general maintenance.
This change makes it explicit that those files are at least
partially copyrighted by the Blender Foundation.
Note that this does not make it so the Blender Foundation is
the only holder of the copyright in those files, and developers
who do not have a signed contract with the foundation still
hold the copyright as well.
Another aspect of this change is using SPDX format for the
header. We already used it for the license specification,
and now we state it for the copyright as well, following the
FAQ:
https://reuse.software/faq/
Internal private struct was using `owner_id`/`self_id`, while the public
callback data struct was using `id_owner`/`id_self`.
Now using internal naming everywhere in lib_query related code, as
`owner_id` is already used in very low-level 'fundamental' part of the
code, e.g. in the `PointerRNA` struct, or in ID's 'loopback' pointers
for embedded data.
Note that this is only a very small first step toward proper naming
consistency for these type of data, the mismatch is currently spread all
over the code base.
We also need to document more formally the meaning and differences
between `self` and `owner` here.
Include the term "overwrite" so users of this function are aware
this function will delete the destination file (unlike LIBC rename).
- Add BLI_rename_overwrite (previously called BLI_rename).
- BLI_rename is now a wrapper for rename.
Use BLI_rename when renaming in the file selector.
See: https://projects.blender.org/blender/blender/issues/103343
Changes:
1. Added `BKE_node.hh` file. New file includes old one.
2. Functions moved to new file. Redundant `(void)`, `struct` are removed.
3. All cpp includes replaced from `.h` on `.hh`.
4. Everything in `BKE_node.hh` is on `blender::bke` namespace.
5. All implementation functions moved in namespace.
6. Function names (`BKE_node_*`) changed to `blender::bke::node_*`.
7. `eNodeSizePreset` now is a class, with renamed items.
Pull Request: https://projects.blender.org/blender/blender/pulls/107790
