Code would not keep filedata for read libraries long enough to be able
to pass along to the 'after_liblink' versioning code. Instead, it was
passing the main file data, leading to all kind of potential breakage in
versioning code, e.g. in case of checks on available DNA data.
Fix#111776.
Add a call to `BKE_main_namemap_validate_and_fix` in the 'general
sanity validation' `after_liblink_merged_bmain_process` function.
This will detect, fix and report invalid ID naming issues (case found in
an early Gold production file).
The `lib_link` callback cannot always be fully replaced/removed, as in
some case it is also doing some validation checks, or data editing based
on the result of lib_linking internal ID pointers.
The callback has been renamed for that purpose, from `read_lib` to
`read_after_liblink`. It is now called after all ID pointers have been
fully lib-linked for the current ID, but still before the call to
`do_versions_after_linking`.
This change should not have any behavioral effect. Although in theory
the side-effect of this commit (to split lib linking itself, and the
validation/further processing code) into two completely separated steps
could have some effects, in practice none are expected, and tests did
not show any changes in behavior either..
Part of implementing #105134: Removal of readfile's lib_link & expand code.
The `expand` callback is 'trivial' to replace, since it is only iterating
over ID pointers and calling a callback.
The only change in behavior here is that some pointers that were not
processed previously will now be.
In practice this is not expected to have any real effect (usually
the IDs used by these pointers would have been expanded through other
usages anyway). But it may solve a few corner cases, undocumented issues
though.
Part of implementing #105134: Removal of readfile's lib_link & expand code.
Using this flag from linked data is always a double-edge sword, in one
end some user have been relying on it to keep around data that is not
really used as ID (like e.g. text data-blocks, node trees, see
e.g. #103687, #105687). On the other end, it often causes over-keeping
of linked data reference in production files.
From now on, when an unused linked data is to be kept around, users
should create an ID property to reference it.
Implements #106321.
Pull Request: https://projects.blender.org/blender/blender/pulls/111042
There is no reason at all for each ID read/write callbacks to have to
deal with this generic, common data explicitely. Generic ID read/write
code is the place to handle such data (just like asset, liboverride,
etc. data are handled already).
Note behavioral change expected here.
Recent change to liboverride data model (6a86dd5f34) forgot to add the
new 'RNA item ID pointers references' to the expander code (readfile
area).
Note that this was essentially harmless, as these IDs are expected to be
referenced by actual data (or liboverride reference pointer) as well,
they should never be the only ones pointing at a linked ID. But better
to do it properly nonetheless.
Using ClangBuildAnalyzer on the whole Blender build, it was pointing
out that BLI_math.h is the heaviest "header hub" (i.e. non tiny file
that is included a lot).
However, there's very little (actually zero) source files in Blender
that need "all the math" (base, colors, vectors, matrices,
quaternions, intersection, interpolation, statistics, solvers and
time). A common use case is source files needing just vectors, or
just vectors & matrices, or just colors etc. Actually, 181 files
were including the whole math thing without needing it at all.
This change removes BLI_math.h completely, and instead in all the
places that need it, includes BLI_math_vector.h or BLI_math_color.h
and so on.
Change from that:
- BLI_math_color.h was included 1399 times -> now 408 (took 114.0sec
to parse -> now 36.3sec)
- BLI_simd.h 1403 -> 418 (109.7sec -> 34.9sec).
Full rebuild of Blender (Apple M1, Xcode, RelWithDebInfo) is not
affected much (342sec -> 334sec). Most of benefit would be when
someone's changing BLI_simd.h or BLI_math_color.h or similar files,
that now there's 3x fewer files result in a recompile.
Pull Request #110944
In RNA collections storing ID references, the name of the collection
item may not always be unique, when several IDs from different libraries
are present.
While rare, this situation can become deadly to liboverride, by causing
random but exponential liboverride hierarchies corruptions.
This has already been alleviated by using preferably both name and index
in items lookup (a05419f18b) and by reducing the risk of name collision
in general between liboverrides and their linked reference (b9becc47de).
This commit goes further, by ensuring that references to items of RNA
collections of IDs stored in liboverride operations become completely
unambiguous. This is achieved by storing an extra pointer to the item's
ID itself, when relevant.
Lookup then requires a complete match `name + ID` to be successful,
which is guaranteed to match at most a single item in the whole RNA
collection (since RNA collection of IDs do not allow duplicates, and
the ID pointer is always unique).
Note that this ID pointer is implemented as an `std::optional` one
(either directly in C++ code, or using an new liboverride operation `flag`
in DNA). This allows to smoothly transition from existing data to the
added ID pointer info (when needed), without needing any dedicated
versioning. This solution also preserves forward compatibility as much
as possible.
It may also provide marginal performances improvements in some cases, as
looking up for ID items in RNA collections will first check for the
ID pointer, which should be faster than a string comparision.
Implements #110421.
Pull Request: https://projects.blender.org/blender/blender/pulls/110773
Support name-spaced add-ons, exposed via user configurable extension
repositories.
Directories for add-ons can be added at run-time and are name-spaced to
avoid name-collisions with Python modules or add-ons from other
repositories.
This is exposed as an experimental feature "Extension Repositories".
Details:
- A `bUserExtensionRepo` type which represents a repository which is
listed in the add-ons repository.
- `JunctionModuleHandle` class to manage a package with sub-modules
which can point to arbitrary locations.
- `bpy.app.handlers._extension_repos_update_{pre/post}` internal
callbacks run before/after changes to extension repositories,
callbacks are used to sync the changes to the Python package that
exposes these to add-ons.
- The size of an add-on name has been increased so a user-defined package
prefix can be included without enforcing shorter add-on names.
- Functionality relating to package management has been left out of this
change and will be developed separately.
Further work:
- While a repository can be renamed, enabled add-ons aren't renamed.
Eventually we might want to support this although we could also
disallow renaming repositories with add-ons enabled as the name isn't
all that significant.
- Removing a repository should remove all the add-ons located in this
repository.
- Sub-module names are currently restricted to `[A-Za-z]+[A-Za-z0-9_]*`
we might want to relax this to allow unicode characters (we might
still want to disallow `-` or any characters that would prevent
attribute access in code).
Ref !110869.
Reviewed By: brecht
This implements the main aspects of changes to blendfile compatibility
as designed in #109151:
* Blender files which file minversion is newer than current Blender
executable won't be loaded at all.
* Blender files which file version is newer than current Blender will
triger systematic warning to user:
* In the status info bar (lower right corner in default UI).
* When attempting to save (overwrite) them.
This means that the file minversion becomes a hard limit, and not a
soft, warning-only as it used to be. Further more, forward compatibility
warning is now systematic (instead of depending on file minversion),
and more visible for users.
See also https://wiki.blender.org/wiki/Process/Compatibility_Handling
for details over the new policy.
Technically:
* Opening any file with a minversion newer than current Blender file one
now triggers an early abort, with an error message reported to the user.
This is handled by a new utils called from `blo_decode_and_check`.
* Any file newer than current Blender version sets a new
`has_forward_compatibility_issues` flag in Main struct at read time.
* Status bar info area is turned into a template, which uses this flag
to display special warning UI and tooltip when set.
* A new confirmation popup appears when user tries to save (overwrite)
such a 'newer' blendfile, stating potential loos of data, and
proposing by default to 'save as' instead.
* The 'quit unsaved' popup has also been updated to 'save as' instead of
'save' when the edited file is has potential forward compitibility
issues.
Part of #109151 (PR !110109).
This formats code that is disabled using `#if 0`. Formatting was achieved
by temporarily changing `#if 0` to `#if 1 /*something*/`, then formatting,
and then changing it back to `#if 0`.
This data-block was originally added in eb4e3bbe68.
However, that original plan wasn't fully implemented, with simulations
now integrated with geometry nodes and modifiers instead of a separate
data-block. We kept the data-block around anyway since we have the
loose plan of using a similar data-block to make global simulations
connected between multiple objects. But it may be a while before we
implement that, and in the meantime having this just causes confusion.
By removing the extra complete check/remapping of ID pointers in undo
case, ebb5643e59 merely revealed how broken the 'undo_preserve' code of
Scene was.
It cumulated a flock of issues, all more or less related to ID pointers:
* 'source of truth' should be the _old_ data (toolsettings), not the new
one, since this is the one kept at the end of the process;
* In some cases, some paint data may exist in one, but not the other
of the two 'old' and 'new' toolsettings data.
* Since this data is preserved to its latest version accross undos, its
ID pointers can become completely unrelated to these read from the
undo memfile, _even when the Scene itself is detected as unchanged_!
This implies that:
+ undo_preserve code has to be called even when there is no liblinking
(when the ID is detected as unchanged and re-used 'as-is').
+ Using existing ID addresses to find/validate an ID pointer in
undo_preserve process is like playing Russian roulette - invalid
memory access and crash is guaranteed at some point or another.
Use `session_uuid` value instead to ensure a valid ID pointer is set
(or null in case none can be found).
NOTE: while these issues also exist in previous releases (including both
latest LTSs), they were hidden by the code later in `setup_app_data`,
preventing any crash to happen. So backporting this fix would be far too
risky for a very minimal benefit imho.
This commit affects:
* Reading undo steps from memfile (aka 'Global Undo');
* Handling of UI IDs (WindowManager, Workspaces and Screens) when
opening a .blend file.
While no major changes are expected from a user PoV, there may be some
unexpected changes in rare edge-cases. None has been identified so far.
Undo step loading should be marginally faster (`setup_app_data` itself
is 2-3 times faster, as it does not do remapping anymore, which makes the
whole 'read undo step' process about 20% faster - but the most
time-consuming step on undo is the depsgraph processing, which remains
unchanged here).
This commit also solves some bugs (crashes) in some relatively uncommon
cases, like e.g. if the WM had an IDProperty pointing at an object and
UI is not loaded when opening a new .blend file with the 'Load UI' option
enabled (as in previous code on file opening WM ID would never be
remapped).
From a more technical side, this commit aims mainly at cleaning things
up, in preparation for the introduction of new 'no undo, no readfile'
type of handling (as part of the Brush Assets project):
- Prevent WM code from doing (too much) horrible ID 'management' on
its WM when opening a new file. It used to remove current WM from
the Main database, store it in a temporary own list, and then free
it itself...
- Trying to make the complex logic behind WM handling on file reading a
bit more easy to follow, at least way more documented in code.
- Keep the handling of 'IDs being re-used from old Main' in a single
place, as much as possible:
-- Readfile code itself in undo case (because it's more efficient,
and undo case is in a way simpler than actual .blend file
reading case). The whole `blo_lib_link_restore` block of code
is also removed.
-- (Mostly) setup_app_data code in actual file reading case.
- Sanitize the usage of the 'libmap' in readfile code in undo case
(waaaaay too many pointers were added there, which was hiding some
other issues in the related code, and potentially causing (in
rare cases) memory addresses collisions.
Pull Request: https://projects.blender.org/blender/blender/pulls/108016
A lot of files were missing copyright field in the header and
the Blender Foundation contributed to them in a sense of bug
fixing and general maintenance.
This change makes it explicit that those files are at least
partially copyrighted by the Blender Foundation.
Note that this does not make it so the Blender Foundation is
the only holder of the copyright in those files, and developers
who do not have a signed contract with the foundation still
hold the copyright as well.
Another aspect of this change is using SPDX format for the
header. We already used it for the license specification,
and now we state it for the copyright as well, following the
FAQ:
https://reuse.software/faq/
Adds the initial stage for the grease pencil 3.0 project.
This patch includes:
* New ID and new object type.
* New DNA structures.
* New drawing engine for grease pencil (gpencil-next).
* Tests for the new grease pencil data-type.
* A few operators for conversion, switching modes and (simple) drawing.
Exposed to the user:
* An experimental option to switch to the new grease pencil.
* This will switch the grease pencil render engine to gpencil-next which can only render the new object type.
Current grease pencil objects will no longer render.
* Changing this option currently requires a restart of blender (for the keymap to update).
* A conversion setting in the `Object` > `Convert To` operator.
* A drawing operator in `Draw Mode`.
Pull Request: https://projects.blender.org/blender/blender/pulls/106848
In readfile code, when looking up for an already existing Library ID
based on the filepath, the logic handling said file path was wrong.
NOTE: This probably does not have any effect in practice, but better be
safe than sorry.
Found while investigating issues when opening
`lib/tests/libraries_and_linking/libraries/main_scene.blend`.
The main issue was the fact that if a Scene is overridden, it's content
will be fully invalidated when updating the liboverride at the end of
the file reading process. Since the FileData keeps a pointer to the
active view layer, it needs to be udated then.
As a side consequence, the liblinking of global data also needs to
happen before liboverrides are updated.
The usage of the lib pointer was cryptic to say the least, it was
essentialy used to allow or not local IDs results in the mapping
old_id_pointer -> new_id_pointer lookup result.
Now:
- Explicitely pass a boolean to allow or not local ID as result in
lower-level code.
- Pass the 'self ID' pointer instead of its library to the whole
liblinking code (that was already the case in a few places).
Note that naming of the 'self id' pointer is currently very inconsistent
throughout the readfile liblink code, this will have to be cleaned up in
a separate step later. For now, `self_id` has been chosen for new code
as it matches the terminology in lib_query code.
The later change can also allow finer handling of lookup on undo, based
on how it was re-read (or not), should the need for this arise.
See: https://projects.blender.org/blender/blender/issues/103343
Changes:
1. Added `BKE_node.hh` file. New file includes old one.
2. Functions moved to new file. Redundant `(void)`, `struct` are removed.
3. All cpp includes replaced from `.h` on `.hh`.
4. Everything in `BKE_node.hh` is on `blender::bke` namespace.
5. All implementation functions moved in namespace.
6. Function names (`BKE_node_*`) changed to `blender::bke::node_*`.
7. `eNodeSizePreset` now is a class, with renamed items.
Pull Request: https://projects.blender.org/blender/blender/pulls/107790
Mostly some minor renaming of parameters and variables, adding a few
comments...
The only actual behavior change is a more correct handling of the
`LIB_TAG_KEEP_ON_UNDO` on ID tag. This should not have any consequences
in current code though.
So no actual change in behavior is expected from this commit.
When resyncing linked liboverride data, new IDs may be created that do
not exist in actual library file (since the lib file has not been resynced).
If such 'virtual linked liboverrides' data-blocks are used locally (e.g.
by adding such object to a local collection), on next file read they will be
detected as missing.
Now resync code will list such missing linked IDs that were
liboverrides, and try to re-use them when matching (by root name and
library) with newly generated virtual liboverrides.
The process may not be 100% perfect, especially a perfect one-to-one
remapping cannot be ensured if source data keep being changed over and
over (because of the order in which virtual linked liboverrides
generated by resync may change over time). However, in practice this
should fix the vast majority of issues, especially if sane naming
practices are used on IDs.
---------------
For the record, an attempt was made to write liboverride data together
with the placeholders for linked IDs in .blendfile. In theory, this
should ensure a perfect and fully valid re-usage of such IDs.
However, for this to work, not only the liboverride data of linked IDs need
to be written on disk, but also all ID references in this data has to be
considered as directly linked, to ensure that such liboverride data can
be re-read properly.
Otherwise, these placeholders would get a liboverride data with NULL ID
pointers, which is useless.
Such change feels way to intrusive for the very limited benefit, so for
now would consider current solution as the best option.
Pull Request: https://projects.blender.org/blender/blender/pulls/107144
- Rename name/filename/path to filepath when it's used for full paths.
- Rename name/path to dirpath when it refers to a directory.
- Rename file to filepath or path (when it may be a file or dir).
- Rename ImBuf::name & anim::name to filepath.
