As Relocate code temporarily removes some linked IDs from Main, this can
make collection hierarchy data invalid, in a way that will break
`BKE_main_collections_parent_relations_rebuild`.
For now, simply move that call outside of `BLO_library_link_end` and do
it after adding back all linked IDs to main.
Pull Request: https://projects.blender.org/blender/blender/pulls/135936
Relocation is the only case where old and new linked data may have
a different library. This makes remapping dependencies IDs for the
liboverrides that use these linked data somewhat more challenging and
complicated.
From basic tests with both simple data, and full production-level
Mikassa char, it seems to behave fairly well now.
So far it was only possible to relocate a whole library, now one can
also relocate a single linked ID (pulling in all of its dependencies).
This is essentially linking the new data, remapping local usages of the
old linked data to the new one, removing no more used IDs, and updating
liboverrides if needed.
Note: This commit is essentially non-behavioral change, expect in some
fairly rare edge cases.
This commit does a few things:
* Move the whole BKE_main_namemap code to modern C++.
* Split API calls to work with the global namemap, or the local ones.
* Simplify and make the code easier to follow and understand.
* Reduce 'default' memory usage by using growing BitVector for numeric
suffix management, instead of a fixed 1K items.
* Fix inconsistent handling of 'same base name and numeric suffix,
different name' issues (e.g. 'Foo.1' and 'Foo.001'), see
`re_create_equivalent_numeric_suffixes` new unittest.
* Fix completely broken handling of `global` namemaps. This was
(probably!) OK so far because of their currently very limited
use-cases.
It also adds a few minor improvements to existing behavior (essentially
in exotic rare edge cases):
* Names that get too long are now only shortened by one char at a time,
trying to modify the requested base name as little as possible.
* Names that are short, but for which all the manageable numeric suffixes
are already in use, are extended with an (increasing) number, instead
of being shortened.
This work also allowed to detect a few (apparently harmless?) bugs in
existing code, which have been fixed already in 4.4 and main, or in this
commit as well when they depend on changes in namemap code itself.
About performances: This commit introduces a minor slow-down. Some tests
heavily relying on this code (like `bl_id_management` and `blendkernel`
e.g.) get slightly slower (resp. about 1% and 5%). This seems to come
mostly from the added complexity to handle correctly multiple different
names with the same base and numeric suffix value ('Foo.1' and
'Foo.001', but also in the global namemap context where IDs from
different libraries can have the same name).
Pull Request: https://projects.blender.org/blender/blender/pulls/135199
This replaces the `set_listbasepointers` function with `BKE_main_lists_get`
which returns an array of `ListBase *`. This simplifies the caller a bit. In
some cases, it can be simplifed further by changing the order in which we iterate
over the listbase. For historical reasons, we iterate from the back to front in
most cases but sometimes the order does not matter. I did keep the iteration order
in this patch though, to avoid regressions.
Pull Request: https://projects.blender.org/blender/blender/pulls/134242
Move `Library.runtime` to be a pointer, move the related
`LibraryRuntime` struct to `BKE_library.hh`. Similar to e.g.
Mesh.runtime, that pointer is expected to always be valid, and is
allocated at readtime or when creating a new Library ID.
Related smaller changes:
* Write code now uses standard ID writing codepath for Library IDs too.
* Runtime pointer is reset to nullptr before writing.
* Looking up a library by its absolute path is now handled through a
dedicated utils, `search_filepath_abs`, instead of using
`BLI_findstring`.
Pull Request: https://projects.blender.org/blender/blender/pulls/134188
Move the `ID_TAG_ID_LINK_PLACEHOLDER` bit of `id->tag` to
`id->runtime.readfile_data->tags.is_id_link_placeholder`. It also
introduces the necessary stucts and allocation/freeing code.
Old code:
```cpp
if (id_tag & ID_TAG_ID_LINK_PLACEHOLDER) {
```
New code:
```cpp
if (readfile_id_runtime_tags(id).is_id_link_placeholder) {
```
where `readfile_id_runtime_tags(id)` is a getter for
`id->runtime.readfile_data->tags` that is null-safe for
`id->runtime.readfile_data`. The `readfile_data` is not allocated in
these cases:
1. When reading undo steps, because that doesn't have to deal with
versioning or linking (which are the sole purposes for this
struct).
2. When linking from another file (for example from the 'Link...'
operator). The just-linked IDs will have the `readfile_data`
struct, but already-loaded IDs will already have had those freed.
No functional changes intended.
Pull Request: https://projects.blender.org/blender/blender/pulls/132169
Design Task: #131695
The new --disable-liboverride-auto-resync commandline option will prevent
running liboverride auto-resync immediately after loading a blendfile.
This is an alternative way to setting the User Preferences Debug option
`No Override Auto Resync`.
-------
Some quick performances tests:
Loading a Gold production file goes from 26.5s to 25.9s.
Combined with the recent `--disable-depsgraph-on-file-load` option, it
goes from 3.5s to 2.9s, so another nice extra 15% speedup when only
bare-metal blendfile loading is needed.
Pull Request: https://projects.blender.org/blender/blender/pulls/132017
The `pre` handler is called after blender internal code is done populating
the link/append context with data to be processed, and before this data
starts being linked from library files.
The `post` handler is called after blender is done linking, and
potentailly appending and/or instantiating, the requested data and all
of their dependencies.
Both handlers are called with a single argument, the link/append
context.
An new RNA sets of wrappers have been added to expose relevant info from
these internal C++ structures.
NOTE: !113658 is very similar (but tied to asset drag & drop), whereas
this PR is more general (these could probably live hand-in-hand / side-
by-side).
Implements #122357
Pull Request: https://projects.blender.org/blender/blender/pulls/128279
-----------------
Some quick py example code:
```python
import bpy
def my_handler_pre(lapp_context):
print("About to {}:\n\t".format("link" if "LINK" in lapp_context.options else "append"),
"\n\t".join("{} '{}', from libs ['{}']".format(item.id_type, item.name,
"', '".join([l.filepath for l in item.source_libraries]))
for item in lapp_context.import_items))
def my_handler_post(lapp_context):
print("{}:\n\t".format("Linked" if "LINK" in lapp_context.options else "Appended"),
"\n\t".join("{} '{}', from lib '{}'".format(item.id.id_type, item.id.name, item.source_library.filepath)
for item in lapp_context.import_items))
bpy.app.handlers.link_append_pre.append(my_handler_pre)
bpy.app.handlers.link_append_post.append(my_handler_post)
```
Mainly remove usages of BLI memarena and bitmap, and use owninh C++
containers instead. And replace pointers by references in some of the
internal functions parameters.
Pull Request: https://projects.blender.org/blender/blender/pulls/128041
Issue found here at the studio in some production files using dirty
hacks. Ideally we could fully forbid such assignement in BPY/RNA API
too, but there is no 'proper' way to achieve this without this dirty
hack currenlty.
So instead, check for this case and print a nice error about it for now.
Code executed in `BLO_read_do_version_after_setup` would not update in
any ways the info in link/append context data.
NOTE: this could potentially be extended to other code in this 'complex
do version' area. However, other versionning did not cause issues
apparently so far, so would rather until such changes are proven needed.
Previously, values for `ID.flag` and `ID.tag` used the prefixes `LIB_` and
`LIB_TAG` respectively. This was somewhat confusing because it's not really
related to libraries in general. This patch changes the prefix to `ID_FLAG_` and
`ID_TAG_`. This makes it more obvious what they correspond to, simplifying code.
Pull Request: https://projects.blender.org/blender/blender/pulls/125811
When appending e.g. an object, it's generally expected that it is not just
appended but also added to the active collection. Something similar happens for
adding collections, or just geometry like a mesh data-block. This is called
"loose data instantiation" in the source code.
Which data is instantiated and how depends on context. Generally, it's best to
first do the low-level link or append operation and only afterwards do the
instantiation at a higher level where more context is known. For example, when
dragging an object asset into the 3D view, it's expected that the object is
added to the active collection. When dragging a node group into the node editor
that uses an object internally (e.g. as mesh storage), not so much.
This patch adds a new `BKE_blendfile_link_append_instantiate_loose` method that
does the "default instantiation", i.e. the one we want to happen when using the
general link/append operator to import data. Instead of calling this directly
from low level `BKE_blendfile_append`, it's now called in `wm_link_append_exec`
(and other places where desired).
Furthermore, `view3d_ob_drop_copy_external_asset` does not use this anymore, but
explicitly adds only the dragged object to the active collection. Some places,
e.g. when linking/appending using the Python API, don't do any instantiation at
all. They didn't do this before either, but it was way less obvious (it used
`BLO_library_link_params_init` instead of
`BLO_library_link_params_init_with_context`). Now there is always an explicit
call to do instantiation in higher level code.
Pull Request: https://projects.blender.org/blender/blender/pulls/125814
This replaces two members of `BlendfileLinkAppendContext`:
* `LinkNodePair libraries` -> `Vector<BlendfileLinkAppendContextLibrary *>`
* `LinkNodePair items` -> `std::list<BlendfileLinkAppendContextItem *>`
This simplifies the code quite a bit.
Making `libraries` a `Vector` works because afaik we never iterate over the
libraries while also adding new ones (which would invalidate iterators). It's
also nice, necause we use random access in some cases.
`items` can't become a vector, because afaik more items may be added while
iterating over it. So it's still a linked list now, but with a better type that reduces
boilerplate a lot.
Pull Request: https://projects.blender.org/blender/blender/pulls/124524
This changes the type of `new_id_to_item` from `GHash` to
`Map<ID *, BlendfileLinkAppendContextItem *>` which simplifies code. For this to work
nicely, `BlendfileLinkAppendContext` had to become a non-trivial type. Since it
was allocated with `calloc` before, I gave all members
zero-default-initializers.
This also changes `BKE_blendfile_link_append_context_new` so that it uses
`MEM_new` instead of allocating the `BlendfileLinkAppendContext` as part of the
memory arena, because that's the more common way to do it, and it does not seem
like saving this one allocation here makes a difference considering how this is
used.
Pull Request: https://projects.blender.org/blender/blender/pulls/124516
Linking a datablock from a library does not update the namemap, but
there is an assert added in 47672bd06a that checks this at the end
of library linking.
Usually it does not trigger because the namemap does not exist, and it
is cleared at the end of the process. Now clear it earlier, at the
moment the namemap becomes invalid.
Pull Request: https://projects.blender.org/blender/blender/pulls/122182
Code used to tag liboverrides references as 'pre-existing' to force code
further down the way to always keep these IDs linked.
However, this was a (bad) hack, since it could have uncontrollable
side-effects, abusing a tag for somethig else than its original meaning.
And this should not have been needed for quite some time already, as
liboverrides handling was already properly done by append
post-processing code.
No behavior change expected here.
In a nutshell, the handling of dependencies in the 'append' part of the
code was not fully correct, and could break badly in some complex cases
(like appending complex hierarchies of data partially re-using some
dependencies from other previously appended data).
This could lead to e.g. some linked data referencing some local IDs...
straight way to crash in undo case (among many other problems).
Previous code was fairly compact and tried to be smart and efficient,
making some not-always-correct assumptions, and being quite hard to
fully understand.
So the first step of this fix was some refactoring:
* Splitting the post-process part of the 'link' case into its own
function (it is fairly trivial, and while it does duplicate some
logic to some extent, it makes the overall link/append process
clearer).
* Heavily refactoring the part of the 'append' code that decides how
to handle each linked ID.
The append-related post-processing is now significantly more complex,
but hopefully better divided in reasonably logical steps. And it is now
expected to deal with complex re-usability cases properly.
Pull Request: https://projects.blender.org/blender/blender/pulls/121867
While `do_versions_after_setup` does more than what was previously done
by link/append code (essentially also handles pre-2.50 IPO and sound
proxy conversions), there is no real fundamental reasons to not use it
in linking code.
The IPO and sound proxy conversion codes are likely fully broken for
linked data - but they should then be removed or fixed, since they will
be applied on linked data in blendfile reading case anyway.
And in general, versioning should 'just work' the same when loading a
whole blendfile, or linking some data from a library. Keeping things
separate only makes it harder to work with, and easier to hide issues
with linked data.
While `BKE_blendfile_link` is used in both cases, there is a small
subset of its code only executed in linking case. It now lives into its
own util function, for clarity.
There should be absolutely no functional change in this commit.
Handling of the blendfile handle freeing when linking data from a
blendfile requiring endianness conversion was totally broken, leading
to double-freeing attempts.
Guess that the fact that this was never reported shows how rare
'big-endian' blendfiles are nowadays... But we still have a few in our
test repo.
This adds a new exprimental option to automatically convert GP legacy
data to GPv3 one.
It supports also linking and appending cases. Conversion also happens
when opening a file linking GP legacy data saved by an older .blendfile.
Pull Request: https://projects.blender.org/blender/blender/pulls/118705
This PR fixes several issues with the versioning that replaces the old
auto smooth flag with a modifier.
One issue is that the flag wasn't cleared in the initial versioning
code. That means some objects have the replacement modifier but their
meshes still have the flag set. The fix for that is to make the
versioning idempotent by trying to find an existing node group before
adding a new one. The versioning is now re-run on all objects to clear
the flags. Flags on all meshes are cleared too, even unused meshes.
That could cause loss of the auto-smooth when the mesh is linked from
a different blend-file, but that situation should be very rare.
Another issue was that the versioning wasn't run when linking objects.
That was simple to solve by adding the versioning where the proxy
versioning already existed for that case.
Finally, arguably the largest issue was that the the newly added node
groups were always added as local data-blocks. When linking, having
library data-blocks point to local data-blocks not in that library is
quite bad and breaks assumptions around Blender. This is solved by
having an auto smooth node group per library.
Resolves#119516, #119455, #119447
Pull Request: https://projects.blender.org/blender/blender/pulls/119539
This reverts commit c6497dd9f7.
This wasn't working the way I expected, it seems the LIB_TAG_NEW isn't
set here. Better to address the fundamental issues anyway. Sorry for
the noise.
When appending objects from an older file, versioning needs to run to add
an auto smooth modifier if necessary. However, this was running for all
objects rather than just the newly appended objects. It's quite wrong to
modify existing objects here, so add an explicit check for that. This could
improve performance as well, but skipping checks for objects when
the work is unnecessary.
The crash will hopefully be resolved by other improvements to the auto
smooth versioning (making it properly idempotent and running it again
should fix the problem with the hidden legacy flags still set). But I still think
this PR is worth committing, to be very explicit about only modifying new
objects in versioning code.
Pull Request: https://projects.blender.org/blender/blender/pulls/119467
