This commit is a work forward having less updates during playback, which speeds
things up a lot here. The idea is simple: stop update all copy-on-write
datablocks (which implies full re-evaluation actually) on frame change and
re-use existing evaluated meshes as much as possible.
This brings playback speed to 24 fps on the dino test scene here. Performance
drops down a lot when armature is animated tho, but that's because of need of
tangent space calculation which we can't do much about from just a dependency
graph.
Hopefully this doesn't make copy-on-write too unstable, quick tests here are
surviving fine.
BKE_scene_copy explicitly ignores visibility of "source" collections make all
collections visible. This is also tested by regression tests.
While it seems more logical to simply preserve all possible visibility flags
and overrides, don't feel like submitting to a behavior-changes without talking
to author of those guards first.
This commit fixes cycles material preview.
If object is only listed in collection but not added to any of layers we shouldn't create
placeholder for it, because otherwise we'll leave lots of placeholder ID nodes.
Question: can we make this exception to be more reliable?
Note that some little parts of code have been dissabled because eval_ctx
was not available there. This should be resolved once DerivedMesh is
replaced.
The remapping code was creating plkaceholders for objects coming from legacy
bases, but since those objects were never created by dependency graph (since
they are supposed to be ignored) the copy on write relations creation was
confused.
Now we do some special trickery to clear legacy bases on copy on write.
This is the fake ID nature of compositor again. Need to discard such
pointers before freeing datablock even for scenes (before it was done
for objects only).
Previously it was possible to run into situation when armature is constructed prior to
objects which are used for it's constraints. This was causing wrong scene evaluation.
Now we create placeholders for objects used by armature in case they don't have ID node
yet, which ensures we have proper mapping from original to copy-on-write ID pointer.
This shows the bug when IK solver doesn't update reliably when targeted an external
object and when that object is handled by build_object() after the armature.
The issue was caused by id_copy_no_main() changing pointers of constraints used
in pose to a newly allocated ID. This is correct, but caused confusion too our
copy on write remapping, because we are mimicing inplace duplication by copying
memory over from a temporarily duplicated ID to a proper placeholder. This was
causing dangling pointers in pose to a temporarily allocated ID.
Now we add special code to remapping callback which replaces temporary ID with
a proper one.
This commit makes simple cases to work, for example:
- IK solver to an external object
- Object with Armature modifier, "parented" to the deforming armature
(via animation).
More complicated setups (like agent rig) are crashing still.
< Dependency graph Copy-on-Write >
--------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
This is an initial commit of Copy-on-write support added to dependency graph.
Main priority for now: get playback (Alt-A) and all operators (selection,
transform etc) to work with the new concept of clear separation between
evaluated data coming from dependency graph and original data coming from
.blend file (and stored in bmain).
= How does this work? =
The idea is to support Copy-on-Write on the ID level. This means, we duplicate
the whole ID before we cann it's evaluaiton function. This is currently done
in the following way:
- At the depsgraph construction time we create "shallow" copy of the ID
datablock, just so we know it's pointer in memory and can use for function
bindings.
- At the evaluaiton time, the copy of ID get's "expanded" (needs a better
name internally, so it does not conflict with expanding datablocks during
library linking), which means the content of the datablock is being
copied over and all IDs are getting remapped to the copied ones.
Currently we do the whole copy, in the future we will support some tricks
here to prevent duplicating geometry arrays (verts, edges, loops, faces
and polys) when we don't need that.
- Evaluation functions are operating on copied datablocks and never touching
original datablock.
- There are some cases when we need to know non-ID pointers for function
bindings. This mainly applies to scene collections and armatures. The
idea of dealing with this is to "expand" copy-on-write datablock at
the dependency graph build time. This might introduce some slowdown to the
dependency graph construction time, but allows us to have minimal changes
in the code and avoid any hash look-up from evaluation function (one of
the ideas to avoid using pointers as function bindings is to pass name
of layer or a bone to the evaluation function and look up actual data based
on that name).
Currently there is a special function in depsgraph which does such a
synchronization, in the future we might want to make it more generic.
At some point we need to synchronize copy-on-write version of datablock with
the original version. This happens, i.e., when we change active object or
change selection. We don't want any actual evaluation of update flush happening
for such thins, so now we have a special update tag:
DEG_id_tag_update((id, DEG_TAG_COPY_ON_WRITE)
- For the render engines we now have special call for the dependency graph to
give evaluated datablock for the given original one. This isn't fully ideal
but allows to have Cycles viewport render.
This is definitely a subject for further investigation / improvement.
This call will tag copy-on-write component tagged for update without causing
updates to be flushed to any other objects, causing chain reaction of updates.
This tag is handy when selection in the scene changes.
This basically summarizes ideas underneath this commit. The code should be
reasonably documented.
Here is a demo of dependency graph with all copy-on-write stuff in it:
https://developer.blender.org/F635468
= What to expect to (not) work? =
- Only meshes are properly-ish aware of copy-on-write currently, Non-mesh
geometry will probably crash or will not work at all.
- Armatures will need similar depsgraph built-time expansion of the copied
datablock.
- There are some extra tags / relations added, to keep things demo-able but
which are slowing things down for evaluation.
- Edit mode works for until click selection is used (due to the selection
code using EditDerivedMesh created ad-hoc).
- Lots of tools will lack tagging synchronization of copied datablock for
sync with original ID.
= How to move forward? =
There is some tedious work related on going over all the tools, checking
whether they need to work with original or final evaluated object and make
the required changes.
Additionally, there need synchronization tag done in fair amount of tools
and operators as well. For example, currently it's not possible to change
render engine without re-opening the file or forcing dependency graph for
re-build via python console.
There is also now some thoughts required about copying evaluated properties
between objects or from collection to a new object. Perhaps easiest way
would be to move base flag flush to Object ID node and tag new objects for
update instead of doing manual copy.
here is some WIP patch which moves such evaluaiton / flush:
https://developer.blender.org/F635479
Lots of TODOs in the code, with possible optimization.
= How to test? =
This is a feature under heavy development, so obviously it is disabled by
default. The only reason it goes to 2.8 branch is to avoid possible merge
hell.
In order to enable this feature use WITH_DEPSGRAPH_COPY_ON_WRITE CMake
configuration option.
Was rather weird and only used for time source. It is simpler to make depsgraph
to keep track of time source directly.
No need to introduce extra entitites without actual need.
The idea is to accumulate all new tasks in a thread local queue
first without doing any thread synchronization (aka, locks and
conditional variables) and move those tasks to a scheduler queue
once they are all ready. This way we avoid per-task-pool lock
and only have one lock per bunch of tasks.
This is particularly handy when scheduling new dependency graph
node children. Brings FPS of cached simulation from the linked
below file from ~30 to ~50.
See documentation for BLI_task_pool_delayed_push_{begin, end}
and for TaskThreadLocalStorage::do_delayed_push.
Fixes T50027: Rigidbody playback and simulation performance regression with new depsgraph
Thanks Bastien for the review!
These bits became obsolete with the new layer system, so we can
simplify some code around them or avoid existing workarounds which
were trying to keep things working for them.
There are still work needed to be done for on_visible_change to
avoid unnecessary updates, but that can also happen later.
Suspended pools allows to push huge amount of initial tasks
without any threading synchronization and hence overhead.
This gives ~50% speedup of cached rigid body with file from
T50027 and seems to have no negative affect in other scenes
here.
