Part of #131825
Currently, no change to an RNA property invoked from the UI while inside
Vertex Paint mode creates an undo step. This includes both tool
properties (e.g. brush size, tool settings) as well as mesh properties
(symmetry, active vertex group, selection mode).
This behavior was initially introduced to avoid creating extra undo
steps when painting or sculpting and changing tool settings, however,
it applied to all property changes performed while in any paint mode,
including changes to mesh datablock properties.
To fix this behavior, this commit removes the usage of the sculpt undo
system from Vertex Paint and removes it from the list of modes that
are incompatible with memfile undo.
This switch from Sculpt Undo to Memfile Undo improves the performance of
the undo operator from 90ms to 9ms, representing a 10x speedup.
This speedup is possible because Vertex Paint is not fully integrated
with the Paint BVH drawing system, which normally has the benefit of
skipping depsgraph reevaluation and subsequent BVH rebuilding on each
stroke.
Anecdotally, when painting with large strokes on a mesh of 2 million
vertices, this change to using Memfile undo also uses anywhere from
half as much to a third as much memory compared to the Sculpt Undo
implementation (6 GB vs 2 GB).
---
Similar to !135758
### Implementation Notes / Thoughts
An alternate approach to getting similar if not better performance in Vertex Paint mode is to integrate it with `BKE_sculptsession_use_pbvh_draw` and `flush_update_step`, however this represents an unknown amount of work and does not reduce the number of consumers of Sculpt Undo. Given that the general direction in the module right now is to reduce dependency upon this system and migrate more to using Memfile undo where possible, spending extra effort on integration does not seem like the correct decision to make.
### Testing
* Verified undo step is created when selection mode is changed
* Verified undo step is created when symmetry is changed
* Verified undo step is created when active color attribute is changed
* Verified undo step is created when color attribute underlying data is changed
Pull Request: https://projects.blender.org/blender/blender/pulls/135830
12bb22a130