Part of #131825
Currently, no change to an RNA property invoked from the UI while inside
Vertex Paint mode creates an undo step. This includes both tool
properties (e.g. brush size, tool settings) as well as mesh properties
(symmetry, active vertex group, selection mode).
This behavior was initially introduced to avoid creating extra undo
steps when painting or sculpting and changing tool settings, however,
it applied to all property changes performed while in any paint mode,
including changes to mesh datablock properties.
To fix this behavior, this commit removes the usage of the sculpt undo
system from Vertex Paint and removes it from the list of modes that
are incompatible with memfile undo.
This switch from Sculpt Undo to Memfile Undo improves the performance of
the undo operator from 90ms to 9ms, representing a 10x speedup.
This speedup is possible because Vertex Paint is not fully integrated
with the Paint BVH drawing system, which normally has the benefit of
skipping depsgraph reevaluation and subsequent BVH rebuilding on each
stroke.
Anecdotally, when painting with large strokes on a mesh of 2 million
vertices, this change to using Memfile undo also uses anywhere from
half as much to a third as much memory compared to the Sculpt Undo
implementation (6 GB vs 2 GB).
Pull Request: https://projects.blender.org/blender/blender/pulls/135830
e1d3538980