Even though the brush rotation is computed as a 2D angle (based on the mouse
movement), it currently gets applied by rotating the projected X direction
around the the normal in 3D.
This patch ensures that rotation gets applied first, and only then does the
motion direction get projected into the tangent plane. A potential issue with
the current approach is that the random perturbations will also be applied in
2D, but this seems to be fine from discussions with @JulienKaspar and @Sergey.
Also, there was an error where the location should probably be converted *to*
world coordinates.
All these changes seem to fix the issue described in #116418.
I also noticed some minor "inconsistencies" with how the rotation is applied:
For curve strokes, the direction of the curve corresponded to the upward
direction of the brush. For view plane, area plane mapping and anchored strokes,
the mouse motion indicated the downward direction of the brush.
For compatibility, I tried my best to enforce the latter conventions throughout,
but I'm not so confident about oversights.
Pull Request: https://projects.blender.org/blender/blender/pulls/116539
d16543a155