Fix #132528: Mask by color crashes when used outside of mesh

Caused by a0e6e16da5

In cases where the cursor is not over the mesh, the operator needs to
check for and prevent further operation to avoid accessing incorrect
variant data.

In prior versions, while this operator did not crash, it would read
other custom layer data.

Pull Request: https://projects.blender.org/blender/blender/pulls/132538

source/blender/editors/sculpt_paint/sculpt_ops.cc

@@ -828,6 +828,10 @@ static int mask_by_color_invoke(bContext *C, wmOperator *op, const wmEvent *even
     return OPERATOR_CANCELLED;
   }
 
+  if (std::holds_alternative<std::monostate>(ss.active_vert())) {
+    return OPERATOR_CANCELLED;
+  }
+
   BKE_sculpt_update_object_for_edit(depsgraph, &ob, false);
 
   /* Tools that are not brushes do not have the brush gizmo to update the vertex as the mouse move,