When snapping to edge/vert, check the distance to the ray
instead of the screen-space pixel projection.
This also corrects the conversion of `dist_to_ray_sq` to `dist_px` which was being done incorrectly.
While this change was planned, it fixes T48791, caused by error in b01a56ee.