Rotated coordinate of the ray start was used when calculating the ray direction, ending up with wrong direction.