The issue was that when a 4x4 matrix is normalized, it does not always mean that any inner 3x3 matrix is normalized too.