The normals were inverted because they use geometric normals. These are computed using partial derivatives (`dFdx` & `dFdy`) and are dependent on the view orientation. But the baking views were not all captured from the same side, which flipped sufels normals in 2 out of the 3 views.