Issue was caused by missing X/Y displacement components
flip when flipping the normals (flipping the normals changes
the tangent space apparently and displacement vectors need
to be modified to correspond to new space).
Reported by Jonathan Williamson in IRC.