The problem was that orthographic views can have hit position that are negative. Thus we cannot encode the hit in the sign of the Z component.
The workaround is to store the hit position in screenspace. But since we are using floating point render target, we are loosing quite a bit of precision.
TODO: use RGBA16 instead of RGBA16F. But that means encoding the pdf value somehow.
98dd2a518b