This was caused by small float precision being insuficient. The blue component of R11F_G11F_B10F has lower precision than the other 2 components. This resulted in colors drifting towards a yellowish tone.
Using RGBA16F for the concerned buffer. This double the memory usage of the framebuffers and add subsequent bandwidth usage.
9f3c7c75b7