This was caused by the input matrix not being used. Instead EEVEE was trying to derive the projection matrix again based on the actual viewport state. But this state was not reflecting the input matrix. To fix this, we remove any camera setup from the rv3d during the offscreen drawing and set the lens param to 0. This ensure EEVEE loads the input projection matrix.