This was caused by the shadow map Z range loosing too much
precision on the large object.
This can be replicated by adding a very large object in the scene.
The normaly un-shadowed objects gets shadowed because the
float precision is not enough to represent their depth correctly.
To fix this, we bias the shadow depth by 2 ULP to make sure that
there is no self shadowing.
This bias has already been there, but was somehow removed because
of other fixes that simingly fixed this issue. But they did not.
Pull Request: https://projects.blender.org/blender/blender/pulls/137555
a798c48d0d