This was caused by motion blur camera movement tagging the view as invalid and thus resetting the temporal sampling. Critical fix for 2.90. Need second look, but quite confident. This function is only called once when Motion blur is off. Reviewed by: jbakker Differential Revision: https://developer.blender.org/D8676