This should already be happening elsewhere and was causing depsgraph rebuild to be slower than it should.