There was some remaining issue caused by neighbor re-use. Randomizing them every _prime_number_ of iterations fixes this.