This crash can only happen in case faces in same 'smooth fan' have reversed normals.
To support this, we have to always keep a way to get real values in loop_to_poly,
even when loop itself is tagged as done, it might be needed in computation of one of
its neighbor's split normal later.