Was caused by 0 pixel extent rays which did not trace at all. Bumping the threshold for degenerate rays fixes it.