Only add each vertices to point ibo once. That requires tagging each mvert in the case of modifier preview.