The issue was that the depsgraph was not rebuilt, thus the driver node still stuck around. This then crashed when the depsgraph evaluated. The reason why this wasn't caught in the unit tests, was because the depsgraph was not updated between creating and removing the data. Pull Request: https://projects.blender.org/blender/blender/pulls/141272