Original optimization idea was wrong: it is possible that some other
ID would reference an object which is also used by a base.
Rolled back to a bit more fragile solution.
In the future would be nice to make it somewhat less duplicated with
the builder itself.
Fixes assert failure (and possibly crashes) when adding grease pencil
object and switching to a draw mode.
95b150ba87