This makes more sense to have no displacement if baking
non-normalized displacement map or have mid-level displacement
(which is likely be close to no displacement as well) when
baking normalized displacement.
In any way it's better than using default black color.