While it's useful for click-drag to leave the selection as-is
(when clicking on items that are already selected), it's useful
for a single click to de-select all other elements.
This also removes the need for the initial selection to set the object
as active since this is possible by clicking on it.