While it's useful for click-drag to leave the selection as-is (when clicking on items that are already selected), it's useful for a single click to de-select all other elements. This also removes the need for the initial selection to set the object as active since this is possible by clicking on it.