Toggle support for brushes was added back in 4.5 with
9e1e9b0859. The intent of this feature is
to allow for easy switching between a specified brush upon using a
hotkey.
Whils this behavior works well in the case of switching between brushes,
the initial implementation didn't account for using other non-brush
tools.
To fix this issue, when activating a tool, if it doesn't handle brushes,
clear the last active brush.
Pull Request: https://projects.blender.org/blender/blender/pulls/141161