There was actually two different issues:
* mode and toggle options had "remanant" values, when e.g. you used MODE_EDIT/toggle, these would be used again with the newly defined keymap (which sets eg MODE_OBJECT/notoggle)... Defining those props as PROP_SKIP_SAVE fixed this.
* Toggling was not supported for MODE_OBJECT, fixed this (can indeed be handy to have a shortcut to toggle between current mode and Object one ;) ).