No need to assume it's 2D or 3D.
The previous behavior here was wrong for some specific combinations of settings, non-color RGB channels should never be affected by the alpha channel.
This is the groundwork for supporting loading image textures with arbitrary color spaces through OpenColorIO. Initial patch by Lukas, completed by Brecht. Differential Revision: https://developer.blender.org/D3491