This was suggested by @zanqdo on IRC. Hiding sockets is a user choice (not to be confused with "unavailable" disabled sockets). Hidden sockets suddenly popping up when linking is confusing and intransparent, better just ignore them for the swapping.