For some compiler and linker configurations, linking would fail as the
wayland libs were linked at a high level and not at the actual code
where they were needed.
After talking to Campbell, we decided to clean up this part and now
only link both the X11 and Wayland libs where they are used.
c425599216