The Texture margin 'adjacent faces' algorithm always used the full UV map, even if not all polygons were actually part of the baking. Remedy this by checking the mask if passed in. Pull Request: https://projects.blender.org/blender/blender/pulls/109500