This is a bit annoying, but FFmpeg can't deal with unaligned arrays in some cases. There seems to be an easy workaround with using SWS_ACCURATE_RND flag which should be harmless since we don't really scaling frames, but only using sws_scale to change color space. Additionally, this is what VLC is also using.