The issue was caused by the threading conflict between main thread which
might free environment map and render thread which might be using that
environment map.
Solved by stopping preview jobs when changing environment map.
Sure there are still ways to run into threading conflict, but would rather
solve this things case-by-case.