Build: Make Linux Lib building reproducible

There are two parts for this PR. One is to change some of our build pipeline to make certain libs reproducible. For this part I want to clarify two things:

1. Why change python to use `--disable-optimizations`?
This is because `--enable-optimizations` turns on PGO (Profile Guided Optimization). PGO is sadly not deterministic and will create different binaries on every recompile. So to create reproducible build this needs to be turned off. This also seems to only have been turned on for Linux specifically(?) on our side. So on Windows and Mac our python build already doesn't have PGO.

2. Why split out cython and zstandard from site-packages?
Sadly pip does not seem to respect `SOURCE_DATE_EPOCH`. It also creates temporary folders with random hashes in them that is then recorded into the Cython libraries (I'll touch on this again later). I've looked at the discussions about this upstream and sadly the pip maintainers do not really want people to use pip as a reproducible build system pipeline and instead directs users to other solutions if they want reproducible builds.

The other part is about setting up our pipeline to not introduce any random hashes or build timestamps into our libraries. Here I do two things:

1. We need to set the `SOURCE_DATE_EPOCH` environmental variable to a specific date that will not change.
This is needed as the compile time date is recorded in certain libraries and files. (So hard coding it with this env var will make the end result reproducible)

2. We need to strip the created static and shared libraries. This is because the static libraries are not created in a deterministic way. For shared libraries some of our libraries includes debug symbols which contains paths to temporary files with random hashes. To solve this without stripping in post, we would need to either patch the linker on Rocky8 or patch a lot of our libraries. I think it is better to just do this as a post build step. (This seems to be what most linux distributions do as well).

With all this, we can make our Linux library builds is almost 100% reproducible. (At least on my machine where I tested)
By almost, I mean that there is sadly a catch in that certain libraries like Cython saves the source code path in their libraries for error messages. However now the builds are reproducible if the folder path is the same.
IE if the libraries are always built in `/home/builder/build_linux/deps_x64`, then they should now be reproducible.

Pull Request: https://projects.blender.org/blender/blender/pulls/134221
This commit is contained in:
Sebastian Parborg
2025-05-09 15:25:16 +02:00
committed by Sebastian Parborg
parent fd3ca68b5e
commit 6d442ca4f1
12 changed files with 188 additions and 10 deletions

View File

@@ -428,6 +428,8 @@ ifneq "$(findstring clean, $(MAKECMDGOALS))" ""
DEPS_TARGET = clean
endif
# Set the SOURCE_DATE_EPOCH to make builds reproducible (locks timestamps to the specified date).
deps: export SOURCE_DATE_EPOCH = 1745584760
deps: .FORCE
@echo
@echo Configuring dependencies in \"$(DEPS_BUILD_DIR)\", install to \"$(DEPS_INSTALL_DIR)\"