6d442ca4f1
There are two parts for this PR. One is to change some of our build pipeline to make certain libs reproducible. For this part I want to clarify two things: 1. Why change python to use `--disable-optimizations`? This is because `--enable-optimizations` turns on PGO (Profile Guided Optimization). PGO is sadly not deterministic and will create different binaries on every recompile. So to create reproducible build this needs to be turned off. This also seems to only have been turned on for Linux specifically(?) on our side. So on Windows and Mac our python build already doesn't have PGO. 2. Why split out cython and zstandard from site-packages? Sadly pip does not seem to respect `SOURCE_DATE_EPOCH`. It also creates temporary folders with random hashes in them that is then recorded into the Cython libraries (I'll touch on this again later). I've looked at the discussions about this upstream and sadly the pip maintainers do not really want people to use pip as a reproducible build system pipeline and instead directs users to other solutions if they want reproducible builds. The other part is about setting up our pipeline to not introduce any random hashes or build timestamps into our libraries. Here I do two things: 1. We need to set the `SOURCE_DATE_EPOCH` environmental variable to a specific date that will not change. This is needed as the compile time date is recorded in certain libraries and files. (So hard coding it with this env var will make the end result reproducible) 2. We need to strip the created static and shared libraries. This is because the static libraries are not created in a deterministic way. For shared libraries some of our libraries includes debug symbols which contains paths to temporary files with random hashes. To solve this without stripping in post, we would need to either patch the linker on Rocky8 or patch a lot of our libraries. I think it is better to just do this as a post build step. (This seems to be what most linux distributions do as well). With all this, we can make our Linux library builds is almost 100% reproducible. (At least on my machine where I tested) By almost, I mean that there is sadly a catch in that certain libraries like Cython saves the source code path in their libraries for error messages. However now the builds are reproducible if the folder path is the same. IE if the libraries are always built in `/home/builder/build_linux/deps_x64`, then they should now be reproducible. Pull Request: https://projects.blender.org/blender/blender/pulls/134221
45 lines
1.5 KiB
CMake
45 lines
1.5 KiB
CMake
# SPDX-FileCopyrightText: 2017-2023 Blender Authors
|
|
#
|
|
# SPDX-License-Identifier: GPL-2.0-or-later
|
|
|
|
if(WIN32 AND BUILD_MODE STREQUAL Debug)
|
|
set(SITE_PACKAGES_EXTRA --global-option build --global-option --debug)
|
|
# zstandard is determined to build and link release mode libs in a debug
|
|
# configuration, the only way to make it happy is to bend to its will
|
|
# and give it a library to link with.
|
|
set(
|
|
PIP_CONFIGURE_COMMAND ${CMAKE_COMMAND} -E copy
|
|
${LIBDIR}/python/libs/python${PYTHON_SHORT_VERSION_NO_DOTS}_d.lib
|
|
${LIBDIR}/python/libs/python${PYTHON_SHORT_VERSION_NO_DOTS}.lib
|
|
)
|
|
else()
|
|
set(PIP_CONFIGURE_COMMAND echo ".")
|
|
endif()
|
|
|
|
ExternalProject_Add(external_python_site_packages
|
|
DOWNLOAD_COMMAND ""
|
|
CONFIGURE_COMMAND ${PIP_CONFIGURE_COMMAND}
|
|
BUILD_COMMAND ""
|
|
PREFIX ${BUILD_DIR}/site_packages
|
|
|
|
# setuptools is downgraded to 63.2.0 (same as python 3.10.8) since numpy 1.23.x seemingly has
|
|
# issues building on windows with the newer versions that ships with python 3.10.9+.
|
|
# We do not build numpy, cython, or zstandard here as the pip builds are not reproducible.
|
|
INSTALL_COMMAND ${PYTHON_BINARY} -m pip install --no-cache-dir ${SITE_PACKAGES_EXTRA}
|
|
setuptools==63.2.0
|
|
idna==${IDNA_VERSION}
|
|
charset-normalizer==${CHARSET_NORMALIZER_VERSION}
|
|
urllib3==${URLLIB3_VERSION}
|
|
certifi==${CERTIFI_VERSION}
|
|
requests==${REQUESTS_VERSION}
|
|
autopep8==${AUTOPEP8_VERSION}
|
|
pycodestyle==${PYCODESTYLE_VERSION}
|
|
meson==${MESON_VERSION}
|
|
--no-binary :all:
|
|
)
|
|
|
|
add_dependencies(
|
|
external_python_site_packages
|
|
external_python
|
|
)
|