There are two parts for this PR. One is to change some of our build pipeline to make certain libs reproducible. For this part I want to clarify two things:
1. Why change python to use `--disable-optimizations`?
This is because `--enable-optimizations` turns on PGO (Profile Guided Optimization). PGO is sadly not deterministic and will create different binaries on every recompile. So to create reproducible build this needs to be turned off. This also seems to only have been turned on for Linux specifically(?) on our side. So on Windows and Mac our python build already doesn't have PGO.
2. Why split out cython and zstandard from site-packages?
Sadly pip does not seem to respect `SOURCE_DATE_EPOCH`. It also creates temporary folders with random hashes in them that is then recorded into the Cython libraries (I'll touch on this again later). I've looked at the discussions about this upstream and sadly the pip maintainers do not really want people to use pip as a reproducible build system pipeline and instead directs users to other solutions if they want reproducible builds.
The other part is about setting up our pipeline to not introduce any random hashes or build timestamps into our libraries. Here I do two things:
1. We need to set the `SOURCE_DATE_EPOCH` environmental variable to a specific date that will not change.
This is needed as the compile time date is recorded in certain libraries and files. (So hard coding it with this env var will make the end result reproducible)
2. We need to strip the created static and shared libraries. This is because the static libraries are not created in a deterministic way. For shared libraries some of our libraries includes debug symbols which contains paths to temporary files with random hashes. To solve this without stripping in post, we would need to either patch the linker on Rocky8 or patch a lot of our libraries. I think it is better to just do this as a post build step. (This seems to be what most linux distributions do as well).
With all this, we can make our Linux library builds is almost 100% reproducible. (At least on my machine where I tested)
By almost, I mean that there is sadly a catch in that certain libraries like Cython saves the source code path in their libraries for error messages. However now the builds are reproducible if the folder path is the same.
IE if the libraries are always built in `/home/builder/build_linux/deps_x64`, then they should now be reproducible.
Pull Request: https://projects.blender.org/blender/blender/pulls/134221
Bump autopep8 as there has been various improvements & bug-fixes.
Also update pycodestyle 2.8 -> 2.12.1.
autopep8 imported a module "toml" which has been superseded by
Python's own "tomllib". The "toml" module has been removed.
Ref !127553
Listing the "Blender Foundation" as copyright holder implied the Blender
Foundation holds copyright to files which may include work from many
developers.
While keeping copyright on headers makes sense for isolated libraries,
Blender's own code may be refactored or moved between files in a way
that makes the per file copyright holders less meaningful.
Copyright references to the "Blender Foundation" have been replaced with
"Blender Authors", with the exception of `./extern/` since these this
contains libraries which are more isolated, any changed to license
headers there can be handled on a case-by-case basis.
Some directories in `./intern/` have also been excluded:
- `./intern/cycles/` it's own `AUTHORS` file is planned.
- `./intern/opensubdiv/`.
An "AUTHORS" file has been added, using the chromium projects authors
file as a template.
Design task: #110784
Ref !110783.
Having all packages on one line made reviewing changes difficult.
Also note why Python modules are needed (with some TODO's where I wasn't
able to find any reason given for their inclusion).
- Added comment for additional tasks to do when bumping python
- updated sqlite to 3.39.4
- downgrade setuptools to 63.2.0 to avoid numpy build issues
- numpy 1.23.5
This updates the libraries dependencies for VFX platform 2023, and adds various
new libraries. It also enables Python bindings and switches from static to
shared for various libraries.
The precompiled libraries for all platforms will be updated to these new
versions in the coming weeks.
New:
Fribidi 1.0.12
Harfbuzz 5.1.0
MaterialX 1.38.6 (shared lib with python bindings)
Minizipng 3.0.7
Pybind11 2.10.1
Shaderc 2022.3
Vulkan 1.2.198
Updated:
Boost 1.8.0 (shared lib)
Cython 0.29.30
Numpy 1.23.2
OpenColorIO 2.2.0 (shared lib with python bindings)
OpenImageIO 2.4.6.0 (shared lib with python bindings)
OpenSubdiv 3.5.0
OpenVDB 10.0.0 (shared lib with python bindings)
OSL 1.12.7.1 (enable nvptx backend)
TBB (shared lib)
USD 22.11 (shared lib with python bindings, enable hydra)
yaml-cpp 0.8.0
Includes contributions by Ray Molenkamp, Brecht Van Lommel, Georgiy Markelov
and Campbell Barton.
Ref T99618
D14686 added autopep8 which implicitly dragged in
toml and pycodestyle which were not versioned, this
diff adds explicit versions of these deps so there
won't be any version changes if we rebuild in the
future.
Reviewed By: brecht, sybren
Differential Revision: https://developer.blender.org/D14793
Use a shorter/simpler license convention, stops the header taking so
much space.
Follow the SPDX license specification: https://spdx.org/licenses
- C/C++/objc/objc++
- Python
- Shell Scripts
- CMake, GNUmakefile
While most of the source tree has been included
- `./extern/` was left out.
- `./intern/cycles` & `./intern/atomic` are also excluded because they
use different header conventions.
doc/license/SPDX-license-identifiers.txt has been added to list SPDX all
used identifiers.
See P2788 for the script that automated these edits.
Reviewed By: brecht, mont29, sergey
Ref D14069
This package allows Python scripts to handle compressed blend files (see
rB2ea66af742bc). This is for example needed by Blender Asset Tracer to
send files to a Flamenco render farm.
This change includes a new `WITH_PYTHON_INSTALL_ZSTANDARD` build-time
option, to control whether to actually install the package. For this the
already-existing approach for Requests was copied.
Reviewed By: LazyDodo, mont29, brecht
Differential Revision: https://developer.blender.org/D12777
certifi : 2020.12.5 → 2021.10.8
chardet : 4.0.0 → charset-normalizer 2.0.6
cython : 0.29.21 → 0.29.24
idna : 2.10 → 3.2
numpy : 1.19.5 → 1.21.2 (which makes it possible to remove our patch)
requests: 2.25.1 → 2.26.0
urllib3 : 1.26.3 → 1.26.7
Nowadays `requests` no longer depends on `chardet` but on
`charset-normalizer`. That project describes itself as:
> A library that helps you read text from an unknown charset encoding.
> Motivated by chardet, I'm trying to resolve the issue by taking a new
> approach. All IANA character set names for which the Python core library
> provides codecs are supported.
Reviewed By: LazyDodo, mont29, brecht
Differential Revision: https://developer.blender.org/D12777
Building NumPy from source with default options of builder
causes it to link against Accelerate framework which is buggy and
raises a warning mentioned in [2].
"RankWarning: Polyfit may be poorly conditioned"
Accelerate is deprecated with NumPy 1.20+.[1]
So either we build OpenBLAS in dependencies also and set appropriate
env variables suggested in [1] while building NumPy for it to find
OpenBLAS. Or download NumPy wheel from pip and never allow pip to
build NumPy from source while installing.
After this change, pip wheels are used for NumPy for macOS with x86_64.
[1] https://numpy.org/doc/stable/user/building.html#lapack
[2] https://github.com/numpy/numpy/issues/15947
Reviewed By: #platform_macos, sebbas
Differential Revision: https://developer.blender.org/D10368
For the debug version of cython pip was trying to link
against the release version of python for some strange
reason. Passing some flags to explicitly target the
debug version fixes the issue.
Given other platforms do not have different builds for
debug/release this is not an issue there.
The following packages also have received updates:
- IDNA 2.10
- CHARDET 4.0.0
- URLLIB3 1.26.3
- CERTIFI 2020.12.5
- REQUESTS 2.25.1
- NUMPY 1.19.5
numpy has gained a hard dependency on cython:
- CYTHON 0.29.21
Notes:
- This only updates the build environment files,
once these are built, Blender can default to Python 3.9.
- The 'm' suffix for Python binaries/libs has been removed.
- The macOS patch in Python 3.7 is has been removed.
Reviewed By: sybren, campbellbarton, sebbas
Ref D10257
