There are two parts for this PR. One is to change some of our build pipeline to make certain libs reproducible. For this part I want to clarify two things:
1. Why change python to use `--disable-optimizations`?
This is because `--enable-optimizations` turns on PGO (Profile Guided Optimization). PGO is sadly not deterministic and will create different binaries on every recompile. So to create reproducible build this needs to be turned off. This also seems to only have been turned on for Linux specifically(?) on our side. So on Windows and Mac our python build already doesn't have PGO.
2. Why split out cython and zstandard from site-packages?
Sadly pip does not seem to respect `SOURCE_DATE_EPOCH`. It also creates temporary folders with random hashes in them that is then recorded into the Cython libraries (I'll touch on this again later). I've looked at the discussions about this upstream and sadly the pip maintainers do not really want people to use pip as a reproducible build system pipeline and instead directs users to other solutions if they want reproducible builds.
The other part is about setting up our pipeline to not introduce any random hashes or build timestamps into our libraries. Here I do two things:
1. We need to set the `SOURCE_DATE_EPOCH` environmental variable to a specific date that will not change.
This is needed as the compile time date is recorded in certain libraries and files. (So hard coding it with this env var will make the end result reproducible)
2. We need to strip the created static and shared libraries. This is because the static libraries are not created in a deterministic way. For shared libraries some of our libraries includes debug symbols which contains paths to temporary files with random hashes. To solve this without stripping in post, we would need to either patch the linker on Rocky8 or patch a lot of our libraries. I think it is better to just do this as a post build step. (This seems to be what most linux distributions do as well).
With all this, we can make our Linux library builds is almost 100% reproducible. (At least on my machine where I tested)
By almost, I mean that there is sadly a catch in that certain libraries like Cython saves the source code path in their libraries for error messages. However now the builds are reproducible if the folder path is the same.
IE if the libraries are always built in `/home/builder/build_linux/deps_x64`, then they should now be reproducible.
Pull Request: https://projects.blender.org/blender/blender/pulls/134221
* VS2022 is required.
* Only OpenPGL and DPCPP are disabled, other libraries are supported.
* Embree is built with LLVM and VS2019 tools, and for that reason has
its own cmake file as it is quite different.
* TBB and USD patches should become obsolete once these are upstreamed
and Blender upgrades to the latest versions.
Ref #119126
Pull Request: https://projects.blender.org/blender/blender/pulls/117036
- Commands which have arguments split over multiple lines use
indented lines.
- Wrap lines where multiple commands run using "&&".
- Blank lines between multiple commands helps the text from becoming
too dense.
Listing the "Blender Foundation" as copyright holder implied the Blender
Foundation holds copyright to files which may include work from many
developers.
While keeping copyright on headers makes sense for isolated libraries,
Blender's own code may be refactored or moved between files in a way
that makes the per file copyright holders less meaningful.
Copyright references to the "Blender Foundation" have been replaced with
"Blender Authors", with the exception of `./extern/` since these this
contains libraries which are more isolated, any changed to license
headers there can be handled on a case-by-case basis.
Some directories in `./intern/` have also been excluded:
- `./intern/cycles/` it's own `AUTHORS` file is planned.
- `./intern/opensubdiv/`.
An "AUTHORS" file has been added, using the chromium projects authors
file as a template.
Design task: #110784
Ref !110783.
This is used for most Python release builds and has been reported to
give a modest 5-10% speedup (depending on the workload).
This could be enabled on macOS too but needs to be tested.
- Added comment for additional tasks to do when bumping python
- updated sqlite to 3.39.4
- downgrade setuptools to 63.2.0 to avoid numpy build issues
- numpy 1.23.5
This updates the libraries dependencies for VFX platform 2023, and adds various
new libraries. It also enables Python bindings and switches from static to
shared for various libraries.
The precompiled libraries for all platforms will be updated to these new
versions in the coming weeks.
New:
Fribidi 1.0.12
Harfbuzz 5.1.0
MaterialX 1.38.6 (shared lib with python bindings)
Minizipng 3.0.7
Pybind11 2.10.1
Shaderc 2022.3
Vulkan 1.2.198
Updated:
Boost 1.8.0 (shared lib)
Cython 0.29.30
Numpy 1.23.2
OpenColorIO 2.2.0 (shared lib with python bindings)
OpenImageIO 2.4.6.0 (shared lib with python bindings)
OpenSubdiv 3.5.0
OpenVDB 10.0.0 (shared lib with python bindings)
OSL 1.12.7.1 (enable nvptx backend)
TBB (shared lib)
USD 22.11 (shared lib with python bindings, enable hydra)
yaml-cpp 0.8.0
Includes contributions by Ray Molenkamp, Brecht Van Lommel, Georgiy Markelov
and Campbell Barton.
Ref T99618
THis is bumping dependencies to fix known CVEs, with the exception of
OpenImageIO which also includes bugfixes for performance and correctness
with some image types.
zlib 1.2.12 -> 1.2.13
freetype 2.11.1 -> 2.12.1
openimageio 2.3.13.0 -> 2.3.20.0
python 3.10.2 -> 3.10.8
openjpeg 2.4.0 -> 2.5.0
ffmpeg 5.0 -> 5.1.2
sndfile 1.0.28 -> 1.1.0
xml2 2.9.10 -> 2.10.3
expat 2.4.4 -> 2.4.9
openssl 1.1.1g/i -> 1.1.1q
sqlite 3.31.1 -> 3.37.2
Notable changes:
* AOM: the hack we had in place to make it not detect pthreads on windows no
longer worked with a more recent cmake version. Disabled pthreads with a
diff on Windows.
* Python: embedded copy of zlib 2.1.12 swapped out for our 2.1.13 copy with
some folder manipulation on Windows.
* Freetype: was harbouring a copy of zlib 2.1.12 as well, so that had to end.
* FFmpeg: patch used to fix D11796 is no longer needed. Add new patch to deal
with simple_idct.asm generating an object file with no sections in it,
backport from upstream commit.
* TinyXML: still being downloaded but no longer used by OpenColorIO, removed.
* GMP applied upstream patch to fix CVE-2021-43618, as there is no release yet.
* SQLite and Libsndfile patches no longer needed.
Includes contributes by Ray Molenkamp, Campbell Barton and Brecht Van Lommel.
Ref T101403
Differential Revision: https://developer.blender.org/D16269
Use a shorter/simpler license convention, stops the header taking so
much space.
Follow the SPDX license specification: https://spdx.org/licenses
- C/C++/objc/objc++
- Python
- Shell Scripts
- CMake, GNUmakefile
While most of the source tree has been included
- `./extern/` was left out.
- `./intern/cycles` & `./intern/atomic` are also excluded because they
use different header conventions.
doc/license/SPDX-license-identifiers.txt has been added to list SPDX all
used identifiers.
See P2788 for the script that automated these edits.
Reviewed By: brecht, mont29, sergey
Ref D14069
Allow downloading of source packages of Blender's dependencies, so that
it's easier to provide a "full source archive" that contains the blender
source + all dependencies archives. A `make` command for this will be
introduced soon.
This changes the deps builder slightly to be more flexible with the
origin of our source packages.
To support this a new CMake variable has been added called `PACKAGE_DIR`
where all sources archives will be stored.
default: a directory called `packages` in the build folder.
alternative-default: if a directory called `packages` exists in the
blender source folder that will be used. This is to support the "full
source archive" use case.
The download phase have been moved from the build phase to the configure
phase. Configure will download all sources validate the hashes while
downloading.
All `[depname].cmake` files have been changed to take a local
`file://[path_to_local_tarball]` path rather than a remote URI.
A second requirement was that there needed to be an option to grab the
sources from the blender SVN mirror rather than upstream. For this an
option has been added PACKAGE_USE_UPSTREAM_SOURCES (default ON). The
exact location in SVN still needs to be worked out, I tested with my
local webserver and codewise it checks out. The path that is in there
currently will not work (given there is no mirror there yet).
To build this mirror our local package caches can be used.
Reviewed By: lazydodo
Differential Revision: https://developer.blender.org/D10598
rBc70eb30240f8b5d5a8f2ac509f0eb585936142b5 added patch to use static
zlib on Linux. But also added flags to use the zlib in LIBDIR for
Python on macOS. That causes some shared libraries (binascii for one)
to link against libz.1.dylib which will not be there on users' systems.
Reuse the said patch for macOS also to avoid rpath issues.
Fix T85648.
Reviewed By: #platform_macos, brecht
Differential Revision: https://developer.blender.org/D10479
The following packages also have received updates:
- IDNA 2.10
- CHARDET 4.0.0
- URLLIB3 1.26.3
- CERTIFI 2020.12.5
- REQUESTS 2.25.1
- NUMPY 1.19.5
numpy has gained a hard dependency on cython:
- CYTHON 0.29.21
Notes:
- This only updates the build environment files,
once these are built, Blender can default to Python 3.9.
- The 'm' suffix for Python binaries/libs has been removed.
- The macOS patch in Python 3.7 is has been removed.
Reviewed By: sybren, campbellbarton, sebbas
Ref D10257
Enabling all `make deps` dependencies with the exception of Embree and OIDN.
After that, Blender can be compiled on an Apple Silicon Mac just like on any
Intel based Mac. There are still compiler warnings that need to be
investigated and there are probably a couple of bug still to be discovered
and to be fixed.
Most patches to the dependencies are simple and are about disabling SSE and
setting the proper architecture to compiile for. Notable exception is Python,
where I back ported a yet to be accepted PR for upstream Python:
https://github.com/python/cpython/pull/21249
Cross compiling or buliding a Universal Binary is not supported yet.
The minimum macOS target version for x86_64 remains at 10.13, the target
for arm64 is 11.00.
Differential Revision: https://developer.blender.org/D8236
The configuration was confused about gettext installed via Homebrew
and isysroot passed to Python's compilation but not to test programs.
After this change `import gettext` still works, but it is unclear how
to test it further,
Differential Revision: https://developer.blender.org/D8231
Mainly this is following Linux to build own xml2/lzma/ssl/sqlite and linking
them all statically. This ensures the Python ssl module uses a recent openssl
version rather than a very old one shipped with macOS.
There is system-wide libz development package installed by default,
needed for some other dependencies. This patch ensures Python will
use our own self-compiled Zlib.
This involved getting SSL compiled from sources first, ensuring
it is a static library placement independent code. Configuration
is based on what Debian is using. CFlags required to have own
configuration file, which i didn't find a better place that next
to the corresponding CMake file.
It is OpenSSL btw.
It is set to Python via --with-openssl= configuration argument.
This works fine in a clean chroot, but having libssl-dev installed
might make Python to prefer system wide library, This was worked
around by using libssl_pic.a name for the library and modifying
setup.py. Would be cool to ensure system wide libraries are not
a problem, but official release builder is safe against this,
since it will catch possible non-static dependencies.
There is also a new map file which shadows bunch of Python
symbols. Without this Python's shared libraries might bring
conflicting symbols to Blender namespace at runtime.
Hopefully this doesn't break other platforms.
-No more hardcoded python35/36 tokens in the scripts
-disabled python module for boost, was not used
-Updated patches for python to support building with msvc2013
