Some Linux multi lib setups have a helper include file for Jemalloc that
in turn includes the actual header file. This makes our version regex fail.
As the Jemalloc version we are checking for is no longer in any of
the currently supported LTS linux distros, we can safely drop it.
Pull Request: https://projects.blender.org/blender/blender/pulls/139225
on a freshly installed environment diff.exe requires newer runtime
dll's. For now only fix this on X64 as on windows on arm there's
issues with newer msys2 versions (but I do now know exactly what
they are anthony has brought it up in the past) this can however
not remain in a broken state, so fixing it for X64 only.
Also fix typo in download_package function
These two fixes together should bring a fresh vmprep install back
to working order.
While shaderc is a c++ library, its api is a C based one so
we do not need a debug version of this library, the blender
side was already updated in f2b9dd3462 this cleans up the
builder side of things.
OpenColorIO is now a dynamic library, and these are included in it. The
legacy code for the static library case was causing the Homebrew libraries
to be found and linked to.
Pull Request: https://projects.blender.org/blender/blender/pulls/139087
This new version brings several fixes that Blender no
longer needs to patch manually. In addition, it includes
an internal change related to GPU memory management which,
together with the new API, could address several tickets
and issues currently reported in relation to Embree GPU execution.
Pull Request: https://projects.blender.org/blender/blender/pulls/138176
Split out from #138161.
I checked with a locally built OSL (on Linux), and all tests (incl. OptiX OSL)
still pass without the Cycles-side changes in that PR, so we can merge this and
update the libs separately.
The only file that needs to be updated in the deps is `liboslexec.so`,
and probably `llvm/lib/clang/17/include/__clang_cuda_device_functions.h`
(we don't use this when building Blender, but since it's changed,
it's probably cleaner to update it anyways).
Pull Request: https://projects.blender.org/blender/blender/pulls/138788
Broken by PR#134221
- Couple of missing deps
- Windows is not happy with -J when building python deps
as multiple processes will try to write to the same .pdb
- Wrong .lib file name for the zstandard python module.
Pretty bare bones but gets the job done, unlike the gcc
tooling, this will work for release builds, the performance cost
of it is on the high side of things, the full test suite tests take over
an hour for me with code coverage support enabled on a release build.
I have not timed a debug build. Given developers can just run their
tests to get coverage data over what they are working on, I feel this
is still useful tooling to have.
This adds the 3 targets for clang and adds a single gcc target
coverage-reset - this removes the collected code coverage data and
report
coverage-report - This merges the collected data and generates the
report (new for gcc)
coverage-show - This merges the collected data and generates the report
and opens it in the browser
This relies on llvm-cov and llvm-profdata being available if not found
code coverage is disabled.
Note: A full test run requires an obscene amount of disk space, a
complete test run takes about 125GB and takes 12 minutes to merge, so
provision the COMPILER_CODE_COVERAGE_DATA_DIR folder accordingly
Example report in PR
There are two parts for this PR. One is to change some of our build pipeline to make certain libs reproducible. For this part I want to clarify two things:
1. Why change python to use `--disable-optimizations`?
This is because `--enable-optimizations` turns on PGO (Profile Guided Optimization). PGO is sadly not deterministic and will create different binaries on every recompile. So to create reproducible build this needs to be turned off. This also seems to only have been turned on for Linux specifically(?) on our side. So on Windows and Mac our python build already doesn't have PGO.
2. Why split out cython and zstandard from site-packages?
Sadly pip does not seem to respect `SOURCE_DATE_EPOCH`. It also creates temporary folders with random hashes in them that is then recorded into the Cython libraries (I'll touch on this again later). I've looked at the discussions about this upstream and sadly the pip maintainers do not really want people to use pip as a reproducible build system pipeline and instead directs users to other solutions if they want reproducible builds.
The other part is about setting up our pipeline to not introduce any random hashes or build timestamps into our libraries. Here I do two things:
1. We need to set the `SOURCE_DATE_EPOCH` environmental variable to a specific date that will not change.
This is needed as the compile time date is recorded in certain libraries and files. (So hard coding it with this env var will make the end result reproducible)
2. We need to strip the created static and shared libraries. This is because the static libraries are not created in a deterministic way. For shared libraries some of our libraries includes debug symbols which contains paths to temporary files with random hashes. To solve this without stripping in post, we would need to either patch the linker on Rocky8 or patch a lot of our libraries. I think it is better to just do this as a post build step. (This seems to be what most linux distributions do as well).
With all this, we can make our Linux library builds is almost 100% reproducible. (At least on my machine where I tested)
By almost, I mean that there is sadly a catch in that certain libraries like Cython saves the source code path in their libraries for error messages. However now the builds are reproducible if the folder path is the same.
IE if the libraries are always built in `/home/builder/build_linux/deps_x64`, then they should now be reproducible.
Pull Request: https://projects.blender.org/blender/blender/pulls/134221
Briefly about this change:
- OpenColorIO C-API is removed.
- The information about color spaces in ImBuf module is removed.
It was stored in global ListBase in colormanagement.cc.
- Both OpenColorIO and fallback implementation supports GPU drawing.
- Fallback implementation supports white point, RGB curves, etc.
- Removed check for support of GPU drawing in IMB.
Historically it was implemented in a separate library with C-API, this
is because way back C++ code needed to stay in intern. This causes all
sort of overheads, and even calls that are strictly considered bad
level.
This change moves OpenColorIO integration into a module within imbuf,
next to movie, and next to IMB_colormanagement which is the main user
of it. This allows to avoid copy of color spaces, displays, views etc
in the ImBuf: they were used to help quickly querying information to
be shown on the interface. With this change it can be stored in the
same data structures as what is used by the OpenColorIO integration.
While it might not be fully avoiding duplication it is now less, and
there is no need in the user code to maintain the copies.
In a lot of cases this change also avoids allocations done per access
to the OpenColorIO. For example, it is not needed anymore to allocate
image descriptor in a heap.
The bigger user-visible change is that the fallback implementation now
supports GLSL drawing, with the whole list of supported features, such
as curve mapping and white point. This should help simplifying code
which relies on color space conversion on GPU: there is no need to
figure out fallback solution in such cases. The only case when drawing
will not work is when there is some actual bug, or driver issue, and
shader has failed to compile.
The change avoids having an opaque type for color space, and instead
uses forward declaration. It is a bit verbose on declaration, but helps
avoiding unsafe type-casts. There are ways to solve this in the future,
like having a header for forward declaration, or to flatten the name
space a bit.
There should be no user-level changes under normal operation.
When building without OpenColorIO or the configuration has a typo or
is missing a fuller set of color management tools is applies (such as the
white point correction).
Pull Request: https://projects.blender.org/blender/blender/pulls/138433
While shaderc is a c++ library that normally requires debug libs
for ABI reasons, shaderc only exports a C interface, so it will
link release mode libs just fine even for debug builds.
This drops the time required for GPU_shader_compile_static in a
debug build from 5 mins to 3 sec for @pragma37
no changes for release configurations.
For the official GitHub mirror and other repositories that do not include LFS
files, this adds an `lfs-fallback` remote. It will be used automatically if a
file can't be found on the regular remote.
Ref #137215
Co-authored-by: Sergey Sharybin <sergey@blender.org>
Pull Request: https://projects.blender.org/blender/blender/pulls/137615
This change moves the tests data files and publish folder of assets
repository to the main blender.git repository as LFS files.
The goal of this change is to eliminate toil of modifying tests,
cherry-picking changes to LFS branches, adding tests as part of a
PR which brings new features or fixes.
More detailed explanation and conversation can be found in the
design task.
Ref #137215
Pull Request: https://projects.blender.org/blender/blender/pulls/137219
Enable optimizations on Debug builds for executables used for the build
process (datatoc and glsl_preprocess) and enable unity builds for
shader preprocessing targets.
Debug: From 28.9s to 5.7s
Release: From 4.9s to 3.5s
Pull Request: https://projects.blender.org/blender/blender/pulls/138274
See #129009 for context.
The preprocessor parses metadata and writes a header file containing
an inline function that inits the `GPUSource` with the metadata.
These header files are then included inside `gpu_shader_dependency.cc`.
This still keep the usage of the `metadata` enums and classes to avoid
pulling the whole blender module inside the preprocessor executable.
This speeds-up startup time in Debug build:
`gpu_shader_dependency_init`
- Before : 37ms
- After : 4ms
I didn't measure release, but it is unlikely to be noticeable (in the
order of 4ms > 1ms).
Pull Request: https://projects.blender.org/blender/blender/pulls/138070
Adds the 'manifold' solver option to the Boolean geo node and to
the Boolean modifier. This solver is about as fast, or faster,
than the current float solver, and is robust against floating
point issues like the Exact solver. But currently it only
works on mesh arguments that are strictly manifold.
See https://projects.blender.org/blender/blender/issues/120182
for many more details.
This patch changes the default macOS x64 linker from the slower legacy
"classic" linker, to the much faster default modern macOS linker. A new
CMake option `WITH_LEGACY_MACOS_X64_LINKER` was also introduced to use
the old linker, set to OFF by default.
As an example from testing on an Intel Macbook, this makes linking time
go from 3min25s to 10s after this change. The reason why the legacy
linker was enforced on x64 in the first place was to silence "platform
load command not found" warnings during linking. Since this behavior is
still desired on the BuildBot, the `WITH_LEGACY_MACOS_X64_LINKER` option
is enforced in the BuildBot macOS configs.
Pull Request: https://projects.blender.org/blender/blender/pulls/134639
Adds a C++ based FBX importer, using 3rd party ufbx library (design task:
#131304). The old Python based importer is still there; the new one is marked
as "(experimental)" in the menu item. Drag-and-drop uses the old Python
importer; the new one is only in the menu item.
The new importer is generally 2x-5x faster than the old one, and often uses
less memory too. There's potential to make it several times faster still.
- ASCII FBX files are supported now
- Binary FBX files older than 7.1 (SDK 2012) version are supported now
- Better handling of "geometric transform" (common in 3dsmax), manifesting
as wrong rotation for some objects when in a hierarchy (e.g. #131172)
- Some FBX files that the old importer was failing to read are supported now
(e.g. cases 47344, 134983)
- Materials import more shader parameters (IOR, diffuse roughness,
anisotropy, subsurface, transmission, coat, sheen, thin film) and shader
models (e.g. OpenPBR or glTF2 materials from 3dsmax imports much better)
- Importer now creates layered/slotted animation actions. Each "take" inside
FBX file creates one action, and animated object within it gets a slot.
- Materials that use the same texture several times no longer create
duplicate images; the same image is used
- Material diffuse color animations were imported, but they only animated
the viewport color. Now they also animate the nodetree base color too.
- "Ignore Leaf Bones" option no longer ignores leaf bones that are actually
skinned to some parts of the mesh.
- Previous importer was creating orphan invisible Camera data objects for
some files (mostly from MotionBuilder?), new one properly creates these
cameras.
Import settings that existed in Python importer, but are NOT DONE in the new
one (mostly because not sure if they are useful, and no one asked for them
from feedback yet):
- Manual Orientation & Forward/Up Axis: not sure if actually useful. FBX
file itself specifies the axes fairly clearly. USD/glTF/Alembic also do
not have settings to override them.
- Use Pre/Post Rotation (defaults on): feels like it should just always be
on. ufbx handles that internally.
- Apply Transform (defaults off, warning icon): not sure if needed at all.
- Decal Offset: Cycles specific. None of other importers have it.
- Automatic Bone Orientation (defaults off): feels like current behavior
(either on or off) often produces "nonsensical bones" where bone direction
does not go towards the children with either setting. There are discussions
within I/O and Animation modules about different ways of bone
visualizations and/or different bone length axes, that would solve this
in general.
- Force Connect Children (defaults off): not sure when that would be useful.
On several animated armatures I tried, it turns armature animation
into garbage.
- Primary/Secondary Bone Axis: again not sure when would be useful.
Importer UI screenshots, performance benchmark details and TODOs for later
work are in the PR.
Pull Request: https://projects.blender.org/blender/blender/pulls/132406
This was introduced during EEVEE-next developement
cycle to not make the buildbot fail because of EEVEE
render tests.
These have stabilized now and we can remove this option.
Pull Request: https://projects.blender.org/blender/blender/pulls/137545
This patch replaces our own FindTBB module with TBB's own TBBConfig
module. On the technical side of things, this harvests the TBB CMake
config modules, and switches our TBB CMake find_package calls from
Module mode to Config mode. Integration was done using OpenPGL as a
blueprint for parsing TBB target's property back into CMake variables
(TBB_LIBRARIES and TBB_INCLUDE_DIRS).
Pull Request: https://projects.blender.org/blender/blender/pulls/137063
Removes '/' prefix from gtest filter for parameterized tests. This prefix
was causing tests in fixtures like VKRenderGraphTestScheduler and
VKRenderGraphTestRender to be skipped in `make test`.
Only parameterized tests with an InstantiationName will have a '*/' in
their names as part of a prefix. But according to the spec, InstantiationName
is optional, and if not set it wont generate prefixes. In these cases, the
gtest_filter produced will not match the test names, and cause `make test`
to not run the tests.
Removing this '*/' prefix should still match any tests with a non-empty
InstantiationName.
Pull Request: https://projects.blender.org/blender/blender/pulls/136987
Majority of projects within generated VS IDE solution are put under
their respective source folders, except the test executables. Majority
of which are various libmv tests. Use the same cmake macro to arrange
them in folders as what is used for regular libraries.
Pull Request: https://projects.blender.org/blender/blender/pulls/137028
