This fixes most "One Definition Rule" violations inside blender proper
resulting from duplicate structures of the same name. The fixes were
made similar to that of !135491. See also #120444 for how this has come
up in the past.
These were found by using the following compile options:
-flto=4 -Werror=odr -Werror=lto-type-mismatch -Werror=strict-aliasing
Note: There are still various ODR issues remaining that require
more / different fixes than what was done here.
Pull Request: https://projects.blender.org/blender/blender/pulls/136371
The general idea is to keep the 'old', C-style MEM_callocN signature, and slowly
replace most of its usages with the new, C++-style type-safer template version.
* `MEM_cnew<T>` allocation version is renamed to `MEM_callocN<T>`.
* `MEM_cnew_array<T>` allocation version is renamed to `MEM_calloc_arrayN<T>`.
* `MEM_cnew<T>` duplicate version is renamed to `MEM_dupallocN<T>`.
Similar templates type-safe version of `MEM_mallocN` will be added soon
as well.
Following discussions in !134452.
NOTE: For now static type checking in `MEM_callocN` and related are slightly
different for Windows MSVC. This compiler seems to consider structs using the
`DNA_DEFINE_CXX_METHODS` macro as non-trivial (likely because their default
copy constructors are deleted). So using checks on trivially
constructible/destructible instead on this compiler/system.
Pull Request: https://projects.blender.org/blender/blender/pulls/134771
Followup to 48e26c3afe, and discussions in !134771 about keeping
'C-style' and 'C++ template type-safe style' implementations of our
guardedalloc separated. And it makes `MEM_freeN<T>` code simpler.
Also skip type-checking in `MEM_freeN<T>` only with MSVC, as clang-cl on
windows-arm64 does work fine with DNA structs using
`DNA_DEFINE_CXX_METHODS`.
Pull Request: https://projects.blender.org/blender/blender/pulls/134861
The main goal of these changes are to improve static (i.e. build-time)
checks on whether a given data can be allocated and freed with `malloc`
and `free` (C-style), or requires proper C++-style construction and
destruction (`new` and `delete`).
* Add new `MEM_malloc_arrayN_aligned` API.
* Make `MEM_freeN` a template function in C++, which does static assert on
type triviality.
* Add `MEM_SAFE_DELETE`, similar to `MEM_SAFE_FREE` but calling
`MEM_delete`.
The changes to `MEM_freeN` was painful and useful, as it allowed to fix a bunch
of invalid calls in existing codebase already.
It also highlighted a fair amount of places where it is called to free incomplete
type pointers, which is likely a sign of badly designed code (there should
rather be an API to destroy and free these data then, if the data type is not fully
publicly exposed). For now, these are 'worked around' by explicitly casting the
freed pointers to `void *` in these cases - which also makes them easy to search for.
Some of these will be addressed separately (see blender/blender!134765).
Finally, MSVC seems to consider structs defining new/delete operators (e.g. by
using the `MEM_CXX_CLASS_ALLOC_FUNCS` macro) as non-trivial. This does not
seem to follow the definition of type triviality, so for now static type checking in
`MEM_freeN` has been disabled for Windows. We'll likely have to do the same
with type-safe `MEM_[cm]allocN` API being worked on in blender/blender!134771
Based on ideas from Brecht in blender/blender!134452
Pull Request: https://projects.blender.org/blender/blender/pulls/134463
194e233d86 caused a discussion in the chat about the initialization
behavior of `MEM_new()`, and agreement was to not rely on
zero-initialization ever. Noted this in the API comment now.
Some people found the existing comment useful but it still left some
questions. Tried to clarify that now.
This is a crucial memory management function, it's important to have
behavior documented well, even if a full explanation is out-of-scope.
Also added another link in case people want to check more details.
Pull Request: https://projects.blender.org/blender/blender/pulls/134577
This implements the proposal from #124512. For that it contains the following
changes:
* Remove the global override of `new`/`delete` when `WITH_CXX_GUARDEDALLOC` was
enabled.
* Always use `MEM_CXX_CLASS_ALLOC_FUNCS` where it is currently used. This used
to be guarded by `WITH_CXX_GUARDEDALLOC` in some but not all cases. This means
that a few classes which didn't use our guarded allocator by default before,
are now using it.
Pull Request: https://projects.blender.org/blender/blender/pulls/130181
Add a new API to store data that is guaranteed to not be freed
before the memleak detector has run.
This will be used in next commit by the readfile code to improve
reporting on leaks from blendfile readingi process.
This is done by a two-layer approach:
A new templated `MEM_construct_leak_detection_data` allows to
create any type of data. Its ownership and lifetime are handled
internally, and guaranteed to not be destroyed before the memleak
detector has run.
Add a new template-based 'allocation string storage' system to
`intern/memutil`. This uses the new `Guardedalloc Persistent Storage`
system to store all 'complex' allocation messages, that cannot be
defined as literals.
Internally, the storage is done through an owning reference (a
`shared_ptr`) of the created data into a mutex-protected static
vector.
`MEM_init_memleak_detection` code ensures that this static storage
is created before the memleak detection data, so that it is destructed
after the memleak detector has ran.
The main container (`AllocStringStorageContainer`) is wrapping a
map of `{string -> AllocStringStorage<key_type, hash_type>}`.
The key is a storage identifier.
Each storage is also a map wrapped into a simple templated API
class (`AllocStringStorage`), where the values are the alloc strings,
and the keys type is defined by the user code.
Pull Request: https://projects.blender.org/blender/blender/pulls/125320
The main change from this commit is the usage of ASAN poisoning (if
available) to trigger an ASAN report on the erroring memory block.
The main benefit is the report of the allocation backtrace of that
faulty memory block.
Pull Request: https://projects.blender.org/blender/blender/pulls/124231
This commit will error (and abort if enabled) when trying to call
`MEM_freeN` (and related `MEM_dupallocN`, `MEM_reallocN` and
`MEM_recallocN` functions) with a pointer created the C++ way (i.e.
through `MEM_new`, or the guardedalloc-overloaded `new` operator).
To do so, it adds internal use only implementations for `malloc_alligned`
and `free`, which take an extra parameter indicating whether they are
dealing with data created/deleted the 'C++ way' (using `new`/`delete`
and similar).
The cpp-created data are flagged with the new
`MEMHEAD_FLAG_FROM_CPP_NEW`, either in the lower two-bytes len value for
lockfree allocator, or as a new flag member of the guarded allocator
header data.
The public `MEM_new`/`MEM_delete` template functions, and the
guardedalloc-overloaded versions of `new`/`delete` operators are updated
accordingly.
These changes have been successfully tested both with and without
`WITH_CXX_GUARDEDALLOC`.
NOTE: A lot of mismatches have already been fixed in `main` before merging
this change. There are likely some less easy to trigger ones still in our
codebase though.
Pull Request: https://projects.blender.org/blender/blender/pulls/123740
Almost certainly not an issue in current codebase (this 'copy' version
of `MEM_cnew` does not seem much used in the first place), but better be
consistent with the 'allocating' version.
Pull Request: https://projects.blender.org/blender/blender/pulls/123445
Sync a bit better the checks on the alignment value between
`MEM_lockfree_mallocN_aligned` and `MEM_guarded_mallocN_aligned`.
The only significant change, in `MEM_guarded_mallocN_aligned`, is the
usage of `ALIGNED_MALLOC_MINIMUM_ALIGNMENT` instead of 'magic value' `8`.
This should not have any effect on 64bits platforms, but on 32bits ones
the minimum alignment would be reduced from `8` to `4` now.
NOTE: we could also consider making these checks part of a utils
function, instead of duplicating them in the codebase.
Ensure that the MemHead and MemHeadAligned are such that memory
allocation followed with the head offset keeps the allocation
aligned to at least MEM_MIN_CPP_ALIGNMENT.
Pull Request: https://projects.blender.org/blender/blender/pulls/120582
This `operator new` added in ecc3e78d78
are only called if the alignment is greater than `__STDCPP_DEFAULT_NEW_ALIGNMENT__`.
This is generally 8 or 16 depending on the platform. `MEM_mallocN` does
guarantee 16 byte alignment currently (in fact it's usually not 16 byte aligned
because of `MemHead`). Now `MEM_mallocN_aligned` is used with the default
alignment, even if we don't know that the type does not require it.
An alternative would be to pass the alignment to `MEM_CXX_CLASS_ALLOC_FUNCS`,
but that would be more intrusive.
Pull Request: https://projects.blender.org/blender/blender/pulls/118568
Previously, the alignment of structs that use `MEM_CXX_CLASS_ALLOC_FUNCS`
were not taken into account when doing the allocation. This can cause some data
to be mis-aligned and leads to crashes when cpu instructions or code expect the
data to be aligned.
The fix is to provide an overload of `operator new` that accepts the alignment as parameter.
More info: https://en.cppreference.com/w/cpp/language/new (search for `align_val_t`).
Pull Request: https://projects.blender.org/blender/blender/pulls/118526
Enable huge pages for jemalloc. This requests the Linux kernel to use
huge (2 MB) pages for large allocations. This has the effect of speeding
up first accesses to those allocations, and possibly also speeds up future
accesses by reducing TLB faults.
By default, 4 KB pages are used unless the user enables huge pages through
a kernel parameter or an obscure sysfs setting.
For Cycles benchmarks, this gives about a 5% CPU rendering performance
improvement. It likely also improves performance in other areas of Blender.
Pull Request: https://projects.blender.org/blender/blender/pulls/116663
Bundling many tests in a single binary reduces build time and disk space
usage, but is less convenient for running individual tests command line
as filter flags need to be used.
This adds WITH_TESTS_SINGLE_BINARY to generate one executable file per
source file. Note that enabling this option requires a significant amount
of disk space.
Due to refactoring, the resulting ctest names are a bit different than
before. The number of tests is also a bit different depending if this
option is used, as one uses gtests discovery and the other is organized
purely by filename, which isn't always 1:1.
Co-authored-by: Sergey Sharybin <sergey@blender.org>
Pull Request: https://projects.blender.org/blender/blender/pulls/114604
Along with the 4.1 libraries upgrade, we are bumping the clang-format
version from 8-12 to 17. This affects quite a few files.
If not already the case, you may consider pointing your IDE to the
clang-format binary bundled with the Blender precompiled libraries.
Suppress false positive Valgrind warnings which flooded the output.
- BLI_mempool alloc/free & iteration.
- Set alignment padding bytes at the end of MEM_* allocations
as "defined" since this causes many false positive warnings
in blend file writing and MEMFILE comparisons.
- Set MEM_* allocations as undefined when `--debug-memory`
is passed in to account for debug initialization.
- Initialize pad bytes in TextLine allocations.
Recently [0] replaced back-traces from `execinfo.h` with ASAN's
`__asan_describe_address` since the linking options to hide symbols
cause the stack-traces only to include addresses (without functions).
Although using ASAN makes sense when enabled, in my tests the
stack-traces are sometimes empty. Using CMAKE_BUILD_TYPE=RelWithDebInfo
for e.g. wasn't showing a stack-trace for the leak fixed in [1].
A utility is now included to conveniently expand the addresses from
these stack traces (`tools/utils/addr2line_backtrace.py`).
Restore support for the execinfo stack-trace reporting, used when ASAN
is disabled.
[0]: 2e79ca3205
[1]: a9f0d19197
