This adds initial support for showing the difference between old and new coverage
data. The difference is only shown in index.html currently. This is mainly useful
when adding new tests right now, to see the impact.
To use this, rename the `coverage/analysis` folder in the build directory to
`reference`. It will automatically be used by the report script if that reference
folder exists.
This only works with GCC and has only been tested on Linux. The main goal is to
automatically generate the code coverage reports on the buildbot and to publish
them. With some luck, this motivates people to increase test coverage in their
respective areas. Nevertheless, it should be easy to generate the reports
locally too (at least on supported software stacks).
Usage:
1. Create a **debug** build using **GCC** with **WITH_COMPILER_CODE_COVERAGE**
enabled.
2. Run tests. This automatically generates `.gcda` files in the build directory.
3. Run `make/ninja coverage-report` in the build directory.
If everything is successful, this will open a browser with the final report
which is stored in `build-dir/coverage/report/`. For a bit more control one can
also run `coverage.py` script directly. This allows passing in the
`--no-browser` option which may be benefitial when running it on the buildbot.
Running `make/ninja coverage-reset` deletes all `.gcda` files which resets the
line execution counts.
The final report has a main entry point (`index.html`) and a separate `.html`
file for every source code file that coverage data was available for. This also
contains some code that is not in Blender's git repository. We could filter
those out, but it also seems interesting (to me anyway), so I just kept it in.
Doing the analysis and writing the report takes ~1 min. The slow part is running
all tests in a debug build which takes ~12 min for me. Since the coverage data
is fairly large and the report also includes the entire source code, file
compression is used in two places:
* The intermediate analysis results for each file are stored in compressed zip
files. This data is still independent from the report html and could be used
to build other tools on top of. I could imagine storing the analysis data for
each day for example to gather greater insights into how coverage changes over
time in different parts of the code.
* The analysis data and source code is compressed and base64 encoded embedded
into the `.html` files. This makes them much smaller than embedding the data
without compression (5-10x).
Pull Request: https://projects.blender.org/blender/blender/pulls/126181
