test/release/release_notes/bug_fixes_per_major_release.py

#!/usr/bin/env python3
# SPDX-FileCopyrightText: 2025 Blender Authors
#
# SPDX-License-Identifier: GPL-2.0-or-later

r"""
### What this script does
This script looks at the fix commits between two versions of Blender
and tries to figure out if the commit fixed a issue that has been in
Blender for one or more major releases, or if the commit fixed a issue
introduced in this release.

This is done so we can create a "bug fixes" page in our release notes
listing commits that fix old issues.

The list generated by this script isn't 100% accurate (there will be
some missing commits), but it's significantly better than nothing.

---

### How to use the script
- Make sure the list `LIST_OF_OFFICIAL_BLENDER_VERSIONS` is up to date.
- Open a terminal in your Blender source code folder and make sure the
  branches you're interested in are up to date.
- Launch `bug_fixes_per_major_release.py` with relevant launch arguments. The required arguments are:
  - --current-version (-cv)
  - --previous-version (-pv)
  - --current-release-tag (-ct)
  - --previous-release-tag (-pt)
  - --backport-tasks (-bpt) (Optional, but recommended)
- Here is an example if you wish to collect the list for Blender 5.0 during
the Alpha stage of development.
  - `python bug_fixes_per_major_release.py -cv 5.0 -pv 4.5 -ct main -pt blender-v4.5-release -bpt 124452 135860 141871`
- Here is an example if you wish to collect the list for Blender 4.5 during
the Beta stage of development.
  - `python bug_fixes_per_major_release.py -cv 4.5 -pv 4.4 -ct blender-v4.5-release -pt blender-v4.4-release -bpt 109399 124452 135860`
- Wait for the script to finish (This can take upwards of 20 minutes).
- Follow the guide printed to terminal.

#### Additional usage
Sometimes commits can't be automatically sorted by this script.
Specifically the fixed issue listed in the commit message may be
incorrect.

In situations like this it can be easier to simply override the issue that
the commit claims to fix. This can be done by adding a entry to the overrides
issue: https://projects.blender.org/blender/blender/issues/137983

---

### How the script works
- First the script gathers all commits that contain `Fix #NUMBER` that
occurred between the two versions of Blender you're interested in.
  - This is done using:
  `git --no-pager log PREVIOUS_VERSION..CURRENT_VERSION --oneline -i -P --grep "Fix.*#+\d+"`
- The script then extracts all report numbers (`#NUMBER`)
  from the commit message.
- The script then iterates through those reports, checking the
  "Broken" and "Working" fields to try and figure out whether
  the commit fixed a issue that existed in earlier versions of Blender
  or not.
- Finally the list of commits and further instructions is printed to the
  terminal either for review or to be posted in the release notes.

---

### Limitations
- Revert commits are not handled automatically
  - Revert commits can either fix a bug, or revert a commit that fixes
  a bug. Distinguishing between the two can be difficult, so the script
  doesn't process these commits and instead leaves it to be manually
  sorted by the person creating the release notes.
- Fix commits that do not contain `Fix #NUMBER` in their commit message
are not included in the output of this script.
  - This is because the script figures out if a commit fixed a old
  issue or not based on the information in the bug report it fixed.
  If a bug report isn't listed in the commit message, then the script
  can't sort it and so it's ignored.
- If a user puts weird version numbers in their "Broken" or "Working"
fields on their bug report, then this script could accidentally sorted
the associated fix commit into the wrong category.
  - This is a consequence of the simple nature of the system the script
  uses to extract Blender versions from a bug report.
  - The triaging team will do their best to remove problematic
  version numbers from reports when they first encounter them.
- The script does not know about long term projects. So some fix
commits that obviously shouldn't go into the release notes can end up
in the output.
  - Take for example Grease Pencil v3 (GPv3). GPv3 was in development
  for multiple versions of Blender. As a result there are bug reports
  from 4.2 Alpha and 4.3 (the version it officially released in).
  If a fix commit in 4.3 claims to fix one of the reports made in 4.2,
  then that fix commit will show up in the list of "commits that fixed
  old issues" list. Which is incorrect as GPv3 was only released in 4.3.
  - These commits will need to be manually removed before posting the
  list to the release notes.

### Filling out bug report forms with relevant information
This script relies on bug reports having enough information in the
"Broken" and "Working" fields to be sorted. If a bug report does not
have enough information, it can not be sorted automatically and will
need to be manually sorted. The recommended process for manually
sorting these commits is to go to the relevant report that was fixed,
and update the "Broken" and "Working" fields with relevant information.

Figuring out how much information you need to put on a report for this
script can be confusing. So this section will try to clarify the
information required.

The script is looking for `Major.Minor` Blender versions on lines
that start with either "Broken" or "Working". So all the important
information should go on these lines. So if a report says:
"It worked in 4.3", then it needs to be rephrased to "Worked: 4.3"
for the script to find it.

The amount and type of information you put on each report will vary.
Here is a general outline of the order in which the script checks these
fields and this can help guide how much information needs to go on the
report.

From now on I will assume you are using this script to get the release
notes for Blender 4.4.

- The script looks at all "Broken" versions of Blender. If at least
one of them is below the current release, then the commit can be sorted.
  - E.g. `Broken: 4.4, 4.3` will allow the commit to be sorted as `4.3`
  is below the current release.
  - So if you know the bug is also present in a older version of
  Blender, then add that version to the list of broken versions.
- If the script can't sort based on the Broken field, it will try and
sort based on combined information from the Broken and Working fields.
  - E.g. `Broken: 4.4`. At this current point in time the script knows
  the issue is there in 4.4, but doesn't know if the issue was
  introduced in 4.4, or has been around for longer and only reported in
  4.4.
  - So the script then looks at the "Working" field. And if the working
  version is the current release, or the previous release, then we
  assume the issue was introduced in this release and can sort the
  report. For example `Worked: 4.3` or `Worked: 4.4` will allow the
  script to sort this report. But `Worked: 4.2` will not.
  - So if you test the previous version and find it works, then add
  it to the working field. Or if you bisect 4.4 to find the cause then
  add something like `Worked: Prior to 4.4 COMMIT_HASH`

Along with those guide lines above, it can be confusing about what to
do for bug reports for features introduced in the current release.
The broken version is 4.4, so the script can't sort based on that, but
there is no working version.

My recommendation is to include the current or previous version in the
working field with a comment explaining it. For example:
`Worked: Never, as this feature was introduced in 4.4`
Or
`Worked: 4.3 as it did not have this feature`
"""

__all__ = (
    "main",
)

import re
import sys
import json
import subprocess
import argparse
import urllib.error
import urllib.request
import urllib.robotparser

from time import time, sleep
from typing import Any
from pathlib import Path


# -----------------------------------------------------------------------------
# Constants used throughout the script

BLENDER_API_URL = "https://projects.blender.org/api/v1"

UNKNOWN = "UNKNOWN"

FIXED_NEW_ISSUE = "FIXED NEW"
NEEDS_MANUAL_SORTING = "MANUALLY SORT"
FIXED_OLD_ISSUE = "FIXED OLD"
FIXED_PR = "FIXED PR"
REVERT = "REVERT"
IGNORED = "IGNORED"

SORTED_CLASSIFICATIONS = [FIXED_NEW_ISSUE, FIXED_OLD_ISSUE, IGNORED]
VALID_CLASSIFICATIONS = [FIXED_NEW_ISSUE, NEEDS_MANUAL_SORTING, FIXED_OLD_ISSUE, FIXED_PR, REVERT, IGNORED]

OLDER_VERION = "OLDER"
NEWER_VERION = "NEWER"
SAME_VERION = "SAME"

# Add recent Blender versions to this list, including in-development versions.
# This list is used to identify if a version number found in a report is a valid version number.
# This is to help eliminate dates and other weird information people put
# in their reports in the format of a version number.

LIST_OF_OFFICIAL_BLENDER_VERSIONS = (
    # 1.x.
    '1.0', '1.60', '1.73', '1.80',
    # 2.0X.
    '2.04',
    # 2.2x.
    '2.26', '2.27', '2.28',
    # 2.3x.
    '2.30', '2.31', '2.32', '2.33', '2.34', '2.35', '2.36', '2.37', '2.39',
    # 2.4x.
    '2.40', '2.41', '2.42', '2.43', '2.44', '2.45', '2.46', '2.47', '2.48', '2.49',
    # 2.5x.
    '2.50', '2.53', '2.54', '2.55', '2.56', '2.57', '2.58', '2.59',
    # 2.6x.
    '2.60', '2.61', '2.62', '2.63', '2.64', '2.65', '2.66', '2.67', '2.68', '2.69',
    # 2.7x.
    '2.70', '2.71', '2.72', '2.73', '2.74', '2.75', '2.76', '2.77', '2.78', '2.79',
    # 2.8x.
    '2.80', '2.81', '2.82', '2.83',
    # 2.9x.
    '2.90', '2.91', '2.92', '2.93',
    # 3.x.
    '3.0', '3.1', '3.2', '3.3', '3.4', '3.5', '3.6',
    # 4.x.
    '4.0', '4.1', '4.2', '4.3', '4.4', '4.5',
    # 5.x.
    '5.0', '5.1',
)

# Catch duplicates
assert len(set(LIST_OF_OFFICIAL_BLENDER_VERSIONS)) == len(LIST_OF_OFFICIAL_BLENDER_VERSIONS)


# -----------------------------------------------------------------------------
# Private Utilities

CRAWL_DELAY = 2
last_checked_time = None


def set_crawl_delay() -> None:
    global CRAWL_DELAY
    # Conform to Blenders crawl delay request:
    # https://projects.blender.org/robots.txt
    try:
        projects = urllib.robotparser.RobotFileParser(url="https://projects.blender.org/robots.txt")
        projects.read()
        projects_crawl_delay = projects.crawl_delay("*")
        if projects_crawl_delay is not None:
            assert isinstance(projects_crawl_delay, int)
            CRAWL_DELAY = projects_crawl_delay
    except:
        pass


def url_json_get(url: str) -> Any:
    global last_checked_time

    if last_checked_time is not None:
        sleep(max(CRAWL_DELAY - (time() - last_checked_time), 0))
    last_checked_time = time()

    try:
        # Make the HTTP request and store the response in a 'response' object
        with urllib.request.urlopen(url) as response:
            result_bytes = response.read()
    except urllib.error.URLError as ex:
        print(url)
        print(f"Error making HTTP request: {ex}")
        return None

    # Convert the response content to a JSON object containing the user information.
    result = json.loads(result_bytes)
    assert result is None or isinstance(result, (dict, list))
    return result


# -----------------------------------------------------------------------------
# Commit Info Class

class CommitInfo:
    __slots__ = (
        "hash",
        "commit_title",
        "backport_list",
        "classification",
        "fixed_reports",
        "has_been_overwritten",
        "is_revert",
        "module",
        "needs_update",
        "report_title",
    )

    def __init__(self, commit_line: str) -> None:
        split_message = commit_line.split()

        # Commit line is in the format:
        # COMMIT_HASH Title of commit
        self.hash = split_message[0]
        self.commit_title = " ".join(split_message[1:])

        self.set_defaults()

    def set_defaults(self) -> None:
        self.is_revert = 'revert' in self.commit_title.lower()

        self.fixed_reports = self.check_full_commit_message_for_fixed_reports()

        # Setup some "useful" empty defaults.
        self.backport_list: list[str] = []
        self.module = UNKNOWN
        self.report_title = UNKNOWN
        self.classification = UNKNOWN

        # Variables below this point should not be saved to the cache.
        self.needs_update = True
        self.has_been_overwritten = False

    def check_full_commit_message_for_fixed_reports(self) -> list[str]:
        command = ['git', 'show', '-s', '--format=%B', self.hash]
        command_output = subprocess.run(command, capture_output=True).stdout.decode('utf-8')

        # Find every instance of `SPACE#NUMBER`. These are the report that the commit claims to fix.
        # We are looking for the `SPACE` part because otherwise commits that fix issues in other repositories,
        # E.g. Fix `blender/blender-manual#NUMBER`, will be picked out for processing.
        match = re.findall(r'\s#+(\d+)', command_output)
        if match:
            return match
        return []

    def get_backports(self, dict_of_backports: dict[str, list[str]]) -> None:
        # Figures out if the commit was back-ported, and to what version(s).
        if self.needs_update:
            for version_number in dict_of_backports:
                for backported_commit in dict_of_backports[version_number]:
                    if self.hash.startswith(backported_commit):
                        self.backport_list.append(version_number)
                        break

        if len(self.backport_list) > 0:
            # If the fix was back-ported to a old release, then it fixed a old issue.
            self.classification = FIXED_OLD_ISSUE

    def override_report_info(self, new_classification: str, new_title: str, new_module: str) -> bool:
        if new_classification in SORTED_CLASSIFICATIONS:
            # Clear classifications are more important then any other. So always override in this case.
            self.classification = new_classification
            self.report_title = new_title
            self.module = new_module
            return True

        if new_classification in (NEEDS_MANUAL_SORTING, FIXED_PR):
            if (self.classification == UNKNOWN) or ((new_classification ==
                                                     NEEDS_MANUAL_SORTING) and (self.classification == FIXED_PR)):
                # Only replace information if the previous classification was the default (UNKNOWN)
                # or the new classification is NEEDS_MANUAL_SORTING and the old one was
                # FIXED_PR (NEEDS_MANUAL_SORTING is more useful than FIXED_PR).
                self.classification = new_classification
                self.report_title = new_title
                self.module = new_module

        return False

    def get_module(self, labels: list[dict[Any, Any]]) -> str:
        # Figures out what module the report that was fixed belongs too.
        for label in labels:
            if "module" in label['name'].lower():
                # Module labels are typically in the format Module/NAME.
                return " ".join(label['name'].split("/")[1:])

        return UNKNOWN

    def classify(
            self,
            *,
            current_version: str,
            previous_version: str,
    ) -> None:
        if not self.needs_update:
            # The data was loaded from cache, no need to reprocess it.
            return

        if self.is_revert:
            self.classification = REVERT
            # Give reverts commits their commit title so when it is printed to terminal, it has a useful name.
            self.report_title = self.commit_title
            return

        for report_number in self.fixed_reports:
            report_information = url_json_get(f"{BLENDER_API_URL}/repos/blender/blender/issues/{report_number}")
            if report_information is None:
                print(f"ERROR: Could not gather information from report number: {report_number}\n")
                continue

            report_title = report_information['title']
            module = self.get_module(report_information['labels'])

            if "pull" in report_information['html_url']:
                # The fixed issue turns out to be a pull request.
                # This was probably a typo, but note it down away so we can check and fix it.
                self.override_report_info(FIXED_PR, self.commit_title, UNKNOWN)
            else:
                classification = classify_based_on_report(
                    report_information['body'],
                    current_version=current_version,
                    previous_version=previous_version,
                )
                if self.override_report_info(classification, report_title, module):
                    # The commit has been sorted. No need to process more reports.
                    break

    def generate_release_note_ready_string(self) -> str:
        def sort_version_numbers(input_version_num: str) -> str:
            # Pad to three digits to be future proof.
            pad = 3
            major, minor, patch = input_version_num.split(".")
            return major.zfill(pad) + minor.zfill(pad) + patch.zfill(pad)

        # Breakup report_title based on words, and remove `:` if it's at the end of the first word.
        # This is because the website the release notes are being posted to applies some undesirable
        # formatting to ` * Word:`.
        title = self.report_title
        split_title = title.split()
        split_title[0] = split_title[0].strip(":")

        # Capitalize the first letter of the issue title.
        split_title[0] = split_title[0][0].upper() + split_title[0][1:]

        title = " ".join(split_title)

        formatted_string = (
            f" * {title} [[{self.hash[:11]}](https://projects.blender.org/blender/blender/commit/{self.hash})]"
        )

        self.backport_list.sort(key=sort_version_numbers)

        if len(self.backport_list) > 0:
            formatted_string += f" - Backported to "
            if len(self.backport_list) > 2:
                # In case of three or more backports, create a list that looks like:
                # "Backported to 3.6, 4.2, and 4.3"
                formatted_string += f"{', '.join(self.backport_list[:-1])}, and {self.backport_list[-1]}"
            else:
                formatted_string += " and ".join(self.backport_list)
        formatted_string += "\n"

        return formatted_string

    def prepare_for_cache(self) -> tuple[str, dict[str, Any]]:
        return self.hash, {
            'is_revert': self.is_revert,
            'fixed_reports': self.fixed_reports,
            'backport_list': self.backport_list,
            'module': self.module,
            'report_title': self.report_title,
            'classification': self.classification,
        }

    def read_from_cache(self, cache_data: dict[str, Any]) -> None:
        self.is_revert = cache_data['is_revert']
        self.fixed_reports = cache_data['fixed_reports']
        self.backport_list = cache_data['backport_list']
        self.module = cache_data['module']
        self.report_title = cache_data['report_title']
        self.classification = cache_data['classification']

        self.needs_update = False

    def read_from_override(self, override_data: str) -> None:
        self.set_defaults()
        if "ignore" in override_data.lower():
            self.classification = IGNORED
            self.needs_update = False
        else:
            self.fixed_reports = [override_data]
            self.needs_update = True

        # Revert commits that have been overwritten should be processed like normal commits.
        self.is_revert = False
        self.has_been_overwritten = True

# ---


def setup_commit_info(commit: str) -> CommitInfo | None:
    commit_information = CommitInfo(commit)
    if len(commit_information.fixed_reports) > 0:
        return commit_information
    return None


def get_fix_commits(
        *,
        current_release_tag: str,
        previous_release_tag: str,
        single_thread: bool,
) -> list[CommitInfo]:
    # --no-pager means it prints everything all at once rather than providing a interactive scrollable page.
    # --no-abbrev-commit tells git to always show the full commit hash.
    # -i tells grep to ignore case when searching through commits.
    # -P tells grep to use a specific type of regular expression.

    # This searches for Fix{anything}{one_or_more #}{number}.
    # .* = {anything}
    # #+ = {one_or_more #}
    # \d+ = {number}
    # This captures the common `Fix #123`, but also the less common `Fixes #123`, `Fix for #123`, and `Fix ##123`.
    command = [
        'git',
        '--no-pager',
        'log',
        f'{previous_release_tag}..{current_release_tag}',
        '--oneline',
        '--no-abbrev-commit',
        '-i',
        '-P',
        '--grep',
        r'Fix.*#+\d+',
    ]

    git_log_command_output = subprocess.run(command, capture_output=True).stdout.decode('utf-8')
    git_log_output = git_log_command_output.splitlines()

    if single_thread:
        # Original non-multiprocessing method.
        intial_list_of_commits = []
        for commit in git_log_output:
            intial_list_of_commits.append(setup_commit_info(commit))
    else:
        # Although setup_commit_info is not compute intensive, it is time consuming due to hundreds of git log calls.
        # Multiprocessing can significantly reduce the time taken (E.g. 19s -> 4s on a 32 thread CPU).
        import multiprocessing
        with multiprocessing.Pool() as pool:
            intial_list_of_commits = pool.map(setup_commit_info, git_log_output)

    list_of_commits = [result for result in intial_list_of_commits if result]
    return list_of_commits


# -----------------------------------------------------------------------------
# Utility Functions for `classify_based_on_report()`

def get_version_numbers(broken_lines: str, working_lines: str) -> tuple[list[str], list[str]]:
    def extract_numbers(string: str) -> list[str]:
        return re.findall(r"(\d+\.\d+)", string)

    # Extracts all version numbers from the broken and working fields
    # (Sometimes including weird version numbers like dates).
    temp_broken_versions = extract_numbers(broken_lines)
    temp_working_versions = extract_numbers(working_lines)

    broken_versions = []
    working_versions = []

    for version_number in temp_broken_versions:
        # Filter out any numbers picked up in the previous step that aren't official Blender version numbers.
        if version_number in LIST_OF_OFFICIAL_BLENDER_VERSIONS:
            broken_versions.append(version_number)
    for version_number in temp_working_versions:
        # Filter out any numbers picked up in the previous step that aren't official Blender version numbers.
        if version_number in LIST_OF_OFFICIAL_BLENDER_VERSIONS:
            working_versions.append(version_number)

    return broken_versions, working_versions


def version_extraction(report_body: str) -> tuple[list[str], list[str]]:
    broken_lines = ''
    working_lines = ''
    for line in report_body.splitlines():
        lower_line = line.lower()
        example_in_line = 'example' in lower_line
        if lower_line.startswith('brok') and not example_in_line:
            # Use `brok` to be able to detect different variations of "broken".
            broken_lines += f'{line}\n'
        if lower_line.startswith('work'):
            # Use `work` to be able to detect both "worked" and "working".
            if (not example_in_line) and not ("brok" in lower_line):
                # Don't add the line to the working_lines if it contains the letters `brok`.
                # because it means the user probably wrote something like "Worked: It was also broken in X.X"
                # which lead to incorrect information.
                working_lines += f'{line}\n'

    return get_version_numbers(broken_lines, working_lines)


def compare_versions(comparing_version: str, reference_version: str) -> str:
    # Compare two versions of Blender and return how they compare relative to each other.
    comp_version = comparing_version.split(".")
    ref_version = reference_version.split(".")
    comparing_major = int(comp_version[0])
    comparing_minor = int(comp_version[1])
    reference_major = int(ref_version[0])
    reference_minor = int(ref_version[1])

    if comparing_major < reference_major:
        return OLDER_VERION

    if comparing_major == reference_major:
        # The major version matches, so we must compare based on the minor version number.
        if comparing_minor < reference_minor:
            return OLDER_VERION
        if comparing_minor == reference_minor:
            return SAME_VERION

    return NEWER_VERION


# ---

def classify_based_on_report(
        report_body: str,
        *,
        current_version: str,
        previous_version: str,
) -> str:
    if "skip_for_bug_fix_release_notes" in report_body.lower():
        return IGNORED
    # Get a list of broken and working versions of Blender according to the report that was fixed.
    broken_versions, working_versions = version_extraction(report_body)

    broken_is_current_or_newer = False

    for broken_version in broken_versions:
        relative_version = compare_versions(broken_version, current_version)
        if relative_version == OLDER_VERION:
            # Broken version is older than current release. So the issue is from a older version.
            return FIXED_OLD_ISSUE
        if relative_version in (SAME_VERION, NEWER_VERION):
            broken_is_current_or_newer = True

    for working_version in working_versions:
        relative_version = compare_versions(working_version, current_version)
        if relative_version in (SAME_VERION, NEWER_VERION):
            # Working version is current version or newer. So the issue was introduced in this version.
            return FIXED_NEW_ISSUE

    if broken_is_current_or_newer and (previous_version in working_versions):
        # Issue is in current release, but wasn't in previous release.
        # So it must of been introduced in the current release.
        return FIXED_NEW_ISSUE

    return NEEDS_MANUAL_SORTING


# -----------------------------------------------------------------------------
# Utility Functions for `classify_commits()`

def get_backported_commits(issue_number: str) -> dict[str, list[str]]:
    # Adapted from https://projects.blender.org/blender/blender/src/branch/main/release/lts/lts_issue.py

    issues_url = f"{BLENDER_API_URL}/repos/blender/blender/issues/"

    response = url_json_get(issues_url + issue_number)
    description = response["body"]

    lines = description.split("\n")
    current_version = None

    dict_of_backports: dict[str, list[str]] = {}

    blender_version_start = "## Blender "
    for line in lines:
        if line.startswith(blender_version_start):
            current_version = line.strip(blender_version_start)
        if current_version is None:
            # We haven't got a Blender version yet.
            continue
        if not line.strip():
            continue
        if "|" not in line:
            # Not part of the backports table.
            continue
        if line.startswith("| **Report**"):
            continue
        if line.find("| -- |") != -1:
            continue

        items = line.split("|")
        commit_string = items[2].strip()
        commit_string = commit_string.split(",")[0]
        commit_string = commit_string.split("]")[0]
        commit_string = commit_string.replace("[", "")

        pattern = r"blender/blender@([a-zA-Z0-9]+)"
        matches = re.findall(pattern, commit_string)
        if len(matches) > 0:
            try:
                dict_of_backports[current_version] += matches
            except KeyError:
                dict_of_backports[current_version] = matches

    return dict_of_backports


def get_backports(backport_tasks: list[str]) -> dict[str, list[str]]:
    dict_of_backports: dict[str, list[str]] = {}
    for task in backport_tasks:
        dict_of_backports.update(get_backported_commits(task))

    return dict_of_backports


# ---

def classify_commits(
        backport_tasks: list[str],
        list_of_commits: list[CommitInfo],
        *,
        current_version: str,
        previous_version: str,
) -> None:
    number_of_commits = len(list_of_commits)

    print("Identifying if fixes are for a bug introduced in this release, or if the bug was there in a previous release.")
    print("This requires querying information from Gitea, and can take a while.\n")

    dict_of_backports = get_backports(backport_tasks)

    i = 0
    start_time = time()
    for commit in list_of_commits:
        # Simple progress bar.
        i += 1
        print(
            f"{i}/{number_of_commits} - Estimated time remaining:",
            f"{(((time() - start_time) / i) * (number_of_commits - i)) / 60:.1f} minutes",
            end="\r",
            flush=True
        )

        commit.classify(
            current_version=current_version,
            previous_version=previous_version,
        )
        commit.get_backports(dict_of_backports)

    # Print so we're away from the progress bar.
    print("\n\n\n")


# ---

def organize_commits(list_of_commits: list[CommitInfo]) -> dict[str, dict[str, list[CommitInfo]]]:
    # This function takes in a list of commits, and sorts them based on their classification and module.

    dict_of_sorted_commits: dict[str, dict[str, list[CommitInfo]]] = {}
    for item in VALID_CLASSIFICATIONS:
        dict_of_sorted_commits[item] = {}

    for commit in list_of_commits:
        commit_classification = commit.classification
        if commit_classification in VALID_CLASSIFICATIONS:
            commit_module = commit.module
            try:
                # Try to append to a list. If it fails (The list doesn't exist), create the list.
                dict_of_sorted_commits[commit_classification][commit_module].append(commit)
            except KeyError:
                dict_of_sorted_commits[commit_classification][commit_module] = [commit]

    for item in VALID_CLASSIFICATIONS:
        # Sort modules alphabetically
        dict_of_sorted_commits[item] = dict(sorted(dict_of_sorted_commits[item].items()))

    return dict_of_sorted_commits


def print_list_of_commits(title: str, dict_of_commits: dict[str, list[CommitInfo]]) -> None:
    commits_message = ""
    number_of_commits = 0
    unknown_module_commit_message = ""
    for module in dict_of_commits:
        commits_in_this_module = len(dict_of_commits[module])
        number_of_commits += commits_in_this_module
        module_label = f"\n## {module}: {commits_in_this_module}\n"
        module_is_unknown = (module == UNKNOWN)
        if module_is_unknown:
            unknown_module_commit_message += module_label
        else:
            commits_message += module_label

        for commit in dict_of_commits[module]:
            printed_line = commit.generate_release_note_ready_string()

            if module_is_unknown:
                unknown_module_commit_message += printed_line
            else:
                commits_message += printed_line

    if number_of_commits != 0:
        print(f"{title} {number_of_commits}")
        print(commits_message)
        print(unknown_module_commit_message)
        print("\n\n\n")


# ---

def print_release_notes(list_of_commits: list[CommitInfo]) -> None:
    dict_of_sorted_commits = organize_commits(list_of_commits)

    print_list_of_commits("Commits that fixed old issues:", dict_of_sorted_commits[FIXED_OLD_ISSUE])

    print_list_of_commits(
        "Revert commits. Add overrides to https://projects.blender.org/blender/blender/issues/137983:",
        dict_of_sorted_commits[REVERT])

    print_list_of_commits("Commits that need manual sorting:", dict_of_sorted_commits[NEEDS_MANUAL_SORTING])

    print_list_of_commits(
        "Commits that need a override in https://projects.blender.org/blender/blender/issues/137983 as they claim to fix a PR:",
        dict_of_sorted_commits[FIXED_PR])

    print_list_of_commits("Ignored commits:", dict_of_sorted_commits[IGNORED])

    # Currently disabled as this information isn't particularly useful.
    # print_list_of_commits(dict_of_sorted_commits[FIXED_NEW_ISSUE])

    print(r"""What to do with this output:
    - Go through every commit in the "Commits that need manual sorting" section and:
      - Find the corresponding issue that was fixed (it will be in the commit message)
      - Update the "Broken" and/or "Working" fields of the report with relevant information so this script can sort it.
        - Add a module label if it's missing one.
      - Rerun this script.
    - Repeat the previous steps until there are no commits that need manual sorting.
    - If it is too difficult to track down the broken or working field for a report, then you can add
    `<!-- skip_for_bug_fix_release_notes -->` to the report body and the script will ignore it on subsequent runs.
    - This should be done by the triaging module through out the release cycle, so the list should be quite small.

    - Go through the "Revert commits" section and add entries to https://projects.blender.org/blender/blender/issues/137983
      for the reverted commits.
    - Double check if there are any obvious commits in the
      "Commits that fixed old issues" section that shouldn't be there and remove them
      (E.g. A fix for a feature that has been in development over a few releases,
      but was only enabled in this release).
    - Add the output of the "Commits that fixed old issues" section to the release notes:
      https://projects.blender.org/blender/blender-developer-docs/src/branch/main/docs/release_notes
      Here is the release notes for a previous release for reference:
      https://projects.blender.org/blender/blender-developer-docs/src/branch/main/docs/release_notes/4.3/bugfixes.md""")


# -----------------------------------------------------------------------------
# Caching Utilities

def cached_commits_load(list_of_commits: list[CommitInfo], path_to_cached_commits: Path) -> None:
    if path_to_cached_commits.exists():
        with open(str(path_to_cached_commits), 'r', encoding='utf-8') as file:
            cached_data = json.load(file)
        for commit in list_of_commits:
            if commit.hash in cached_data:
                commit.read_from_cache(cached_data[commit.hash])


def cached_commits_store(list_of_commits: list[CommitInfo], path_to_cached_commits: Path) -> None:
    # Cache information for commits that have been sorted.
    # Commits that still need sorting are not cached.
    # This is done so if a user is repeatably running this script so they can sort
    # the "needs sorting" section, they don't have to wait for information requests to GITEA
    # on commits that are already sorted (and they're not interested in).
    data_to_cache = {}
    for commit in list_of_commits:
        if (
            (commit.classification not in (NEEDS_MANUAL_SORTING, IGNORED)) and
            (commit.has_been_overwritten is False) and
            (commit.module != UNKNOWN)
        ):
            commit_hash, data = commit.prepare_for_cache()
            data_to_cache[commit_hash] = data

    with open(str(path_to_cached_commits), 'w', encoding='utf-8') as file:
        json.dump(data_to_cache, file, indent=4)


# -----------------------------------------------------------------------------
# Override Utilities


def overrides_read(silence: bool) -> dict[str, str]:
    override_data: dict[str, str] = {}
    override_report = url_json_get(f"{BLENDER_API_URL}/repos/blender/blender/issues/137983")
    description = override_report["body"].splitlines()

    for line in description:
        if "|" not in line:
            continue
        if line.startswith("| Commit"):
            continue
        if line.startswith("| -"):
            continue

        split_line = line.split("|")
        info: list[str] = []
        for entry in split_line:
            # Remove empty strings and strip "#" off the issue number
            entry = entry.strip().strip("#")
            if len(entry) != 0:
                info.append(entry)

        try:
            hash = info[0]
            fixed_issue = info[1]
            if len(hash) < 10:
                print("\n" * 3)
                print(f"ERROR: Hash is too short in this override data: {info}")
                if not silence:
                    input("Press enter to acknowledge: ")
                continue
            override_data[hash] = fixed_issue
        except IndexError:
            print("\n" * 3)
            print(f"INDEX ERROR: Failed to process overrides with this data: {info}")
            if not silence:
                input("Press enter to acknowledge: ")

    return override_data


def overrides_apply(list_of_commits: list[CommitInfo], silence: bool) -> None:
    override_data = overrides_read(silence)
    if len(override_data) == 0:
        return

    for commit_hash in override_data:
        for commit in list_of_commits:
            if commit.hash.startswith(commit_hash):
                commit.read_from_override(override_data[commit_hash])
                break


# -----------------------------------------------------------------------------
# Argument Parsing

def argparse_create() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description=__doc__,
        # Don't re-format multi-line text.
        formatter_class=argparse.RawTextHelpFormatter,
    )
    parser.add_argument(
        "-st",
        "--single-thread",
        action="store_true",
        help=(
            "Run one of the parts of this script in single threaded mode "
            "(Only really useful for debugging)."
        ),
    )
    parser.add_argument(
        "-c",
        "--cache",
        action="store_true",
        help=(
            "Use caching to speed up re-runs on this script "
            "(IMPORTANT: Leave caching off when collecting the final release notes)."
        ),
    )

    parser.add_argument(
        "-cv",
        "--current-version",
        help=(
            "The common major.minor name of the current version of Blender (E.g. 4.2, 4.3, 4.4)."
        ),
    )
    parser.add_argument(
        "-pv",
        "--previous-version",
        help=(
            "The common major.minor name of the previous version of Blender (E.g. 4.2, 4.3, 4.4)."
        ),
    )

    parser.add_argument(
        "-ct",
        "--current-release-tag",
        help=(
            "The tag for the current release of Blender. "
            "These can be tags (like `v4.3.0`), commit hashes, or branches."
        ),
    )
    parser.add_argument(
        "-pt",
        "--previous-release-tag",
        help=(
            "The tag for the previous release of Blender. "
            "These can be tags (like `v4.3.0`), commit hashes, or branches."
        ))

    parser.add_argument(
        "-bpt",
        "--backport-tasks",
        nargs='+',
        help=(
            "A list of backport tasks. "
            "Backport tasks can be found on the Blender milestones page: "
            "https://projects.blender.org/blender/blender/milestones"
        ),
        default=[],
    )

    parser.add_argument(
        "-s",
        "--silence",
        action="store_true",
        help="Silence some warnings.",
    )

    return parser


def validate_arguments(args: argparse.Namespace) -> bool:
    def print_error(variable_name: str, argument_1: str, argument_2: str) -> None:
        print(f"ERROR: {variable_name} (defined with '{argument_1}' or '{argument_2}') is not defined.")
        print("This script can not proceed without this variable defined.\n")

    should_quit = False
    if args.cache and not args.silence:
        print("WARNING: You are using a cache, this may lead to outdated information on some commits.")
        print("Do not use the cache to generate the final release notes.\n")
    if args.current_version is None:
        print_error("Current version", "-cv", "--current-version")
        should_quit = True
    if args.previous_version is None:
        print_error("Previous version", "-pv", "--previous-version")
        should_quit = True
    if args.current_release_tag is None:
        print_error("Current Release Tag", "-ct", "--current-release-tag")
        should_quit = True
    if args.previous_release_tag is None:
        print_error("Previous Release Tag", "-pt", "--previous-release-tag")
        should_quit = True
    if len(args.backport_tasks) == 0:
        print("WARNING: (Optional) -bpt/--backport-tasks is not defined.")
        if not (args.silence or should_quit):
            yes_no = input("Do you want to proceed without it? (y/n)")
            if yes_no.lower() == "n":
                should_quit = True

    return not should_quit


# -----------------------------------------------------------------------------
# Main Function

def gather_and_sort_commits(
        current_release_tag: str,
        current_version: str,
        previous_release_tag: str,
        previous_version: str,
        backport_tasks: list[str],
        cache: bool = False,
        silence: bool = False,
        single_thread: bool = False,
) -> list[CommitInfo]:
    set_crawl_delay()

    dir_of_sciprt = Path(__file__).parent.resolve()
    # Replace "/" with "-" to avoid issues with directories in case someone
    # uses "remote/branch" as their current or previous tag.
    path_to_cached_commits = dir_of_sciprt.joinpath(
        f'cached_commits_{previous_release_tag.replace("/", "-")}..{current_release_tag.replace("/", "-")}.json')

    list_of_commits = get_fix_commits(
        current_release_tag=current_release_tag,
        previous_release_tag=previous_release_tag,
        single_thread=single_thread,
    )

    if cache:
        cached_commits_load(list_of_commits, path_to_cached_commits)

    overrides_apply(list_of_commits, silence)

    classify_commits(
        backport_tasks,
        list_of_commits,
        current_version=current_version,
        previous_version=previous_version,
    )

    if cache:
        cached_commits_store(list_of_commits, path_to_cached_commits)

    return list_of_commits


def main() -> int:
    args = argparse_create().parse_args()

    if not validate_arguments(args):
        return 0

    list_of_commits = gather_and_sort_commits(
        args.current_release_tag,
        args.current_version,
        args.previous_release_tag,
        args.previous_version,
        args.backport_tasks,
        args.cache,
        args.silence,
        args.single_thread,
    )

    print_release_notes(list_of_commits)
    return 0


if __name__ == "__main__":
    sys.exit(main())