Re-design of submodules used in blender.git

This commit implements described in the #104573.

The goal is to fix the confusion of the submodule hashes change, which are not
ideal for any of the supported git-module configuration (they are either always
visible causing confusion, or silently staged and committed, also causing
confusion).

This commit replaces submodules with a checkout of addons and addons_contrib,
covered by the .gitignore, and locale and developer tools are moved to the
main repository.

This also changes the paths:
- /release/scripts are moved to the /scripts
- /source/tools are moved to the /tools
- /release/datafiles/locale is moved to /locale

This is done to avoid conflicts when using bisect, and also allow buildbot to
automatically "recover" wgen building older or newer branches/patches.

Running `make update` will initialize the local checkout to the changed
repository configuration.

Another aspect of the change is that the make update will support Github style
of remote organization (origin remote pointing to thy fork, upstream remote
pointing to the upstream blender/blender.git).

Pull Request #104755
This commit is contained in:
Sergey Sharybin
2023-02-21 16:39:58 +01:00
committed by Sergey Sharybin
parent 3e721195b0
commit 03806d0b67
570 changed files with 2039093 additions and 211 deletions

View File

@@ -0,0 +1,192 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
"""
Takes 2 args
qtc_assembler_preview.py <build_dir> <file.c/c++>
Currently GCC is assumed
"""
import sys
import os
import shlex
import subprocess
VERBOSE = os.environ.get("VERBOSE", False)
BUILD_DIR = sys.argv[-2]
SOURCE_FILE = sys.argv[-1]
# TODO, support other compilers
COMPILER_ID = 'GCC'
def find_arg(source, data):
source_base = os.path.basename(source)
for l in data:
# chances are high that we found the file
if source_base in l:
# check if this file is in the line
l_split = shlex.split(l)
for w in l_split:
if w.endswith(source_base):
if os.path.isabs(w):
if os.path.samefile(w, source):
# print(l)
return l
else:
# check trailing path (a/b/c/d/e.c == d/e.c)
w_sep = os.path.normpath(w).split(os.sep)
s_sep = os.path.normpath(source).split(os.sep)
m = min(len(w_sep), len(s_sep))
if w_sep[-m:] == s_sep[-m:]:
# print(l)
return l
def find_build_args_ninja(source):
make_exe = "ninja"
process = subprocess.Popen(
[make_exe, "-t", "commands"],
stdout=subprocess.PIPE,
cwd=BUILD_DIR,
)
while process.poll():
time.sleep(1)
out = process.stdout.read()
process.stdout.close()
# print("done!", len(out), "bytes")
data = out.decode("utf-8", errors="ignore").split("\n")
return find_arg(source, data)
def find_build_args_make(source):
make_exe = "make"
process = subprocess.Popen(
[make_exe, "--always-make", "--dry-run", "--keep-going", "VERBOSE=1"],
stdout=subprocess.PIPE,
cwd=BUILD_DIR,
)
while process.poll():
time.sleep(1)
out = process.stdout.read()
process.stdout.close()
# print("done!", len(out), "bytes")
data = out.decode("utf-8", errors="ignore").split("\n")
return find_arg(source, data)
def main():
import re
# currently only supports ninja or makefiles
build_file_ninja = os.path.join(BUILD_DIR, "build.ninja")
build_file_make = os.path.join(BUILD_DIR, "Makefile")
if os.path.exists(build_file_ninja):
if VERBOSE:
print("Using Ninja")
arg = find_build_args_ninja(SOURCE_FILE)
elif os.path.exists(build_file_make):
if VERBOSE:
print("Using Make")
arg = find_build_args_make(SOURCE_FILE)
else:
sys.stderr.write(f"Can't find Ninja or Makefile ({build_file_ninja!r} or {build_file_make!r}), aborting")
return
if arg is None:
sys.stderr.write(f"Can't find file {SOURCE_FILE!r} in build command output of {BUILD_DIR!r}, aborting")
return
# now we need to get arg and modify it to produce assembler
arg_split = shlex.split(arg)
# get rid of: 'cd /a/b/c && ' prefix used by make (ninja doesn't need)
try:
i = arg_split.index("&&")
except ValueError:
i = -1
if i != -1:
del arg_split[:i + 1]
if COMPILER_ID == 'GCC':
# --- Switch debug for optimized ---
for arg, n in (
# regular flags which prevent asm output
("-o", 2),
("-MF", 2),
("-MT", 2),
("-MMD", 1),
# debug flags
("-O0", 1),
(re.compile(r"\-g\d*"), 1),
(re.compile(r"\-ggdb\d*"), 1),
("-fno-inline", 1),
("-fno-builtin", 1),
("-fno-nonansi-builtins", 1),
("-fno-common", 1),
("-DDEBUG", 1), ("-D_DEBUG", 1),
# ASAN flags.
(re.compile(r"\-fsanitize=.*"), 1),
):
if isinstance(arg, str):
# exact string compare
while arg in arg_split:
i = arg_split.index(arg)
del arg_split[i: i + n]
else:
# regex match
for i in reversed(range(len(arg_split))):
if arg.match(arg_split[i]):
del arg_split[i: i + n]
# add optimized args
arg_split += ["-O3", "-fomit-frame-pointer", "-DNDEBUG", "-Wno-error"]
# not essential but interesting to know
arg_split += ["-ftree-vectorizer-verbose=1"]
arg_split += ["-S"]
# arg_split += ["-masm=intel"] # optional
# arg_split += ["-fverbose-asm"] # optional but handy
else:
sys.stderr.write(f"Compiler {COMPILER_ID!r} not supported")
return
source_asm = f"{SOURCE_FILE}.asm"
# Never overwrite existing files
i = 1
while os.path.exists(source_asm):
source_asm = f"{SOURCE_FILE}.asm.{i:d}"
i += 1
arg_split += ["-o", source_asm]
# print("Executing:", arg_split)
kwargs = {}
if not VERBOSE:
kwargs["stdout"] = subprocess.DEVNULL
os.chdir(BUILD_DIR)
subprocess.call(arg_split, **kwargs)
del kwargs
if not os.path.exists(source_asm):
sys.stderr.write(f"Did not create {source_asm!r} from calling {arg_split!r}")
return
if VERBOSE:
print(f"Running: {arg_split}")
print(f"Created: {source_asm!r}")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<externaltool id="qtc_assembler_preview">
<description>Create an assembler file from source (C/C++)</description>
<displayname>Assembler Preview</displayname>
<category>Compiler</category>
<executable output="showinpane" error="showinpane" modifiesdocument="no">
<path>qtc_assembler_preview.py</path>
<arguments>%{CurrentProject:BuildPath} %{CurrentDocument:FilePath}</arguments>
<workingdirectory>%{CurrentProject:BuildPath}</workingdirectory>
</executable>
</externaltool>

View File

@@ -0,0 +1,49 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
"""
Takes 1 arg
qtc_blender_diffusion.py <file> <row>
Currently GCC is assumed
"""
import sys
import os
import subprocess
SOURCE_FILE = sys.argv[-2]
SOURCE_ROW = sys.argv[-1]
BASE_URL = "https://developer.blender.org/diffusion/B/browse"
def main():
dirname, _filename = os.path.split(SOURCE_FILE)
process = subprocess.Popen(
["git", "rev-parse", "--symbolic-full-name", "--abbrev-ref",
"@{u}"], stdout=subprocess.PIPE, cwd=dirname, universal_newlines=True)
output = process.communicate()[0]
branchname = output.rstrip().rsplit('/', 1)[-1]
process = subprocess.Popen(
["git", "rev-parse", "--show-toplevel"],
stdout=subprocess.PIPE, cwd=dirname, universal_newlines=True)
output = process.communicate()[0]
toplevel = output.rstrip()
filepath = os.path.relpath(SOURCE_FILE, toplevel)
url = '/'.join([BASE_URL, branchname, filepath]) + "$" + SOURCE_ROW
print(url)
# Maybe handy, but also annoying?
if "--browse" in sys.argv:
import webbrowser
webbrowser.open(url)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<externaltool id="qtc_blender_diffusion">
<description>Print a URL to Diffusion on developer.blender.org for online reference</description>
<displayname>Blender Diffusion</displayname>
<category>Documentation</category>
<executable output="showinpane" error="showinpane" modifiesdocument="no">
<path>qtc_blender_diffusion.py</path>
<arguments>%{CurrentDocument:FilePath} %{CurrentDocument:Row}</arguments>
<workingdirectory>%{CurrentDocument:Path}</workingdirectory>
</executable>
</externaltool>

View File

@@ -0,0 +1,75 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
"""
Convert C++ Style Comments:
// hello
// world
To This:
/* hello
* world
*/
"""
import sys
txt = sys.stdin.read()
data = txt.split("\n")
# TODO. block comments
# first detect blocks
def block_data(data, i_start):
i_begin = -1
i_index = -1
i_end = -1
i = i_start
while i < len(data):
l = data[i]
if "//" in l:
i_begin = i
i_index = l.index("//")
break
i += 1
if i_begin != -1:
i_end = i_begin
for i in range(i_begin + 1, len(data)):
l = data[i]
if "//" in l and l.lstrip().startswith("//") and l.index("//") == i_index:
i_end = i
else:
break
if i_begin != i_end:
# do a block comment replacement
data[i_begin] = data[i_begin].replace("//", "/*", 1)
for i in range(i_begin + 1, i_end + 1):
data[i] = data[i].replace("//", " *", 1)
data[i_end] = "%s */" % data[i_end].rstrip()
# done with block comment, still go onto do regular replace
return max(i_end, i_start + 1)
i = 0
while i < len(data):
i = block_data(data, i)
i = 0
while "//" not in data[i] and i > len(data):
i += 1
for i, l in enumerate(data):
if "//" in l: # should check if it's in a string.
text, comment = l.split("//", 1)
l = "%s/* %s */" % (text, comment.strip())
data[i] = l
print("\n".join(data), end="")

View File

@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<externaltool id="qtc_cpp_to_c_comments">
<description>Convert blocks of C++ comments into C style comments.</description>
<displayname>C++ to C (Comments)</displayname>
<category>Formatting</category>
<executable output="replaceselection" error="showinpane" modifiesdocument="no">
<path>qtc_cpp_to_c_comments.py</path>
<input>%{CurrentDocument:Selection}</input>
<workingdirectory>%{CurrentDocument:Path}</workingdirectory>
</executable>
</externaltool>

View File

@@ -0,0 +1,45 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
"""
This script takes 2-3 args: [--browse] <Doxyfile> <sourcefile>
Where Doxyfile is a path relative to source root,
and the sourcefile as an absolute path.
--browse will open the resulting docs in a web browser.
"""
import sys
import os
import subprocess
import tempfile
def find_gitroot(filepath_reference):
path = filepath_reference
path_prev = ""
while not os.path.exists(os.path.join(path, ".git")) and path != path_prev:
path_prev = path
path = os.path.dirname(path)
return path
doxyfile, sourcefile = sys.argv[-2:]
doxyfile = os.path.join(find_gitroot(sourcefile), doxyfile)
os.chdir(os.path.dirname(doxyfile))
tempfile = tempfile.NamedTemporaryFile(mode='w+b')
doxyfile_tmp = tempfile.name
tempfile.write(open(doxyfile, "r+b").read())
tempfile.write(b'\n\n')
tempfile.write(b'INPUT=' + os.fsencode(sourcefile) + b'\n')
tempfile.flush()
subprocess.call(("doxygen", doxyfile_tmp))
del tempfile
# Maybe handy, but also annoying?
if "--browse" in sys.argv:
import webbrowser
webbrowser.open("html/files.html")

View File

@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<externaltool id="qtc_doxygen_file">
<description>Doxygen a single file</description>
<displayname>Doxygen File</displayname>
<category>Documentation</category>
<executable output="showinpane" error="showinpane" modifiesdocument="no">
<path>qtc_doxy_file.py</path>
<arguments>--browse doc/doxygen/Doxyfile %{CurrentDocument:FilePath}</arguments>
<workingdirectory>%{CurrentProject:BuildPath}</workingdirectory>
</executable>
</externaltool>

View File

@@ -0,0 +1,22 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
import sys
# TODO, get from QtCreator
TABSIZE = 4
txt = sys.stdin.read()
data = txt.split("\n")
for i, l in enumerate(data):
l_lstrip = l.lstrip("\t")
l_lstrip_tot = (len(l) - len(l_lstrip))
if l_lstrip_tot:
l_pre_ws, l_post_ws = l[:l_lstrip_tot], l[l_lstrip_tot:]
else:
l_pre_ws, l_post_ws = "", l
# expand tabs and remove trailing space
data[i] = l_pre_ws + l_post_ws.expandtabs(TABSIZE).rstrip(" \t")
print("\n".join(data), end="")

View File

@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<externaltool id="qtc_expand_tabmix">
<description>Expand non-leading tabs into spaces.</description>
<displayname>Expand Tab Mix</displayname>
<category>Formatting</category>
<executable output="replaceselection" error="showinpane" modifiesdocument="no">
<path>qtc_expand_tabmix.py</path>
<input>%{CurrentDocument:Selection}</input>
<workingdirectory>%{CurrentDocument:Path}</workingdirectory>
</executable>
</externaltool>

View File

@@ -0,0 +1,38 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
"""
This is just a wrapper to run Blender's QtCreator project file generator,
knowing only the CMake build path.
qtc_project_update.py <project_path>
"""
import sys
import os
PROJECT_DIR = sys.argv[-1]
def cmake_find_source(path):
import re
match = re.compile(r"^CMAKE_HOME_DIRECTORY\b")
cache = os.path.join(path, "CMakeCache.txt")
with open(cache, 'r', encoding='utf-8') as f:
for l in f:
if re.match(match, l):
return l[l.index("=") + 1:].strip()
return ""
SOURCE_DIR = cmake_find_source(PROJECT_DIR)
cmd = (
"python",
os.path.join(SOURCE_DIR, "build_files/cmake/cmake_qtcreator_project.py"),
"--build-dir",
PROJECT_DIR,
)
print(cmd)
os.system(" ".join(cmd))

View File

@@ -0,0 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<externaltool id="qtc_project_update">
<description>Regenerate the project file</description>
<displayname>Project File Regenerate</displayname>
<category>Project</category>
<executable output="showinpane" error="showinpane" modifiesdocument="no">
<path>qtc_project_update.py</path>
<arguments>%{CurrentProject:BuildPath}</arguments>
</executable>
</externaltool>

View File

@@ -0,0 +1,44 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
import sys
# TODO, get from QtCreator
TABSIZE = 4
txt = sys.stdin.read()
data = txt.split("\n")
maxlen = 0
# tabs -> spaces
for i, l in enumerate(data):
l = l.replace("\t", " " * TABSIZE)
l = l.rstrip()
maxlen = max(maxlen, len(l))
data[i] = l
for i, l in enumerate(data):
ws = l.rsplit(" ", 1)
if len(l.strip().split()) == 1 or len(ws) == 1:
pass
else:
j = 1
while len(l) < maxlen:
l = (" " * j).join(ws)
j += 1
data[i] = l
# add tabs back in
for i, l in enumerate(data):
ls = l.lstrip()
d = len(l) - len(ls)
indent = ""
while d >= TABSIZE:
d -= TABSIZE
indent += "\t"
if d:
indent += (" " * d)
data[i] = indent + ls
print("\n".join(data), end="")

View File

@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<externaltool id="qtc_right_align_trailing_char">
<description>Right align the last character of each line to the existing furthermost character (useful for multi-line macros).</description>
<displayname>Right Align Trailing Char</displayname>
<category>Formatting</category>
<executable output="replaceselection" error="showinpane" modifiesdocument="no">
<path>qtc_right_align_trailing_char.py</path>
<input>%{CurrentDocument:Selection}</input>
<workingdirectory>%{CurrentDocument:Path}</workingdirectory>
</executable>
</externaltool>

View File

@@ -0,0 +1,11 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
import sys
# TODO, accept other characters as args
txt = sys.stdin.read()
print("(", end="")
print(txt, end="")
print(")", end="")

View File

@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<externaltool id="qtc_select_surround">
<description>Surround selection with parentheses or other optionally other characters.</description>
<displayname>Surround selection with parentheses</displayname>
<category>Formatting</category>
<executable output="replaceselection" error="showinpane" modifiesdocument="no">
<path>qtc_select_surround.py</path>
<input>%{CurrentDocument:Selection}</input>
<workingdirectory>%{CurrentDocument:Path}</workingdirectory>
</executable>
</externaltool>

View File

@@ -0,0 +1,39 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
import sys
txt = sys.stdin.read()
data = txt.split("\n")
class PathCMP:
def __init__(self, path):
path = path.strip()
self.path = path
if path.startswith("."):
path = path[1:]
if path.startswith("/"):
path = path[1:]
if path.endswith("/"):
path = path[:-1]
self.level = self.path.count("..")
if self.level == 0:
self.level = (self.path.count("/") - 10000)
def __eq__(self, other):
return self.path == other.path
def __lt__(self, other):
return self.path < other.path if self.level == other.level else self.level < other.level
def __gt__(self, other):
return self.path > other.path if self.level == other.level else self.level > other.level
data.sort(key=lambda a: PathCMP(a))
print("\n".join(data), end="")

View File

@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<externaltool id="qtc_sort_paths">
<description>Path sort selection, taking into account path depth.</description>
<displayname>Sort (Path Depths)</displayname>
<category>Formatting</category>
<executable output="replaceselection" error="showinpane" modifiesdocument="no">
<path>qtc_sort_paths.py</path>
<input>%{CurrentDocument:Selection}</input>
<workingdirectory>%{CurrentDocument:Path}</workingdirectory>
</executable>
</externaltool>

View File

@@ -0,0 +1,41 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0-or-later
import sys
txt = sys.stdin.read()
data = txt.split("\n")
# Check if we're if0
is_comment = False
for l in data:
l_strip = l.strip()
if l_strip:
if l_strip.startswith("#if 0"):
is_comment = True
else:
is_comment = False
break
if is_comment:
pop_a = None
pop_b = None
for i, l in enumerate(data):
l_strip = l.strip()
if pop_a is None:
if l_strip.startswith("#if 0"):
pop_a = i
if l_strip.startswith("#endif"):
pop_b = i
if pop_a is not None and pop_b is not None:
del data[pop_b]
del data[pop_a]
else:
while data and not data[-1].strip():
data.pop()
data = ["#if 0"] + data + ["#endif\n"]
print("\n".join(data), end="")

View File

@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<externaltool id="qtc_toggle_if0">
<description>Toggle if 0 preprocessor block.</description>
<displayname>Toggle #if 0</displayname>
<category>Formatting</category>
<executable output="replaceselection" error="showinpane" modifiesdocument="no">
<path>qtc_toggle_if0.py</path>
<input>%{CurrentDocument:Selection}</input>
<workingdirectory>%{CurrentDocument:Path}</workingdirectory>
</executable>
</externaltool>

View File

@@ -0,0 +1,44 @@
This repository contains utilities to perform various editing operations as well as some utilities to integrate
Uncrustify and Meld.
This is for my own personal use, but I have tried to make the tools generic (where possible) and useful to others.
Installing
==========
All the scripts install to QtCreators ``externaltools`` path:
eg:
``~/.config/QtProject/qtcreator/externaltools/``
Currently QtCreator has no way to reference commands relative to this directory so the ``externaltools`` dir **must**
be added to the systems ``PATH``.
Tools
=====
Here are a list of the tools with some details on how they work.
Assembler Preview
-----------------
``External Tools -> Compiler -> Assembler Preview``
This tool generates the assembly for the current open document,
saving it to a file in the same path with an ".asm" extension.
This can be handy for checking if the compiler is really optimizing out code as expected.
Or if some change really doesn't change any functionality.
The way it works is to get a list of the build commands that would run, and get those commands for the current file.
Then this command runs, swapping out object creation args for arguments that create the assembly.
.. note:: It would be nice to open this file, but currently this isn't supported. It's just created along side the source.
.. note:: Currently only GCC is supported.