diff --git a/scripts/addons_core/bl_pkg/readme.rst b/scripts/addons_core/bl_pkg/readme.rst new file mode 100644 index 00000000000..8ff4bc2478d --- /dev/null +++ b/scripts/addons_core/bl_pkg/readme.rst @@ -0,0 +1,303 @@ + +########## +Extensions +########## + +Extensions Source Code Overview +=============================== + +Add-on: Blender Modules +----------------------- + +- ``__init__.py`` + Add-on containing, preferences, ``bpy.app.handlers`` that respond to adding/removing repositories. + +- ``bl_extension_ui.py`` + Defines the extensions UI, select between add-ons, themes, tag-filtering. + +- ``bl_extension_ops.py`` + Defines extension operators, this is the main entry point for extension logic (except notifications, see below). + + This module defines a mechanism for a modal operator to run commands + defined in ``cli/blender_ext.py`` as sub-processes, monitoring their progress + (via STDOUT, see :ref:`Inter process communication (IPC) `), + see the ``_ExtCmdMixIn`` class. + Actions such a as downloading, installing, updating are supported by calling into lower level functions, + ``cli/blender_ext.py`` does the actual work. + + There are also some operators for the UI (changing tags, allowing online access). + +- ``bl_extension_notify.py`` + + This module checks for updates and is intended to run in the background. + + Checking for updates uses ``bl_extension_utils.CommandBatch`` from a timer. + Checking for updates from a modal-operator is avoided since Blender may cancel + modal operators when loading a file for example. + + The status bar is refreshed if/when updates are found. + +- ``bl_extension_cli.py`` + + The command line interface to support: ``blender -c extension ...`` + + Some commands operate on Blender's preferences (for adding/removing repositories), + other commands such as building packages are forwarded to ``cli/blender_ext.py``. + +Add-on: Other Scripts +--------------------- + +- ``bl_extension_utils.py`` + + This module contains various utilities, + + Note that use of ``bpy`` is intentionally avoided here, + state from preferences or operators is passed in. + + - Generic shared utility functions. + + - Command line sub-process supervisor (``CommandBatch``) + used by Blender operators to call ``cli/blender_ext.py`` and its sub-commands (see doc-string for details). + + - A view on the repositories JSON/TOML data what abstracts the file IO (``RepoCacheStore``). + + - A locking context to prevent multiple Blender instances operating on the same repository at once. + +- ``cli/blender_ext.py`` + This command is responsible for operations on the repository, + primarily downloading & installing extensions. + + It also contains functions to build & validate packages & create a static repository. + + This script typically runs as an external process using the ``subprocess`` module + called from ``bl_extension_utils.py``. + In some cases ``bl_extension_cli.py`` forwards sub-commands directly to this script. + + :ref:`Inter process communication (IPC) ` via Python's ``subprocess`` module + which runs this script using Blender's bundled Python executable. + Signals are used to interrupt the process, pipes are used to read it's output. + + This is done so Blender's UI can show the status of each command. + +- ``extensions_map_from_legacy_addons`` + + This is more of a data file used so users with legacy add-ons can update them to extensions. + +Other Modules +------------- + +Some functionality that's relevant to the extensions system is implemented in other modules, +as it relates to how extensions are loaded by Blender. + +*Paths are relative to Blender's source tree.* + +``./scripts/modules/_bpy_internal/extensions/junction_module.py`` + This stand-alone module allows extensions to appear as if they are all loaded + from a single *package* independent of their file-system location. + + This is done so extensions don't pollute the module name-space + (avoiding naming collisions with packages downloaded from ``https://pypi.org``). + + So each extension's add-on ID follows this format: ``bl_ext.{repository_id}.{extension_id}``. + +``./scripts/modules/_bpy_internal/extensions/wheel_manager.py`` + Extensions may include Python modules as wheels, + these are extracted into an a site-packages directory that is specific to the extensions for this version of Blender. + ``~/.config/blender/X.X/extensions/.local/lib/python3.XX/site-packages/``. + + Once extensions have been installed the list of wheels from each extensions ``blender_manifest.toml`` + is combined and passed in to the main "apply_action" function which will install/uninstall wheels as needed. + + Unfortunately there is no special handling for version conflicts. + When different versions of the same wheel are found, the latest version is installed. + This may break any extensions depending on the old version of a wheel. + +``./scripts/modules/_bpy_internal/extensions/stale_file_manager.py`` + On MS-Windows it's common that files are locked and can't be deleted + (any DLL's loaded into memory), + although it can also happen if other processes are scanning the file system. + + In this case, the file is marked as stale and queued for removal when Blender next starts. + + Unfortunately **upgrading** extensions that use DLL's on MS-Windows isn't + reliable because it is necessary to remove then re-create the extension. + This is an area that could use further development it may be necessary to + support installing on restart. + +``./scripts/modules/_bpy_internal/extensions/{tags,permissions}.py`` + These are definition lists used when building packages, + they are ``https://extensions.blender.org`` specific. + + +C++ Sources +----------- + +``./source/blender/makesdna/DNA_userdef_types.h`` + The repository definition (``bUserExtensionRepo``). + +``./source/blender/editors/space_userpref/userpref_ops.cc`` + Operators for adding/removing repositories as well as dropping URL's to initiate installation. + +``./source/blender/python/intern/bpy_app_handlers.cc`` + Handlers for extensions ``bpy.app.handlers._extension_repos_*``, + *note the leading underscore as they are not part of the public API.* + + Unfortunately these handlers were needed as a way for Python to hook into lower level code paths, + so it's possible (for example) to refresh the extensions from an RNA update function + (``rna_userdef.cc`` and the operators in some cases). + +``wmWindowManager::extensions_updates`` & ``extensions_blocked`` + Status bar drawing uses these values set by ``bl_extension_notify.py``. + + +Functionality Described +======================= + +This section describes how functionality has been implemented. + + +Extension Pre-Flight Compatibility Check +---------------------------------------- + +Since extensions may be from a shared system directory or imported from an older installation +it's necessary to ensure the extension is compatible with Blender on startup. + +An extension may be incompatible for various reasons, + +- Unsupported Blender version. +- Unsupported platform. +- Binary incompatibility (from it's wheels). +- The extension may also have been block-listed. + +Since this runs on every startup, expensive checks are avoided if at all possible. + +In ``./scripts/modules/addon_utils.py`` the private function ``_initialize_extensions_compat_ensure_up_to_date`` +is responsible for ensuring extensions are compatible before loading. + +- On startup run: ``_initialize_extensions_repos_once`` which sets up repositories and handlers. +- If the "compatibility cache" doesn't exist it is created (each extension's TOML file is inspected for compatibility). + A dictionary of incompatible extensions is stored in the compatibility cache which is checked + whenever ``addon_utils.enable(..)`` is used to enable an extension. +- If the "compatibility cache" exists it is validated by each extensions TOML modification-time & size, + re-generating upon any changes. +- The compatibility data stores the reason the extension is disabled, + this is reported if the user attempts to enable it. + + +The details of the compatibility cache are documented in ``addon_utils``, +it's a simple format that stores the Blender version & a magic number that can be bumped at any time, +changes to these files cause the cache to be re-generated. + + +Dragging & Dropping a URL +------------------------- + +Extensions drag & drop is handled with Blender's drop-boxes. +This works in much the same way as dropping images in the 3D viewport or blend files. + +There are two drop-boxes used, one for file-paths another for URL's. +Both check the path contains a ``.zip`` extension, +where the URL logic needs to strips the query string and the fragment from the URL. + +The drop action runs the operator ``extensions.package_install`` (from ``bl_extension_ops.py``) +which checks if the ``url`` property has been set. +If so, the code-path for dropping a URL is activated. + +Once drop is activated: + +- A URL is scanned for blender version range & platform compatibility + to prevent downloading & attempting to install extensions which aren't compatible. + + Note that the query parameters in the URL are optional and serve as a convenient way to fail-early, + if they're not present the user may be prompted to add a new repository only to discover + later that extension they dropped isn't compatible with their Blender version. + + Both ``extensions.blender.org`` and static sites + generated by ``blender -c extension server-generate`` set these parameters. + +- A file-path is considered "local" so its manifest is inspected to check it's compatible. + +Other checks are performed to ensure the repository exists locally. +If the extension isn't found to be incompatible, the user may install it. + +Dropping a URL may prompt the user for actions that need to be done before the extensions may be installed. + +- Dropping a URL with "Online Access" disabled prompts the user to enable online-access. +- Dropping a URL from an unknown remote repository prompts the user to add the repository. +- Otherwise, dropping the URL of a compatible extension will prompt the user to install the extension. + +Unfortunately chaining popups together (setup wizard) or merging popups together is not well supported in Blender. +Causing some fairly bad worst-case scenarios when dropping a URL which isn't part of a known repository. + + +Implementation Details +====================== + +Extension Format +---------------- + +Extensions are intended to be created with the ``blender -c extension build`` +command which creates a ZIP file and performs some checks to catch errors early. + +The ZIP file must contain a ``blender_manifest.toml`` (which may be in a directory), +as well as files for a Python package for add-ons or an XML for themes. + + +Repositories +------------ + +Information about repositories is stored in user preferences. +The main values are a unique name, module path & optionally a remote URL. + +There are 2 kinds of repositories: + +- **Remote** which can be synchronized for updates. +- **Local** where the repository is a file-system location. + + Local repositories may define a source: + + - **User** the user may add/remove extensions to this location. + - **System** this treated as read-only and may be used when extensions + are shared on a network file-system for example. + +Synchronizing a **Remote** repository simply downloads the JSON listing from the remote URL. + +Once extensions have been installed their TOML files are compared with the repository to check for updates. + +.. _IPC: + +Inter Process Communication (IPC) +--------------------------------- + +- Commands that manipulate extensions (such as updating/installing/removing) + are performed by the stand-alone script ``cli/blender_ext.py``. +- Using IPC means these commands can run in the background without blocking Blender's GUI. +- The state of the extensions repository (repository location, blender-version, API tokens etc) + are passed in via command line arguments. +- This can be configured to only output JSON messages to the STDOUT which Blender parses and uses + to send feedback to the user. +- Progress (such as percentage of a file downloaded) is sent to the STDOUT + so the GUI and command-line interface can show progress. +- Input is limited to the request to cancel + (if the user cancels the operator or presses Control-C on the command line). +- Internally functions are responsible for checking if the user has requested to exit. + This is especially important before IO or anything that could cause the process to wait + so as to avoid "hanging" once the user has requested to exit. + +All IPC is handled by ``bl_extension_utils.CommandBatch`` which can run multiple commands, +a common case is running multiple updates at once. + +The caller can use methods on the CommandBatch to access the status and report any problems. + + +Tooling +======= + +The tests are not yet integrated into CTest +because some tests depend on the ``wheel`` module (not distributed with Blender's Python). + +Tests can be run via the ``Makefile`` using the local Python:: + + make -C scripts/addons_core/bl_pkg test + +Run the ``help`` target for a list of convenience targets to run checkers & tests.