ImportEngine API

The proposed extension consists of the following objects:

importlib.engine.ImportEngine

from_engine(self, other)

Create a new import object from another ImportEngine instance. The new object is initialised with a copy of the state in other. When called on importlib engine.sysengine, from_engine() can be used to create an ImportEngine object with a copy of the global import state.

__import__(self, name, globals={}, locals={}, fromlist=[], level=0)

Reimplementation of the builtin __import__() function. The import of a module will proceed using the state stored in the ImportEngine instance rather than the global import state. For full documentation of __import__ functionality, see [2] . __import__() from ImportEngine and its subclasses can be used to customise the behaviour of the import statement by replacing __builtin__.__import__ with ImportEngine().__import__.

import_module(name, package=None)

A reimplementation of importlib.import_module() which uses the import state stored in the ImportEngine instance. See [3] for a full reference.

modules, path, path_hooks, meta_path, path_importer_cache

Instance-specific versions of their process global sys equivalents

importlib.engine.GlobalImportEngine(ImportEngine)

Convenience class to provide engine-like access to the global state. Provides __import__(), import_module() and from_engine() methods like ImportEngine but writes through to the global state in sys.

To support various namespace package mechanisms, when sys.path is altered, tools like pkgutil.extend_path should be used to also modify other parts of the import state (in this case, package __path__ attributes). The path importer cache should also be invalidated when a variety of changes are made.

The ImportEngine API will provide convenience methods that automatically make related import state updates as part of a single operation.

Global variables

importlib.engine.sysengine

A precreated instance of GlobalImportEngine. Intended for use by importers and loaders that have been updated to accept optional engine parameters and with ImportEngine.from_engine(sysengine) to start with a copy of the process global import state.

No changes to finder/loader interfaces

Rather than attempting to update the PEP 302 APIs to accept additional state, this PEP proposes that ImportEngine support the content management protocol (similar to the context substitution mechanisms in the decimal module).

The context management mechanism for ImportEngine would:

• On entry: * Acquire the import lock * Substitute the global import state with the import engine’s own state
• On exit: * Restore the previous global import state * Release the import lock

The precise API for this is TBD (but will probably use a distinct context management object, along the lines of that created by decimal.localcontext).

API design for falling back to global import state

The current proposal relies on the from_engine() API to fall back to the global import state. It may be desirable to offer a variant that instead falls back to the global import state dynamically.

However, one big advantage of starting with an “as isolated as possible” design is that it becomes possible to experiment with subclasses that blur the boundaries between the engine instance state and the process global state in various ways.

Builtin and extension modules must be process global

Due to platform limitations, only one copy of each builtin and extension module can readily exist in each process. Accordingly, it is impossible for each ImportEngine instance to load such modules independently.

The simplest solution is for ImportEngine to refuse to load such modules, raising ImportError. GlobalImportEngine would be able to load them normally.

ImportEngine will still return such modules from a prepopulated module cache - it’s only loading them directly which causes problems.

Scope of substitution

Related to the previous open issue is the question of what state to substitute when using the context management API. It is currently the case that replacing sys.modules can be unreliable due to cached references and there’s the underlying fact that having independent copies of some modules is simply impossible due to platform limitations.

As part of this PEP, it will be necessary to document explicitly:

• Which parts of the global import state can be substituted (and declare code which caches references to that state without dealing with the substitution case buggy)
• Which parts must be modified in-place (and hence are not substituted by the ImportEngine context management API, or otherwise scoped to ImportEngine instances)