Isolation from system site-packages

By default, a virtual environment is entirely isolated from the system-level site-packages directories.

If the pyvenv.cfg file also contains a key include-system-site-packages with a value of true (not case sensitive), the site module will also add the system site directories to sys.path after the virtual environment site directories. Thus system-installed packages will still be importable, but a package of the same name installed in the virtual environment will take precedence.

PEP 370 user-level site-packages are considered part of the system site-packages for venv purposes: they are not available from an isolated venv, but are available from an include-system-site-packages = true venv.

Creating virtual environments

This PEP also proposes adding a new venv module to the standard library which implements the creation of virtual environments. This module can be executed using the -m flag:

python3 -m venv /path/to/new/virtual/environment

A pyvenv installed script is also provided to make this more convenient:

pyvenv /path/to/new/virtual/environment

Running this command creates the target directory (creating any parent directories that don’t exist already) and places a pyvenv.cfg file in it with a home key pointing to the Python installation the command was run from. It also creates a bin/ (or Scripts on Windows) subdirectory containing a copy (or symlink) of the python3 executable, and the pysetup3 script from the packaging standard library module (to facilitate easy installation of packages from PyPI into the new venv). And it creates an (initially empty) lib/pythonX.Y/site-packages (or Lib\site-packages on Windows) subdirectory.

If the target directory already exists an error will be raised, unless the --clear option was provided, in which case the target directory will be deleted and virtual environment creation will proceed as usual.

The created pyvenv.cfg file also includes the include-system-site-packages key, set to true if pyvenv is run with the --system-site-packages option, false by default.

Multiple paths can be given to pyvenv, in which case an identical venv will be created, according to the given options, at each provided path.

The venv module also places “shell activation scripts” for POSIX and Windows systems in the bin or Scripts directory of the venv. These scripts simply add the virtual environment’s bin (or Scripts) directory to the front of the user’s shell PATH. This is not strictly necessary for use of a virtual environment (as an explicit path to the venv’s python binary or scripts can just as well be used), but it is convenient.

In order to allow pysetup and other Python package managers to install packages into the virtual environment the same way they would install into a normal Python installation, and avoid special-casing virtual environments in sysconfig beyond using sys.base_prefix in place of sys.prefix where appropriate, the internal virtual environment layout mimics the layout of the Python installation itself on each platform. So a typical virtual environment layout on a POSIX system would be:

pyvenv.cfg
bin/python3
bin/python
bin/pysetup3
include/
lib/python3.3/site-packages/

While on a Windows system:

pyvenv.cfg
Scripts/python.exe
Scripts/python3.dll
Scripts/pysetup3.exe
Scripts/pysetup3-script.py
        ... other DLLs and pyds...
Include/
Lib/site-packages/

Third-party packages installed into the virtual environment will have their Python modules placed in the site-packages directory, and their executables placed in bin/ or Scripts.

Sysconfig install schemes and user-site

This approach explicitly chooses not to introduce a new sysconfig install scheme for venvs. Rather, by modifying sys.prefix we ensure that existing install schemes which base locations on sys.prefix will simply work in a venv. Installation to other install schemes (for instance, the user-site schemes) whose paths are not relative to sys.prefix, will not be affected by a venv at all.

It may be feasible to create an alternative implementation of Python virtual environments based on a virtual-specific sysconfig scheme, but it would be less robust, as it would require more code to be aware of whether it is operating within a virtual environment or not.

Copies versus symlinks

The technique in this PEP works equally well in general with a copied or symlinked Python binary (and other needed DLLs on Windows). Symlinking is preferable where possible, because in the case of an upgrade to the underlying Python installation, a Python executable copied in a venv might become out-of-sync with the installed standard library and require manual upgrade.

There are some cross-platform difficulties with symlinks:

• Not all Windows versions support symlinks, and even on those that do, creating them often requires administrator privileges.
• On OS X framework builds of Python, sys.executable is just a stub that executes the real Python binary. Symlinking this stub does not work; it must be copied. (Fortunately the stub is also small, and not changed by bugfix upgrades to Python, so copying it is not an issue).

Thus, this PEP proposes to symlink the binary on all platforms except for Windows, and OS X framework builds. A --symlink option is available to force the use of symlinks on Windows versions that support them, if the appropriate permissions are available. (This option has no effect on OS X framework builds, since symlinking can never work there, and has no advantages).

On Windows, if --symlink is not used, this means that if the underlying Python installation is upgraded, the Python binary and DLLs in the venv should be updated, or there could be issues of mismatch with the upgraded standard library. The pyvenv script accepts a --upgrade option for easily performing this upgrade on an existing venv.

Include files

Current virtualenv handles include files in this way:

On POSIX systems where the installed Python’s include files are found in ${base_prefix}/include/pythonX.X, virtualenv creates ${venv}/include/ and symlinks ${base_prefix}/include/pythonX.X to ${venv}/include/pythonX.X. On Windows, where Python’s include files are found in {{ sys.prefix }}/Include and symlinks are not reliably available, virtualenv copies {{ sys.prefix }}/Include to ${venv}/Include. This ensures that extension modules built and installed within the virtualenv will always find the Python header files they need in the expected location relative to sys.prefix.

This solution is not ideal when an extension module installs its own header files, as the default installation location for those header files may be a symlink to a system directory that may not be writable. One installer, pip, explicitly works around this by installing header files to a nonstandard location ${venv}/include/site/pythonX.X/, as in Python there’s currently no standard abstraction for a site-specific include directory.

This PEP proposes a slightly different approach, though one with essentially the same effect and the same set of advantages and disadvantages. Rather than symlinking or copying include files into the venv, we simply modify the sysconfig schemes so that header files are always sought relative to base_prefix rather than prefix. (We also create an include/ directory within the venv, so installers have somewhere to put include files installed within the env).

Better handling of include files in distutils/packaging and, by extension, pyvenv, is an area that may deserve its own future PEP. For now, we propose that the behavior of virtualenv has thus far proved itself to be at least “good enough” in practice.

API

The high-level method described above makes use of a simple API which provides mechanisms for third-party virtual environment creators to customize environment creation according to their needs.

The venv module contains an EnvBuilder class which accepts the following keyword arguments on instantiation:

• system_site_packages - A Boolean value indicating that the system Python site-packages should be available to the environment. Defaults to False.
• clear - A Boolean value which, if true, will delete any existing target directory instead of raising an exception. Defaults to False.
• symlinks - A Boolean value indicating whether to attempt to symlink the Python binary (and any necessary DLLs or other binaries, e.g. pythonw.exe), rather than copying. Defaults to False.

The instantiated env-builder has a create method, which takes as required argument the path (absolute or relative to the current directory) of the target directory which is to contain the virtual environment. The create method either creates the environment in the specified directory, or raises an appropriate exception.

The venv module also provides a module-level create function as a convenience:

def create(env_dir,
           system_site_packages=False, clear=False, use_symlinks=False):
    builder = EnvBuilder(
        system_site_packages=system_site_packages,
        clear=clear,
        use_symlinks=use_symlinks)
    builder.create(env_dir)

Creators of third-party virtual environment tools are free to use the provided EnvBuilder class as a base class.

The create method of the EnvBuilder class illustrates the hooks available for customization:

def create(self, env_dir):
    """
    Create a virtualized Python environment in a directory.

    :param env_dir: The target directory to create an environment in.

    """
    env_dir = os.path.abspath(env_dir)
    context = self.create_directories(env_dir)
    self.create_configuration(context)
    self.setup_python(context)
    self.post_setup(context)

Each of the methods create_directories, create_configuration, setup_python, and post_setup can be overridden. The functions of these methods are:

• create_directories - creates the environment directory and all necessary directories, and returns a context object. This is just a holder for attributes (such as paths), for use by the other methods.
• create_configuration - creates the pyvenv.cfg configuration file in the environment.
• setup_python - creates a copy of the Python executable (and, under Windows, DLLs) in the environment.
• post_setup - A (no-op by default) hook method which can be overridden in third party subclasses to pre-install packages or install scripts in the virtual environment.

In addition, EnvBuilder provides a utility method that can be called from post_setup in subclasses to assist in installing custom scripts into the virtual environment. The method install_scripts accepts as arguments the context object (see above) and a path to a directory. The directory should contain subdirectories “common”, “posix”, “nt”, each containing scripts destined for the bin directory in the environment. The contents of “common” and the directory corresponding to os.name are copied after doing some text replacement of placeholders:

• __VENV_DIR__ is replaced with absolute path of the environment directory.
• __VENV_NAME__ is replaced with the environment name (final path segment of environment directory).
• __VENV_BIN_NAME__ is replaced with the name of the bin directory (either bin or Scripts).
• __VENV_PYTHON__ is replaced with the absolute path of the environment’s executable.

The DistributeEnvBuilder subclass in the reference implementation illustrates how the customization hook can be used in practice to pre-install Distribute into the virtual environment. It’s not envisaged that DistributeEnvBuilder will be actually added to Python core, but it makes the reference implementation more immediately useful for testing and exploratory purposes.

Splitting the meanings of sys.prefix

Any virtual environment tool along these lines (which attempts to isolate site-packages, while still making use of the base Python’s standard library with no need for it to be symlinked into the virtual environment) is proposing a split between two different meanings (among others) that are currently both wrapped up in sys.prefix: the answers to the questions “Where is the standard library?” and “Where is the site-packages location where third-party modules should be installed?”

This split could be handled by introducing a new sys attribute for either the former prefix or the latter prefix. Either option potentially introduces some backwards-incompatibility with software written to assume the other meaning for sys.prefix. (Such software should preferably be using the APIs in the site and sysconfig modules to answer these questions rather than using sys.prefix directly, in which case there is no backwards-compatibility issue, but in practice sys.prefix is sometimes used.)

The documentation for sys.prefix describes it as “A string giving the site-specific directory prefix where the platform independent Python files are installed,” and specifically mentions the standard library and header files as found under sys.prefix. It does not mention site-packages.

Maintaining this documented definition would mean leaving sys.prefix pointing to the base system installation (which is where the standard library and header files are found), and introducing a new value in sys (something like sys.site_prefix) to point to the prefix for site-packages. This would maintain the documented semantics of sys.prefix, but risk breaking isolation if third-party code uses sys.prefix rather than sys.site_prefix or the appropriate site API to find site-packages directories.

The most notable case is probably setuptools and its fork distribute, which mostly use distutils and sysconfig APIs, but do use sys.prefix directly to build up a list of site directories for pre-flight checking where pth files can usefully be placed.

Otherwise, a Google Code Search turns up what appears to be a roughly even mix of usage between packages using sys.prefix to build up a site-packages path and packages using it to e.g. eliminate the standard-library from code-execution tracing.

Although it requires modifying the documented definition of sys.prefix, this PEP prefers to have sys.prefix point to the virtual environment (where site-packages is found), and introduce sys.base_prefix to point to the standard library and Python header files. Rationale for this choice:

• It is preferable to err on the side of greater isolation of the virtual environment.
• Virtualenv already modifies sys.prefix to point at the virtual environment, and in practice this has not been a problem.
• No modification is required to setuptools/distribute.

Impact on other Python implementations

The majority of this PEP’s changes occur in the standard library, which is shared by other Python implementations and should not present any problem.

Other Python implementations will need to replicate the new sys.prefix-finding behavior of the interpreter bootstrap, including locating and parsing the pyvenv.cfg file, if it is present.