Abstract

Following PEP 440, a distribution can be identified by a name and either a version, or a direct URL reference (see PEP440 Direct References). After installation, the name and version are captured in the project metadata, but currently there is no way to obtain details of the URL used when the distribution was identified by a direct URL reference.

This proposal defines additional metadata, to be added to the installed distribution by the installation front end, which records the Direct URL Origin for use by consumers which introspect the database of installed packages (see PEP 376).

Provisional acceptance

In accordance with the PyPA’s specification process, this PEP has been provisionally accepted for initial implementation in pip and other PyPA tools.

During this time, the specification is still subject to revision based on real world experience with those initial implementations.

Motivation

The original motivation of this PEP was to permit tools with a “freeze” operation allowing a Python environment to be recreated to work in a broader range of situations.

Specifically, the PEP originated from the desire to address pip issue #609: i.e. improving the behavior of pip freeze in the presence of distributions installed from direct URL references. It follows a thread on discuss.python.org about the best course of action to implement it.

Rationale

This PEP specifies a new direct_url.json metadata file in the .dist-info directory of an installed distribution.

The fields specified are sufficient to reproduce the source archive and VCS URLs supported by pip. They are also sufficient to reproduce PEP440 Direct References, as well as Pipfile and Pipfile.lock entries. Finally, they are sufficient to record the branch, tag, and/or Git ref origin of the installed version that is already available for editable installs by virtue of a VCS checkout being present.

Since at least three different ways already exist to encode this type of information, this PEP uses a dictionary format, so as not to make any assumption on how a direct URL reference must ultimately be encoded in a requirement or lockfile. See also the Alternatives section below for more discussion about this choice.

Information has been taken from Ruby’s bundler manual to verify it has similar capabilities and inform the selection and naming of fields in this specifications.

The JSON format allows for the addition of additional fields in the future.

Specification

This PEP specifies a direct_url.json file in the .dist-info directory of an installed distribution, to record the Direct URL Origin of the distribution.

The canonical source for the name and semantics of this metadata file is the Recording the Direct URL Origin of installed distributions document.

This file MUST be created by installers when installing a distribution from a requirement specifying a direct URL reference (including a VCS URL).

This file MUST NOT be created when installing a distribution from an other type of requirement (i.e. name plus version specifier).

This JSON file MUST be a dictionary, compliant with RFC 8259 and UTF-8 encoded.

If present, it MUST contain at least two fields. The first one is url, with type string. Depending on what url refers to, the second field MUST be one of vcs_info (if url is a VCS reference), archive_info (if url is a source archives or a wheel), or dir_info (if url is a local directory). These info fields have a (possibly empty) subdictionary as value, with the possible keys defined below.

url MUST be stripped of any sensitive authentication information, for security reasons.

The user:password section of the URL MAY however be composed of environment variables, matching the following regular expression:

\$\{[A-Za-z0-9-_]+\}(:\$\{[A-Za-z0-9-_]+\})?

Additionally, the user:password section of the URL MAY be a well-known, non security sensitive string. A typical example is git in the case of an URL such as ssh://git@gitlab.com.

When url refers to a VCS repository, the vcs_info key MUST be present as a dictionary with the following keys:

• A vcs key (type string) MUST be present, containing the name of the VCS (i.e. one of git, hg, bzr, svn). Other VCS’s SHOULD be registered by writing a PEP to amend this specification. The url value MUST be compatible with the corresponding VCS, so an installer can hand it off without transformation to a checkout/download command of the VCS.
• A requested_revision key (type string) MAY be present naming a branch/tag/ref/commit/revision/etc (in a format compatible with the VCS) to install.
• A commit_id key (type string) MUST be present, containing the exact commit/revision number that was installed. If the VCS supports commit-hash based revision identifiers, such commit-hash MUST be used as commit_id in order to reference the immutable version of the source code that was installed.
• If the installer could discover additional information about the requested revision, it MAY add a resolved_revision and/or resolved_revision_type field. If no revision was provided in the requested URL, resolved_revision MAY contain the default branch that was installed, and resolved_revision_type will be branch. If the installer determines that requested_revision was a tag, it MAY add resolved_revision_type with value tag.

When url refers to a source archive or a wheel, the archive_info key MUST be present as a dictionary with the following key:

• A hash key (type string) SHOULD be present, with value <hash-algorithm>=<expected-hash>. It is RECOMMENDED that only hashes which are unconditionally provided by the latest version of the standard library’s hashlib module be used for source archive hashes. At time of writing, that list consists of ‘md5’, ‘sha1’, ‘sha224’, ‘sha256’, ‘sha384’, and ‘sha512’.

When url refers to a local directory, the dir_info key MUST be present as a dictionary with the following key:

• editable (type: boolean): true if the distribution was installed in editable mode, false otherwise. If absent, default to false.

When url refers to a local directory, it MUST have the file scheme and be compliant with RFC 8089. In particular, the path component must be absolute. Symbolic links SHOULD be preserved when making relative paths absolute.

A top-level subdirectory field MAY be present containing a directory path, relative to the root of the VCS repository, source archive or local directory, to specify where pyproject.toml or setup.py is located.

Examples

{
    "url": "https://github.com/pypa/pip/archive/1.3.1.zip",
    "archive_info": {
        "hash": "sha256=2dc6b5a470a1bde68946f263f1af1515a2574a150a30d6ce02c6ff742fcc0db8"
    }
}

{
    "url": "https://github.com/pypa/pip.git",
    "vcs_info": {
        "vcs": "git",
        "requested_revision": "1.3.1",
        "resolved_revision_type": "tag",
        "commit_id": "7921be1537eac1e97bc40179a57f0349c2aee67d"
    }
}

{
    "url": "file:///home/user/project",
    "dir_info": {}
}

{
    "url": "file:///home/user/project",
    "dir_info": {
        "editable": true
    }
}

Use cases

“Freezing” an environment

Tools, such as pip freeze, which generate requirements from the Database of Installed Python Distributions SHOULD exploit direct_url.json if it is present, and give it priority over the Version metadata in order to generate a higher fidelity output. In the presence of a vcs direct URL reference, the commit_id field SHOULD be used in priority in order to provide the highest possible fidelity to the originally installed version. If supported by their requirement format, tools are encouraged also to output the tag value if present, as it has immutable semantics. Tools MAY choose another approach, depending on the needs of their users.

Note the initial iteration of this PEP does not attempt to make environments that include editable installs or installs from local directories reproducible, but it does attempt to make them readily identifiable. By locating the local project directory via the url and dir_info fields of this specification, tools can implement any strategy that fits their use cases.

Backwards Compatibility

Since this PEP specifies a new file in the .dist-info directory, there are no backwards compatibility implications.

Alternatives

Acknowledgements

Various people helped make this PEP a reality. Paul F. Moore provided the essence of the abstract. Nick Coghlan suggested the direct_url name.

Copyright

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.