Python Enhancement Proposals

PEP 408 – Standard library __preview__ package

PEP
408
Title
Standard library __preview__ package
Author
Nick Coghlan <ncoghlan at gmail.com>, Eli Bendersky <eliben at gmail.com>
Status
Rejected
Type
Standards Track
Created
07-Jan-2012
Python-Version
3.3
Post-History
27-Jan-2012
Resolution
Python-Dev

Contents

Abstract

The process of including a new module into the Python standard library is hindered by the API lock-in and promise of backward compatibility implied by a module being formally part of Python. This PEP proposes a transitional state for modules - inclusion in a special __preview__ package for the duration of a minor release (roughly 18 months) prior to full acceptance into the standard library. On one hand, this state provides the module with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the module’s eventual full inclusion into the standard library, or to the stability of its API, which may change for the next release.

PEP Rejection

Based on his experience with a similar “labs” namespace in Google App Engine, Guido has rejected this PEP [3] in favour of the simpler alternative of explicitly marking provisional modules as such in their documentation.

If a module is otherwise considered suitable for standard library inclusion, but some concerns remain regarding maintainability or certain API details, then the module can be accepted on a provisional basis. While it is considered an unlikely outcome, such modules may be removed from the standard library without a deprecation period if the lingering concerns prove well-founded.

As part of the same announcement, Guido explicitly accepted Matthew Barnett’s ‘regex’ module [4] as a provisional addition to the standard library for Python 3.3 (using the ‘regex’ name, rather than as a drop-in replacement for the existing ‘re’ module).

Proposal - the __preview__ package

Whenever the Python core development team decides that a new module should be included into the standard library, but isn’t entirely sure about whether the module’s API is optimal, the module can be placed in a special package named __preview__ for a single minor release.

In the next minor release, the module may either be “graduated” into the standard library (and occupy its natural place within its namespace, leaving the __preview__ package), or be rejected and removed entirely from the Python source tree. If the module ends up graduating into the standard library after spending a minor release in __preview__, its API may be changed according to accumulated feedback. The core development team explicitly makes no guarantees about API stability and backward compatibility of modules in __preview__.

Entry into the __preview__ package marks the start of a transition of the module into the standard library. It means that the core development team assumes responsibility of the module, similarly to any other module in the standard library.

Which modules should go through __preview__

We expect most modules proposed for addition into the Python standard library to go through a minor release in __preview__. There may, however, be some exceptions, such as modules that use a pre-defined API (for example lzma, which generally follows the API of the existing bz2 module), or modules with an API that has wide acceptance in the Python development community.

In any case, modules that are proposed to be added to the standard library, whether via __preview__ or directly, must fulfill the acceptance conditions set by PEP 2.

It is important to stress that the aim of this proposal is not to make the process of adding new modules to the standard library more difficult. On the contrary, it tries to provide a means to add more useful libraries. Modules which are obvious candidates for entry can be added as before. Modules which due to uncertainties about the API could be stalled for a long time now have a means to still be distributed with Python, via an incubation period in the __preview__ package.

Criteria for “graduation”

In principle, most modules in the __preview__ package should eventually graduate to the stable standard library. Some reasons for not graduating are:

  • The module may prove to be unstable or fragile, without sufficient developer support to maintain it.
  • A much better alternative module may be found during the preview release

Essentially, the decision will be made by the core developers on a per-case basis. The point to emphasize here is that a module’s appearance in the __preview__ package in some release does not guarantee it will continue being part of Python in the next release.

Example

Suppose the example module is a candidate for inclusion in the standard library, but some Python developers aren’t convinced that it presents the best API for the problem it intends to solve. The module can then be added to the __preview__ package in release 3.X, importable via:

from __preview__ import example

Assuming the module is then promoted to the standard library proper in release 3.X+1, it will be moved to a permanent location in the library:

import example

And importing it from __preview__ will no longer work.

Rationale

Benefits for the core development team

Currently, the core developers are really reluctant to add new interfaces to the standard library. This is because as soon as they’re published in a release, API design mistakes get locked in due to backward compatibility concerns.

By gating all major API additions through some kind of a preview mechanism for a full release, we get one full release cycle of community feedback before we lock in the APIs with our standard backward compatibility guarantee.

We can also start integrating preview modules with the rest of the standard library early, so long as we make it clear to packagers that the preview modules should not be considered optional. The only difference between preview APIs and the rest of the standard library is that preview APIs are explicitly exempted from the usual backward compatibility guarantees.

Essentially, the __preview__ package is intended to lower the risk of locking in minor API design mistakes for extended periods of time. Currently, this concern can block new additions, even when the core development team consensus is that a particular addition is a good idea in principle.

Benefits for end users

For future end users, the broadest benefit lies in a better “out-of-the-box” experience - rather than being told “oh, the standard library tools for task X are horrible, download this 3rd party library instead”, those superior tools are more likely to be just be an import away.

For environments where developers are required to conduct due diligence on their upstream dependencies (severely harming the cost-effectiveness of, or even ruling out entirely, much of the material on PyPI), the key benefit lies in ensuring that anything in the __preview__ package is clearly under python-dev’s aegis from at least the following perspectives:

  • Licensing: Redistributed by the PSF under a Contributor Licensing Agreement.
  • Documentation: The documentation of the module is published and organized via the standard Python documentation tools (i.e. ReST source, output generated with Sphinx and published on http://docs.python.org).
  • Testing: The module test suites are run on the python.org buildbot fleet and results published via http://www.python.org/dev/buildbot.
  • Issue management: Bugs and feature requests are handled on http://bugs.python.org
  • Source control: The master repository for the software is published on http://hg.python.org.

Candidates for inclusion into __preview__

For Python 3.3, there are a number of clear current candidates:

Other possible future use cases include:

  • Improved HTTP modules (e.g. requests)
  • HTML 5 parsing support (e.g. html5lib)
  • Improved URL/URI/IRI parsing
  • A standard image API (PEP 368)
  • Encapsulation of the import state (PEP 368)
  • Standard event loop API (PEP 3153)
  • A binary version of WSGI for Python 3 (e.g. PEP 444)
  • Generic function support (e.g. simplegeneric)

Relationship with PEP 407

PEP 407 proposes a change to the core Python release cycle to permit interim releases every 6 months (perhaps limited to standard library updates). If such a change to the release cycle is made, the following policy for the __preview__ namespace is suggested:

  • For long-term support releases, the __preview__ namespace would always be empty.
  • New modules would be accepted into the __preview__ namespace only in interim releases that immediately follow a long-term support release.
  • All modules added will either be migrated to their final location in the standard library or dropped entirely prior to the next long-term support release.

Rejected alternatives and variations

Using __future__

Python already has a “forward-looking” namespace in the form of the __future__ module, so it’s reasonable to ask why that can’t be re-used for this new purpose.

There are two reasons why doing so not appropriate:

1. The __future__ module is actually linked to a separate compiler directives feature that can actually change the way the Python interpreter compiles a module. We don’t want that for the preview package - we just want an ordinary Python package.

2. The __future__ module comes with an express promise that names will be maintained in perpetuity, long after the associated features have become the compiler’s default behaviour. Again, this is precisely the opposite of what is intended for the preview package - it is almost certain that all names added to the preview will be removed at some point, most likely due to their being moved to a permanent home in the standard library, but also potentially due to their being reverted to third party package status (if community feedback suggests the proposed addition is irredeemably broken).

Versioning the package

One proposed alternative [1] was to add explicit versioning to the __preview__ package, i.e. __preview34__. We think that it’s better to simply define that a module being in __preview__ in Python 3.X will either graduate to the normal standard library namespace in Python 3.X+1 or will disappear from the Python source tree altogether. Versioning the _preview__ package complicates the process and does not align well with the main intent of this proposal.

Using a package name without leading and trailing underscores

It was proposed [1] to use a package name like preview or exp, instead of __preview__. This was rejected in the discussion due to the special meaning a “dunder” package name (that is, a name with leading and trailing double-underscores) conveys in Python. Besides, a non-dunder name would suggest normal standard library API stability guarantees, which is not the intention of the __preview__ package.

Preserving pickle compatibility

A pickled class instance based on a module in __preview__ in release 3.X won’t be unpickle-able in release 3.X+1, where the module won’t be in __preview__. Special code may be added to make this work, but this goes against the intent of this proposal, since it implies backward compatibility. Therefore, this PEP does not propose to preserve pickle compatibility.

Credits

Dj Gilcrease initially proposed the idea of having a __preview__ package in Python [2]. Although his original proposal uses the name __experimental__, we feel that __preview__ conveys the meaning of this package in a better way.

References

[1] (1, 2)
Discussed in this thread: https://mail.python.org/pipermail/python-ideas/2012-January/013246.html
[2]
https://mail.python.org/pipermail/python-ideas/2011-August/011278.html
[3]
Guido’s decision: https://mail.python.org/pipermail/python-dev/2012-January/115962.html
[4]
Proposal for inclusion of regex: http://bugs.python.org/issue2636

Source: https://github.com/python-discord/peps/blob/main/pep-0408.txt

Last modified: 2022-03-09 16:04:44 GMT