PEP 527 – Removing Un(der)used file types/extensions on PyPI
- Removing Un(der)used file types/extensions on PyPI
- Donald Stufft <donald at stufft.io>
- Nick Coghlan <ncoghlan at gmail.com>
This PEP recommends deprecating, and ultimately removing, support for uploading
certain unused or under used file types and extensions to PyPI. In particular
it recommends disallowing further uploads of any files of the types
bdist_wininst, leaving PyPI to only accept new uploads of the
bdist_egg file types.
In addition, this PEP proposes removing support for new uploads of sdists using
any other extension besides
Finally, this PEP also proposes limiting the number of allowed sdist uploads for each individual release of a project on PyPI to one instead of one for each allowed extension.
Currently PyPI supports the following file types:
However, these different types of files have varying amounts of usefulness or general use in the ecosystem. Continuing to support them adds a maintenance burden on PyPI as well as tool authors and incurs a cost in both bandwidth and disk space not only on PyPI itself, but also on any mirrors of PyPI.
Python packaging is a multi-level ecosystem where PyPI is primarily suited and used to distribute virtual environment compatible packages directly from their respective project owners. These packages are then consumed either directly by end-users, or by downstream distributors that take these packages and turn them into their respective system level packages (such as RPM, deb, MSI, etc).
While PyPI itself only directly works with these Python specific but platform agnostic packages, we encourage community-driven and commercial conversions of these packages to downstream formats for particular target environments, like:
- The conda cross-platform data analysis ecosystem (conda-forge)
- The deb based Linux ecosystem (Debian, Ubuntu, etc)
- The RPM based Linux ecosystem (Fedora, openSuSE, Mageia, etc)
- The homebrew, MacPorts and fink ecosystems for Mac OS X
- The Windows Package Management ecosystem (NuGet, Chocolatey, etc)
- 3rd party creation of Windows MSIs and installers (e.g. Christoph Gohlke’s work at http://www.lfd.uci.edu/~gohlke/pythonlibs/ )
- other commercial redistribution formats (ActiveState’s PyPM, Enthought Canopy, etc)
- other open source community redistribution formats (Nix, Gentoo, Arch, *BSD, etc)
It is the belief of this PEP that the entire ecosystem is best supported by keeping PyPI focused on the platform agnostic formats, where the limited amount of time by volunteers can be best used instead of spreading the available time out amongst several platforms. Further more, this PEP believes that the people best positioned to provide well integrated packages for a particular platform are people focused on that platform, and not across all possible platforms.
As it’s name implies,
bdist_dumb is not a very complex format, however it
is so simple as to be worthless for actual usage.
For instance, if you’re using something like pyenv on macOS and you’re building
a library using Python 3.5, then
bdist_dumb will produce a
named something like
off the bat this file name is somewhat difficult to differentiate from an
sdist since they both use the same file extension (and with the legacy pre
PEP 440 versions,
1.0-macosx-10.11-x86_64 is a valid, although quite silly,
version number). However, once you open up the created
.tar.gz, you’d find
that there is no metadata inside that could be used for things like dependency
discovery and in fact, it is quite simply a tarball containing hardcoded paths
to wherever files would have been installed on the computer creating the
bdist_dumb. Going back to our pyenv on macOS example, this means that if I
created it, it would contain files like:
bdist_rpm format on PyPI allows people to upload
.rpm files for
end users to manually download by hand and then manually install by hand.
However, the common usage of
rpm is with a specially designed repository
that allows automatic installation of dependencies, upgrades, etc which PyPI
does not provide. Thus, it is a type of file that is barely being used on PyPI
with only ~460 files of this type having been uploaded to PyPI (out a total of
In addition, services like COPR provide a better supported mechanism for publishing and using RPM files than we’re ever likely to get on PyPI.
bdist_dmg, bdist_msi, and bdist_wininst
bdist_winist formats are similar in
that they are an OS specific installer that will only install a library into an
environment and are not designed for real user facing installs of applications
(which would require things like bundling a Python interpreter and the like).
Out of these three, the usage for
bdist_msi is very low,
with only ~500
bdist_msi files and ~50
bdist_dmg files having been
uploaded to PyPI. The
bdist_wininst format has more use, with ~14,000 files
having ever been uploaded to PyPI.
It’s quite easy to look at the low usage of
conclude that removing them will be fairly low impact, however
bdist_wininst has several orders of magnitude more usage. This is somewhat
misleading though, because although it has more people uploading those files
the actual usage of those uploaded files is fairly low. Taking a look at the
previous 30 days, we can see that 90% of all downloads of
files from PyPI were generated by the mirroring infrastructure and 7% of them
were generated by setuptools (which can currently be better covered by
Given the small number of files uploaded for
bdist_wininst is largely existing to either consume bandwidth and
disk space via the mirroring infrastructure or could be trivially replaced
bdist_egg, this PEP proposes to include these three formats in the
list of those to be disallowed.
sdist supports a wide variety of file extensions like
.tbz. However, of those the only extensions which get anything more than
negligible usage is
.tar.gz with 444,338 sdists currently,
58,774 sdists currently, and
.tar.bz2 with 3,265 sdists currently.
Having multiple formats accepted requires tooling both within PyPI and outside
of PyPI to handle all of the various extensions that might be used (even if
nobody is currently using them). This doesn’t only affect PyPI, but ripples out
throughout the ecosystem. In addition, the different formats all have different
requirements for what optional C libraries Python was linked against and
different requirements for what versions of Python they support. In addition,
multiple formats also create a weird situation where there may be two
sdist files for a particular project/release with subtly different content.
It’s easy to advocate that anything outside of
.tar.bz2 should be disallowed. Outside of a tiny handful, nobody has
actively been uploading these other types of files in the ~15 years of PyPI’s
existence so they’ve obviously not been particularly useful. In addition, while
.tar.xz is theoretically a nicer format than the other
due to the better compression ratio achieved by LZMA, it is only available in
Python 3.3+ and has an optional dependency on the lzma C library.
Looking at the three extensions we do have in current use, it’s also fairly
easy to conclude that
.tar.bz2 can be disallowed as well. It has a fairly
small number of files ever uploaded with it and it requires an additional
optional C library to handle the bzip2 compression.
Finally we get down to
.zip. Looking at the pure numbers
for these two, we can see that
.tar.gz is by far the most uploaded format,
with 444,338 total uploaded compared to
.zip’s 58,774 and on POSIX
.tar.gz is also the default produced by all currently
released versions of Python and setuptools. In addition, these two file types
both use the same C library (
zlib) which is also required for
bdist_egg. The two wrinkles with deciding between
.zip is that while on POSIX operating systems
is the default, on Windows
.zip is the default and the
format also uses zip.
Instead of trying to standardize on either
.zip, this PEP
proposes that we allow either
.zip for sdists.
Limiting number of sdists per release
A sdist on PyPI should be a single source of truth for a particular release of software. However, currently PyPI allows you to upload one sdist for each of the sdist file extensions it allows. Currently this allows something like 10 different sdists for a project, but even with this PEP it allows two different sources of truth for a single version. Having multiple sdists oftentimes can account for strange bugs that only expose themselves based on which sdist that the person used.
To resolve this, this PEP proposes to allow one, and only one, sdist per release of a project.
This PEP does NOT propose removing any existing files from PyPI, only disallowing new ones from being uploaded. This restriction will be phased in on a per-project basis to allow projects to adjust to the new restrictions where applicable.
First, any existing projects will be flagged to allow legacy file types to be
uploaded, and any project without that flag (i.e. new projects) will not be
able to upload anything but
sdist with a
bdist_egg. Then, any existing projects that have never
uploaded a file that requires the legacy file type flag will have that flag
removed, also making them fall under the new restrictions. Finally, an email
will be generated to the maintainers of all projects still given the legacy
flag, which will inform them of the upcoming new restrictions on uploads and
tell them that these restrictions will be applied to future uploads to their
projects starting in 1 month. Finally, after 1 month all projects will have the
legacy file type flag removed, and support for uploading these types of files
will cease to exist on PyPI.
This plan should provide minimal disruption since it does not remove any existing files, and the types of files it does prevent from being uploaded are either not particularly useful (or used) types of files or they can continue to upload a similar type of file with a slight change to their process.
This document has been placed in the public domain.
Last modified: 2022-01-21 11:03:51 GMT