Legacy C locale coercion in the standalone Python interpreter binary

When run as a standalone application, CPython has the opportunity to reconfigure the C locale before any locale dependent operations are executed in the process.

This means that it can change the locale settings not only for the CPython runtime, but also for any other locale-aware components running in the current process (e.g. as part of extension modules), as well as in subprocesses that inherit their environment from the current process.

After calling setlocale(LC_ALL, "") to initialize the locale settings in the current process, the main interpreter binary will be updated to include the following call:

const char *ctype_loc = setlocale(LC_CTYPE, NULL);

This cryptic invocation is the API that C provides to query the current locale setting without changing it. Given that query, it is possible to check for exactly the C locale with strcmp:

ctype_loc != NULL && strcmp(ctype_loc, "C") == 0 # true only in the C locale

This call also returns "C" when either no particular locale is set, or the nominal locale is set to an alias for the C locale (such as POSIX).

Given this information, CPython can then attempt to coerce the locale to one that uses UTF-8 rather than ASCII as the default encoding.

Three such locales will be tried:

• C.UTF-8 (available at least in Debian, Ubuntu, Alpine, and Fedora 25+, and expected to be available by default in a future version of glibc)
• C.utf8 (available at least in HP-UX)
• UTF-8 (available in at least some *BSD variants, including Mac OS X)

The coercion will be implemented by setting the LC_CTYPE environment variable to the candidate locale name, such that future calls to setlocale() will see it, as will other components looking for those settings (such as GUI development frameworks and Python’s own locale module).

To allow for better cross-platform binary portability and to adjust automatically to future changes in locale availability, these checks will be implemented at runtime on all platforms other than Windows, rather than attempting to determine which locales to try at compile time.

When this locale coercion is activated, the following warning will be printed on stderr, with the warning containing whichever locale was successfully configured:

Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another
locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion behaviour).

(Note: this warning ended up being silenced by default. See the Implementation Note above for more details)

As long as the current platform provides at least one of the candidate UTF-8 based environments, this locale coercion will mean that the standard Python binary and locale-aware extensions should once again “just work” in the three main failure cases we’re aware of (missing locale settings, SSH forwarding of unknown locales via LANG or LC_CTYPE, and developers explicitly requesting LANG=C).

The one case where failures may still occur is when stderr is specifically being checked for no output, which can be resolved either by configuring a locale other than the C locale, or else by using a mechanism other than “there was no output on stderr” to check for subprocess errors (e.g. checking process return codes).

If none of the candidate locales are successfully configured, or the LC_ALL, locale override is defined in the current process environment, then initialization will continue in the C locale and the Unicode compatibility warning described in the next section will be emitted just as it would for any other application.

If PYTHONCOERCECLOCALE=0 is explicitly set, initialization will continue in the C locale and the Unicode compatibility warning described in the next section will be automatically suppressed.

The interpreter will always check for the PYTHONCOERCECLOCALE environment variable at startup (even when running under the -E or -I switches), as the locale coercion check necessarily takes place before any command line argument processing. For consistency, the runtime check to determine whether or not to suppress the locale compatibility warning will be similarly independent of these settings.

Legacy C locale warning during runtime initialization

By the time that Py_Initialize is called, arbitrary locale-dependent operations may have taken place in the current process. This means that by the time it is called, it is too late to reliably switch to a different locale - doing so would introduce inconsistencies in decoded text, even in the context of the standalone Python interpreter binary.

Accordingly, when Py_Initialize is called and CPython detects that the configured locale is still the default C locale and PYTHONCOERCECLOCALE=0 is not set, the following warning will be issued:

Python runtime initialized with LC_CTYPE=C (a locale with default ASCII
encoding), which may cause Unicode compatibility problems. Using C.UTF-8,
C.utf8, or UTF-8 (if available) as alternative Unicode-compatible
locales is recommended.

(Note: this warning ended up being silenced by default. See the Implementation Note above for more details)

In this case, no actual change will be made to the locale settings.

Instead, the warning informs both system and application integrators that they’re running Python 3 in a configuration that we don’t expect to work properly.

The second sentence providing recommendations may eventually be conditionally compiled based on the operating system (e.g. recommending LC_CTYPE=UTF-8 on *BSD systems), but the initial implementation will just use the common generic message shown above.

New build-time configuration options

While both of the above behaviours would be enabled by default, they would also have new associated configuration options and preprocessor definitions for the benefit of redistributors that want to override those default settings.

The locale coercion behaviour would be controlled by the flag --with[out]-c-locale-coercion, which would set the PY_COERCE_C_LOCALE preprocessor definition.

The locale warning behaviour would be controlled by the flag --with[out]-c-locale-warning, which would set the PY_WARN_ON_C_LOCALE preprocessor definition.

(Note: this compile time warning option ended up being replaced by a runtime PYTHONCOERCECLOCALE=warn option. See the Implementation Note above for more details)

On platforms which don’t use the autotools based build system (i.e. Windows) these preprocessor variables would always be undefined.

Changes to the default error handling on the standard streams

Since Python 3.5, CPython has defaulted to using surrogateescape on the standard streams (sys.stdin, sys.stdout) when it detects that the current locale is C and no specific error handled has been set using either the PYTHONIOENCODING environment variable or the Py_setStandardStreamEncoding API. For other locales, the default error handler for the standard streams is strict.

In order to preserve this behaviour without introducing any behavioural discrepancies between locale coercion and explicitly configuring a locale, the coercion target locales (C.UTF-8, C.utf8, and UTF-8) will be added to the list of locales that use surrogateescape as their default error handler for the standard streams.

No changes are proposed to the default error handler for sys.stderr: that will continue to be backslashreplace.

Changes to locale settings on Android

Independently of the other changes in this PEP, CPython on Android systems will be updated to call setlocale(LC_ALL, "C.UTF-8") where it currently calls setlocale(LC_ALL, "") and setlocale(LC_CTYPE, "C.UTF-8") where it currently calls setlocale(LC_CTYPE, "").

This Android-specific behaviour is being introduced due to the following Android-specific details:

• on Android, passing "" to setlocale is equivalent to passing "C"
• the C.UTF-8 locale is always available

Improving the handling of the C locale

It has been clear for some time that the C locale’s default encoding of ASCII is entirely the wrong choice for development of modern networked services. Newer languages like Rust and Go have eschewed that default entirely, and instead made it a deployment requirement that systems be configured to use UTF-8 as the text encoding for operating system interfaces. Similarly, Node.js assumes UTF-8 by default (a behaviour inherited from the V8 JavaScript engine) and requires custom build settings to indicate it should use the system locale settings for locale-aware operations. Both the JVM and the .NET CLR use UTF-16-LE as their primary encoding for passing text between applications and the application runtime (i.e. the JVM/CLR, not the host operating system).

The challenge for CPython has been the fact that in addition to being used for network service development, it is also extensively used as an embedded scripting language in larger applications, and as a desktop application development language, where it is more important to be consistent with other locale-aware components sharing the same process, as well as with the user’s desktop locale settings, than it is with the emergent conventions of modern network service development.

The core premise of this PEP is that for all of these use cases, the assumption of ASCII implied by the default “C” locale is the wrong choice, and furthermore that the following assumptions are valid:

• in desktop application use cases, the process locale will already be configured appropriately, and if it isn’t, then that is an operating system or embedding application level problem that needs to be reported to and resolved by the operating system provider or application developer
• in network service development use cases (especially those based on Linux containers), the process locale may not be configured at all, and if it isn’t, then the expectation is that components will impose their own default encoding the way Rust, Go and Node.js do, rather than trusting the legacy C default encoding of ASCII the way CPython currently does

Defaulting to “surrogateescape” error handling on the standard IO streams

By coercing the locale away from the legacy C default and its assumption of ASCII as the preferred text encoding, this PEP also disables the implicit use of the “surrogateescape” error handler on the standard IO streams that was introduced in Python 3.5 ([15]), as well as the automatic use of surrogateescape when operating in PEP 540’s proposed UTF-8 mode.

Rather than introducing yet another configuration option to adjust that behaviour, this PEP instead proposes to extend the “surrogateescape” default for stdin and stderr error handling to also apply to the three potential coercion target locales.

The aim of this behaviour is to attempt to ensure that operating system provided text values are typically able to be transparently passed through a Python 3 application even if it is incorrect in assuming that that text has been encoded as UTF-8.

In particular, GB 18030 [12] is a Chinese national text encoding standard that handles all Unicode code points, that is formally incompatible with both ASCII and UTF-8, but will nevertheless often tolerate processing as surrogate escaped data - the points where GB 18030 reuses ASCII byte values in an incompatible way are likely to be invalid in UTF-8, and will therefore be escaped and opaque to string processing operations that split on or search for the relevant ASCII code points. Operations that don’t involve splitting on or searching for particular ASCII or Unicode code point values are almost certain to work correctly.

Similarly, Shift-JIS [13] and ISO-2022-JP [14] remain in widespread use in Japan, and are incompatible with both ASCII and UTF-8, but will tolerate text processing operations that don’t involve splitting on or searching for particular ASCII or Unicode code point values.

As an example, consider two files, one encoded with UTF-8 (the default encoding for en_AU.UTF-8), and one encoded with GB-18030 (the default encoding for zh_CN.gb18030):

$ python3 -c 'open("utf8.txt", "wb").write("ℙƴ☂ℌøἤ\n".encode("utf-8"))'
$ python3 -c 'open("gb18030.txt", "wb").write("ℙƴ☂ℌøἤ\n".encode("gb18030"))'

On disk, we can see that these are two very different files:

$ python3 -c 'print("UTF-8:  ", open("utf8.txt", "rb").read().strip()); \
              print("GB18030:", open("gb18030.txt", "rb").read().strip())'
UTF-8:   b'\xe2\x84\x99\xc6\xb4\xe2\x98\x82\xe2\x84\x8c\xc3\xb8\xe1\xbc\xa4\n'
GB18030: b'\x816\xbd6\x810\x9d0\x817\xa29\x816\xbc4\x810\x8b3\x816\x8d6\n'

That nevertheless can both be rendered correctly to the terminal as long as they’re decoded prior to printing:

$ python3 -c 'print("UTF-8:  ", open("utf8.txt", "r", encoding="utf-8").read().strip()); \
              print("GB18030:", open("gb18030.txt", "r", encoding="gb18030").read().strip())'
UTF-8:   ℙƴ☂ℌøἤ
GB18030: ℙƴ☂ℌøἤ

By contrast, if we just pass along the raw bytes, as cat and similar C/C++ utilities will tend to do:

$ LANG=en_AU.UTF-8 cat utf8.txt gb18030.txt
ℙƴ☂ℌøἤ
�6�6�0�0�7�9�6�4�0�3�6�6

Even setting a specifically Chinese locale won’t help in getting the GB-18030 encoded file rendered correctly:

$ LANG=zh_CN.gb18030 cat utf8.txt gb18030.txt
ℙƴ☂ℌøἤ
�6�6�0�0�7�9�6�4�0�3�6�6

The problem is that the terminal encoding setting remains UTF-8, regardless of the nominal locale. A GB18030 terminal can be emulated using the iconv utility:

$ cat utf8.txt gb18030.txt | iconv -f GB18030 -t UTF-8
鈩櫰粹槀鈩屆羔激
ℙƴ☂ℌøἤ

This reverses the problem, such that the GB18030 file is rendered correctly, but the UTF-8 file has been converted to unrelated hanzi characters, rather than the expected rendering of “Python” as non-ASCII characters.

With the emulated GB18030 terminal encoding, assuming UTF-8 in Python results in both files being displayed incorrectly:

$ python3 -c 'print("UTF-8:  ", open("utf8.txt", "r", encoding="utf-8").read().strip()); \
              print("GB18030:", open("gb18030.txt", "r", encoding="gb18030").read().strip())' \
  | iconv -f GB18030 -t UTF-8
UTF-8:   鈩櫰粹槀鈩屆羔激
GB18030: 鈩櫰粹槀鈩屆羔激

However, setting the locale correctly means that the emulated GB18030 terminal now displays both files as originally intended:

$ LANG=zh_CN.gb18030 \
  python3 -c 'print("UTF-8:  ", open("utf8.txt", "r", encoding="utf-8").read().strip()); \
              print("GB18030:", open("gb18030.txt", "r", encoding="gb18030").read().strip())' \
  | iconv -f GB18030 -t UTF-8
UTF-8:   ℙƴ☂ℌøἤ
GB18030: ℙƴ☂ℌøἤ

The rationale for retaining surrogateescape as the default IO encoding is that it will preserve the following helpful behaviour in the C locale:

$ cat gb18030.txt \
  | LANG=C python3 -c "import sys; print(sys.stdin.read())" \
  | iconv -f GB18030 -t UTF-8
ℙƴ☂ℌøἤ

Rather than reverting to the exception currently seen when a UTF-8 based locale is explicitly configured:

$ cat gb18030.txt \
  | python3 -c "import sys; print(sys.stdin.read())" \
  | iconv -f GB18030 -t UTF-8
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib64/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x81 in position 0: invalid start byte

As an added benefit, environments explicitly configured to use one of the coercion target locales will implicitly gain the encoding transparency behaviour currently enabled by default in the C locale.

Avoiding setting PYTHONIOENCODING during UTF-8 locale coercion

Rather than changing the default handling of the standard streams during interpreter initialization, earlier versions of this PEP proposed setting PYTHONIOENCODING to utf-8:surrogateescape. This turned out to create a significant compatibility problem: since the surrogateescape handler only exists in Python 3.1+, running Python 2.7 processes in subprocesses could potentially break in a confusing way with that configuration.

The current design means that earlier Python versions will instead retain their default strict error handling on the standard streams, while Python 3.7+ will consistently use the more permissive surrogateescape handler even when these locales are explicitly configured (rather than being reached through locale coercion).

Dropping official support for ASCII based text handling in the legacy C locale

We’ve been trying to get strict bytes/text separation to work reliably in the legacy C locale for over a decade at this point. Not only haven’t we been able to get it to work, neither has anyone else - the only viable alternatives identified have been to pass the bytes along verbatim without eagerly decoding them to text (C/C++, Python 2.x, Ruby, etc), or else to largely ignore the nominal C/C++ locale encoding and assume the use of either UTF-8 (PEP 540, Rust, Go, Node.js, etc) or UTF-16-LE (JVM, .NET CLR).

While this PEP ensures that developers that genuinely need to do so can still opt-in to running their Python code in the legacy C locale (by setting LC_ALL=C, PYTHONCOERCECLOCALE=0, or running a custom build that sets --without-c-locale-coercion), it also makes it clear that we don’t expect Python 3’s Unicode handling to be completely reliable in that configuration, and the recommended alternative is to use a more appropriate locale setting (potentially in combination with PEP 540’s UTF-8 mode, if that is available).

Providing implicit locale coercion only when running standalone

The major downside of the proposed design in this PEP is that it introduces a potential discrepancy between the behaviour of the CPython runtime when it is run as a standalone application and when it is run as an embedded component inside a larger system (e.g. mod_wsgi running inside Apache httpd).

Over the course of Python 3.x development, multiple attempts have been made to improve the handling of incorrect locale settings at the point where the Python interpreter is initialised. The problem that emerged is that this is ultimately too late in the interpreter startup process - data such as command line arguments and the contents of environment variables may have already been retrieved from the operating system and processed under the incorrect ASCII text encoding assumption well before Py_Initialize is called.

The problems created by those inconsistencies were then even harder to diagnose and debug than those created by believing the operating system’s claim that ASCII was a suitable encoding to use for operating system interfaces. This was the case even for the default CPython binary, let alone larger C/C++ applications that embed CPython as a scripting engine.

The approach proposed in this PEP handles that problem by moving the locale coercion as early as possible in the interpreter startup sequence when running standalone: it takes place directly in the C-level main() function, even before calling in to the Py_Main() library function that implements the features of the CPython interpreter CLI.

The Py_Initialize API then only gains an explicit warning (emitted on stderr) when it detects use of the C locale, and relies on the embedding application to specify something more reasonable.

That said, the reference implementation for this PEP adds most of the functionality to the shared library, with the CLI being updated to unconditionally call two new private APIs:

if (_Py_LegacyLocaleDetected()) {
    _Py_CoerceLegacyLocale();
}

These are similar to other “pre-configuration” APIs intended for embedding applications: they’re designed to be called before Py_Initialize, and hence change the way the interpreter gets initialized.

If these were made public (either as part of this PEP or in a subsequent RFE), then it would be straightforward for other embedding applications to recreate the same behaviour as is proposed for the CPython CLI.

Allowing restoration of the legacy behaviour

The CPython command line interpreter is often used to investigate faults that occur in other applications that embed CPython, and those applications may still be using the C locale even after this PEP is implemented.

Providing a simple on/off switch for the locale coercion behaviour makes it much easier to reproduce the behaviour of such applications for debugging purposes, as well as making it easier to reproduce the behaviour of older 3.x runtimes even when running a version with this change applied.

Querying LC_CTYPE for C locale detection

LC_CTYPE is the actual locale category that CPython relies on to drive the implicit decoding of environment variables, command line arguments, and other text values received from the operating system.

As such, it makes sense to check it specifically when attempting to determine whether or not the current locale configuration is likely to cause Unicode handling problems.

Explicitly setting LC_CTYPE for UTF-8 locale coercion

Python is often used as a glue language, integrating other C/C++ ABI compatible components in the current process, and components written in arbitrary languages in subprocesses.

Setting LC_CTYPE to C.UTF-8 is important to handle cases where the problem has arisen from a setting like LC_CTYPE=UTF-8 being provided on a system where no UTF-8 locale is defined (e.g. when a Mac OS X ssh client is configured to forward locale settings, and the user logs into a Linux server).

This should be sufficient to ensure that when the locale coercion is activated, the switch to the UTF-8 based locale will be applied consistently across the current process and any subprocesses that inherit the current environment.

Avoiding setting LANG for UTF-8 locale coercion

Earlier versions of this PEP proposed setting the LANG category independent default locale, in addition to setting LC_CTYPE.

This was later removed on the grounds that setting only LC_CTYPE is sufficient to handle all of the problematic scenarios that the PEP aimed to resolve, while setting LANG as well would break cases where LANG was set correctly, and the locale problems were solely due to an incorrect LC_CTYPE setting ([22]).

For example, consider a Python application that called the Linux date utility in a subprocess rather than doing its own date formatting:

$ LANG=ja_JP.UTF-8 LC_CTYPE=C date
2017年  5月 23日 火曜日 17:31:03 JST

$ LANG=ja_JP.UTF-8 LC_CTYPE=C.UTF-8 date  # Coercing only LC_CTYPE
2017年  5月 23日 火曜日 17:32:58 JST

$ LANG=C.UTF-8 LC_CTYPE=C.UTF-8 date  # Coercing both of LC_CTYPE and LANG
Tue May 23 17:31:10 JST 2017

With only LC_CTYPE updated in the Python process, the subprocess would continue to behave as expected. However, if LANG was updated as well, that would effectively override the LC_TIME setting and use the wrong date formatting conventions.

Avoiding setting LC_ALL for UTF-8 locale coercion

Earlier versions of this PEP proposed setting the LC_ALL locale override, in addition to setting LC_CTYPE.

This was changed after it was determined that just setting LC_CTYPE and LANG should be sufficient to handle all the scenarios the PEP aims to cover, as it avoids causing any problems in cases like the following:

$ LANG=C LC_MONETARY=ja_JP.utf8 ./python -c \
  "from locale import setlocale, LC_ALL, currency; setlocale(LC_ALL, ''); print(currency(1e6))"
￥1000000

Skipping locale coercion if LC_ALL is set in the current environment

With locale coercion now only setting LC_CTYPE and LANG, it will have no effect if LC_ALL is also set. To avoid emitting a spurious locale coercion notice in that case, coercion is instead skipped entirely.

Considering locale coercion independently of “UTF-8 mode”

With both this PEP’s locale coercion and PEP 540’s UTF-8 mode under consideration for Python 3.7, it makes sense to ask whether or not we can limit ourselves to only doing one or the other, rather than making both changes.

The UTF-8 mode proposed in PEP 540 has two major limitations that make it a potential complement to this PEP rather than a potential replacement.

First, unlike this PEP, PEP 540’s UTF-8 mode makes it possible to change default behaviours that are not currently configurable at all. While that’s exactly what makes the proposal interesting, it’s also what makes it an entirely unproven approach. By contrast, the approach proposed in this PEP builds directly atop existing configuration settings for the C locale system ( LC_CTYPE, LANG) and Python’s standard streams (PYTHONIOENCODING) that have already been in use for years to handle the kinds of compatibility problems discussed in this PEP.

Secondly, one of the things we know based on that experience is that the proposed locale coercion can resolve problems not only in CPython itself, but also in extension modules that interact with the standard streams, like GNU readline. As an example, consider the following interactive session from a PEP 538 enabled CPython build, where each line after the first is executed by doing “up-arrow, left-arrow x4, delete, enter”:

$ LANG=C ./python
Python 3.7.0a0 (heads/pep538-coerce-c-locale:188e780, May  7 2017, 00:21:13)
[GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print("ℙƴ☂ℌøἤ")
ℙƴ☂ℌøἤ
>>> print("ℙƴ☂ℌἤ")
ℙƴ☂ℌἤ
>>> print("ℙƴ☂ἤ")
ℙƴ☂ἤ
>>> print("ℙƴἤ")
ℙƴἤ
>>> print("ℙἤ")
ℙἤ
>>> print("ἤ")
ἤ
>>>

This is exactly what we’d expect from a well-behaved command history editor.

By contrast, the following is what currently happens on an older release if you only change the Python level stream encoding settings without updating the locale settings:

$ LANG=C PYTHONIOENCODING=utf-8:surrogateescape python3
Python 3.5.3 (default, Apr 24 2017, 13:32:13)
[GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print("ℙƴ☂ℌøἤ")
ℙƴ☂ℌøἤ
>>> print("ℙƴ☂ℌ�")
 File "<stdin>", line 0

   ^
SyntaxError: 'utf-8' codec can't decode bytes in position 20-21:
invalid continuation byte

That particular misbehaviour is coming from GNU readline, not CPython - because the command history editing wasn’t UTF-8 aware, it corrupted the history buffer and fed such nonsense to stdin that even the surrogateescape error handler was bypassed. While PEP 540’s UTF-8 mode could technically be updated to also reconfigure readline, that’s just one extension module that might be interacting with the standard streams without going through the CPython C API, and any change made by CPython would only apply when readline is running directly as part of Python 3.7 rather than in a separate subprocess.

However, if we actually change the configured locale, GNU readline starts behaving itself, without requiring any changes to the embedding application:

$ LANG=C.UTF-8 python3
Python 3.5.3 (default, Apr 24 2017, 13:32:13)
[GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print("ℙƴ☂ℌøἤ")
ℙƴ☂ℌøἤ
>>> print("ℙƴ☂ℌἤ")
ℙƴ☂ℌἤ
>>> print("ℙƴ☂ἤ")
ℙƴ☂ἤ
>>> print("ℙƴἤ")
ℙƴἤ
>>> print("ℙἤ")
ℙἤ
>>> print("ἤ")
ἤ
>>>
$ LC_CTYPE=C.UTF-8 python3
Python 3.5.3 (default, Apr 24 2017, 13:32:13)
[GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print("ℙƴ☂ℌøἤ")
ℙƴ☂ℌøἤ
>>> print("ℙƴ☂ℌἤ")
ℙƴ☂ℌἤ
>>> print("ℙƴ☂ἤ")
ℙƴ☂ἤ
>>> print("ℙƴἤ")
ℙƴἤ
>>> print("ℙἤ")
ℙἤ
>>> print("ἤ")
ἤ
>>>

Enabling C locale coercion and warnings on Mac OS X, iOS and Android

On Mac OS X, iOS, and Android, CPython already assumes the use of UTF-8 for system interfaces, and we expect most other locale-aware components to do the same.

Accordingly, this PEP originally proposed to disable locale coercion and warnings at build time for these platforms, on the assumption that it would be entirely redundant.

However, that assumption turned out to be incorrect, as subsequent investigations showed that if you explicitly configure LANG=C on these platforms, extension modules like GNU readline will misbehave in much the same way as they do on other *nix systems. [21]

In addition, Mac OS X is also frequently used as a development and testing platform for Python software intended for deployment to other *nix environments (such as Linux or Android), and Linux is similarly often used as a development and testing platform for mobile and Mac OS X applications.

Accordingly, this PEP enables the locale coercion and warning features by default on all platforms that use CPython’s autotools based build toolchain (i.e. everywhere other than Windows).

Backporting to Python 3.6.x

If this PEP is accepted for Python 3.7, redistributors backporting the change specifically to their initial Python 3.6.x release will be both allowed and encouraged. However, such backports should only be undertaken either in conjunction with the changes needed to also provide a suitable locale by default, or else specifically for platforms where such a locale is already consistently available.

At least the Fedora project is planning to pursue this approach for the upcoming Fedora 26 release [19].

Backporting to other 3.x releases

While the proposed behavioural change is seen primarily as a bug fix addressing Python 3’s current misbehaviour in the default ASCII-based C locale, it still represents a reasonably significant change in the way CPython interacts with the C locale system. As such, while some redistributors may still choose to backport it to even earlier Python 3.x releases based on the needs and interests of their particular user base, this wouldn’t be encouraged as a general practice.

However, configuring Python 3 environments (such as base container images) to use these configuration settings by default is both allowed and recommended.