Deprecated since Python 3.3

Py_UNICODE and APIs using it has been deprecated since Python 3.3.

Inefficient

All of these APIs are implemented using PyUnicode_FromWideChar. So these APIs are inefficient when user want to encode Unicode object.

Not used widely

When searching from the top 4000 PyPI packages [1], only pyodbc use these APIs.

• PyUnicode_EncodeUTF8()
• PyUnicode_EncodeUTF16()

pyodbc uses these APIs to encode Unicode object into bytes object. So it is easy to fix it. [2]

Replace Py_UNICODE* with PyObject*

As described in the “Alternative APIs” section, some APIs don’t have public alternative APIs accepting PyObject *unicode input. And some public alternative APIs have restrictions like missing errors and byteorder parameters.

Instead of removing deprecated APIs, we can reuse their names for alternative public APIs.

Since we have private alternative APIs already, it is just renaming from private name to public and deprecated names.

Pros:

• We have a more consistent API set.

Cons:

• Backward incompatible.
• We have more public APIs to maintain for rare use cases.
• Existing public APIs are enough for most use cases, and PyUnicode_AsEncodedString() can be used in other cases.

Replace Py_UNICODE* with Py_UCS4*

We can replace Py_UNICODE with Py_UCS4 and undeprecate these APIs.

UTF-8, UTF-16, UTF-32 encoders support Py_UCS4 internally. So PyUnicode_EncodeUTF8(), PyUnicode_EncodeUTF16(), and PyUnicode_EncodeUTF32() can avoid to create a temporary Unicode object.

Pros:

• We can avoid creating temporary Unicode object when encoding from Py_UCS4* into bytes object with UTF-8, UTF-16, UTF-32 codecs.

Cons:

• Backward incompatible.
• We have more public APIs to maintain for rare use cases.
• Other Python implementations that want to support Python/C API need to support these APIs too.
• If we change the Unicode internal representation to UTF-8 in the future, we need to keep UCS-4 support only for these APIs.

Replace Py_UNICODE* with wchar_t*

We can replace Py_UNICODE with wchar_t. Since Py_UNICODE is typedef of wchar_t already, this is status quo.

On platforms where sizeof(wchar_t) == 4, we can avoid to create a temporary Unicode object when encoding from wchar_t* to bytes objects using UTF-8, UTF-16, and UTF-32 codec, like the “Replace Py_UNICODE* with Py_UCS4*” idea.

Pros:

• Backward compatible.
• We can avoid creating temporary Unicode object when encode from Py_UCS4* into bytes object with UTF-8, UTF-16, UTF-32 codecs on platform where sizeof(wchar_t) == 4.

Cons:

• Although Windows is the most major platform that uses wchar_t heavily, these APIs need to create a temporary Unicode object always because sizeof(wchar_t) == 2 on Windows.
• We have more public APIs to maintain for rare use cases.
• Other Python implementations that want to support Python/C API need to support these APIs too.
• If we change the Unicode internal representation to UTF-8 in the future, we need to keep UCS-4 support only for these APIs.

Emit runtime warning

In addition to existing compiler warning, emitting runtime DeprecationWarning is suggested.

But these APIs doesn’t release GIL for now. Emitting a warning from such APIs is not safe. See this example.

PyObject *u = PyList_GET_ITEM(list, i);  // u is borrowed reference.
PyObject *b = PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(u),
        PyUnicode_GET_SIZE(u), NULL);
// Assumes u is still living reference.
PyObject *t = PyTuple_Pack(2, u, b);
Py_DECREF(b);
return t;

If we emit Python warning from PyUnicode_EncodeUTF8(), warning filters and other threads may change the list and u can be a dangling reference after PyUnicode_EncodeUTF8() returned.

Objections

• Removing these APIs removes ability to use codec without temporary Unicode.
  • Codecs can not encode Unicode buffer directly without temporary Unicode object since Python 3.3. All these APIs creates temporary Unicode object for now. So removing them doesn’t reduce any abilities.
• Why not remove decoder APIs too?
  • They are part of stable ABI.
  • PyUnicode_DecodeASCII() and PyUnicode_DecodeUTF8() are used very widely. Deprecating them is not worth enough.
  • Decoder APIs can decode from byte buffer directly, without creating temporary bytes object. On the other hand, encoder APIs can not avoid temporary Unicode object.