Parent

The cc_parent field (accessed for example by a __parent__ or __objclass__ descriptor from Python code) can be any Python object, or NULL. Custom classes are free to set cc_parent to whatever they want. It is only used by the C call protocol if the CCALL_OBJCLASS flag is set.

For methods of extension types, cc_parent points to the class that defines the method (which may be a superclass of type(self)). This is currently non-trivial to retrieve from a method’s code. In the future, this can be used to access the module state via the defining class. See the rationale of PEP 573 for details.

When the flag CCALL_OBJCLASS is set (as it will be for methods of extension types), cc_parent is used for type checks like the following:

>>> list.append({}, "x")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: descriptor 'append' requires a 'list' object but received a 'dict'

For functions of modules, cc_parent is set to the module. Currently, this is exactly the same as __self__. However, using __self__ for the module is a quirk of the current implementation: in the future, we want to allow functions which use __self__ in the normal way, for implementing methods. Such functions can still use cc_parent instead to refer to the module.

The parent would also typically be used to implement __qualname__. The new C API function PyCCall_GenericGetQualname() does exactly that.

Using tp_print

We propose to replace the existing unused field tp_print by tp_ccalloffset. Since Py_TPFLAGS_HAVE_CCALL would not be added to Py_TPFLAGS_DEFAULT, this ensures full backwards compatibility for existing extension modules setting tp_print. It also means that we can require that tp_ccalloffset is a valid offset when Py_TPFLAGS_HAVE_CCALL is specified: we do not need to check tp_ccalloffset != 0. In future Python versions, we may decide that tp_print becomes tp_ccalloffset unconditionally, drop the Py_TPFLAGS_HAVE_CCALL flag and instead check for tp_ccalloffset != 0.

NOTE: the exact layout of PyTypeObject is not part of the stable ABI). Therefore, changing the tp_print field from a printfunc (a function pointer) to a Py_ssize_t should not be a problem, even if this changes the memory layout of the PyTypeObject structure. Moreover, on all systems for which binaries are commonly built (Windows, Linux, macOS), the size of printfunc and Py_ssize_t are the same, so the issue of binary compatibility will not come up anyway.

Checking __objclass__

If the CCALL_OBJCLASS flag is set and if cr_self is NULL (this is the case for unbound methods of extension types), then a type check is done: the function must be called with at least one positional argument and the first (typically called self) must be an instance of cc_parent (which must be a class). If not, a TypeError is raised.

Self slicing

If cr_self is not NULL or if the flag CCALL_SELFARG is not set in cc_flags, then the argument passed as self is simply cr_self.

If cr_self is NULL and the flag CCALL_SELFARG is set, then the first positional argument is removed from args and instead passed as self argument to the C function. Effectively, the first positional argument is treated as __self__. If there are no positional arguments, TypeError is raised.

This process is called “self slicing” and a function is said to have self slicing if cr_self is NULL and CCALL_SELFARG is set.

Note that a CCALL_NOARGS function with self slicing effectively has one argument, namely self. Analogously, a CCALL_O function with self slicing has two arguments.

Descriptor behavior

Classes supporting the C call protocol must implement the descriptor protocol in a specific way.

This is required for an efficient implementation of bound methods: if other code can make assumptions on what __get__ does, it enables optimizations which would not be possible otherwise. In particular, we want to allow sharing the PyCCallDef structure between bound and unbound methods. We also need a correct implementation of _PyObject_GetMethod which is used by the LOAD_METHOD/CALL_METHOD optimization.

First of all, if func supports the C call protocol, then func.__set__ and func.__delete__ must not be implemented.

Second, func.__get__ must behave as follows:

• If cr_self is not NULL, then __get__ must be a no-op in the sense that func.__get__(obj, cls)(*args, **kwds) behaves exactly the same as func(*args, **kwds). It is also allowed for __get__ to be not implemented at all.
• If cr_self is NULL, then func.__get__(obj, cls)(*args, **kwds) (with obj not None) must be equivalent to func(obj, *args, **kwds). In particular, __get__ must be implemented in this case. This is unrelated to self slicing: obj may be passed as self argument to the C function or it may be the first positional argument.
• If cr_self is NULL, then func.__get__(None, cls)(*args, **kwds) must be equivalent to func(*args, **kwds).

There are no restrictions on the object func.__get__(obj, cls). The latter is not required to implement the C call protocol for example. We only specify what func.__get__(obj, cls).__call__ does.

For classes that do not care about __self__ and __get__ at all, the easiest solution is to assign cr_self = Py_None (or any other non-NULL value).

The __name__ attribute

The C call protocol requires that the function has a __name__ attribute which is of type str (not a subclass).

Furthermore, the object returned by __name__ must be stored somewhere; it cannot be a temporary object. This is required because PyEval_GetFuncName uses a borrowed reference to the __name__ attribute (see also [2]).

Generic API functions

This section lists the new public API functions or macros dealing with the C call protocol.

• int PyCCall_Check(PyObject *op): return true if op implements the C call protocol.

All the functions and macros below apply to any instance supporting the C call protocol. In other words, PyCCall_Check(func) must be true.

• PyObject *PyCCall_Call(PyObject *func, PyObject *args, PyObject *kwds): call func with positional arguments args and keyword arguments kwds (kwds may be NULL). This function is meant to be put in the tp_call slot.
• PyObject *PyCCall_FastCall(PyObject *func, PyObject *const *args, Py_ssize_t nargs, PyObject *kwds): call func with nargs positional arguments given by args[0], …, args[nargs-1]. The parameter kwds can be NULL (no keyword arguments), a dict with name:value items or a tuple with keyword names. In the latter case, the keyword values are stored in the args array, starting at args[nargs].

Macros to access the PyCCallRoot and PyCCallDef structures:

• const PyCCallRoot *PyCCall_CCALLROOT(PyObject *func): pointer to the PyCCallRoot structure inside func.
• const PyCCallDef *PyCCall_CCALLDEF(PyObject *func): shorthand for PyCCall_CCALLROOT(func)->cr_ccall.
• uint32_t PyCCall_FLAGS(PyObject *func): shorthand for PyCCall_CCALLROOT(func)->cr_ccall->cc_flags.
• PyObject *PyCCall_SELF(PyOject *func): shorthand for PyCCall_CCALLROOT(func)->cr_self.

Generic getters, meant to be put into the tp_getset array:

• PyObject *PyCCall_GenericGetParent(PyObject *func, void *closure): return cc_parent. Raise AttributeError if cc_parent is NULL.
• PyObject *PyCCall_GenericGetQualname(PyObject *func, void *closure): return a string suitable for using as __qualname__. This uses the __qualname__ of cc_parent if possible. It also uses the __name__ attribute.

Profiling

The profiling events c_call, c_return and c_exception are only generated when calling actual instances of builtin_function_or_method or method_descriptor. This is done for simplicity and also for backwards compatibility (such that the profile function does not receive objects that it does not recognize). In a future PEP, we may extend C-level profiling to arbitrary classes implementing the C call protocol.

C API functions

The following function is added (also to the stable ABI):

• PyObject * PyCFunction_ClsNew(PyTypeObject *cls, PyMethodDef *ml, PyObject *self, PyObject *module, PyObject *parent): create a new object with object structure PyCFunctionObject and class cls. The entries of the PyMethodDef structure are used to construct the new object, but the pointer to the PyMethodDef structure is not stored. The flags for the C call protocol are automatically determined in terms of ml->ml_flags, self and parent.

The existing functions PyCFunction_New, PyCFunction_NewEx and PyDescr_NewMethod are implemented in terms of PyCFunction_ClsNew.

The undocumented functions PyCFunction_GetFlags and PyCFunction_GET_FLAGS are deprecated. They are still artificially supported by storing the original METH_... flags in a bitfield inside cc_flags. Despite the fact that PyCFunction_GetFlags is technically part of the stable ABI, it is highly unlikely to be used that way: first of all, it is not even documented. Second, the flag METH_FASTCALL is not part of the stable ABI but it is very common (because of Argument Clinic). So, if one cannot support METH_FASTCALL, it is hard to imagine a use case for PyCFunction_GetFlags. The fact that PyCFunction_GET_FLAGS and PyCFunction_GetFlags are not used at all by CPython outside of Objects/call.c further shows that these functions are not particularly useful.

Why is this better than PEP 575?

One of the major complaints of PEP 575 was that is was coupling functionality (the calling and introspection protocol) with the class hierarchy: a class could only benefit from the new features if it was a subclass of base_function. It may be difficult for existing classes to do that because they may have other constraints on the layout of the C object structure, coming from an existing base class or implementation details. For example, functools.lru_cache cannot implement PEP 575 as-is.

It also complicated the implementation precisely because changes were needed both in the implementation details and in the class hierarchy.

The current PEP does not have these problems.

Why store the function pointer in the instance?

The actual information needed for calling an object is stored in the instance (in the PyCCallDef structure) instead of the class. This is different from the tp_call slot or earlier attempts at implementing a tp_fastcall slot [1].

The main use case is built-in functions and methods. For those, the C function to be called does depend on the instance.

Note that the current protocol makes it easy to support the case where the same C function is called for all instances: just use a single static PyCCallDef structure for every instance.

Why CCALL_OBJCLASS?

The flag CCALL_OBJCLASS is meant to support various cases where the class of a self argument must be checked, such as:

>>> list.append({}, None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: append() requires a 'list' object but received a 'dict'

>>> list.__len__({})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: descriptor '__len__' requires a 'list' object but received a 'dict'

>>> float.__dict__["fromhex"](list, "0xff")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: descriptor 'fromhex' for type 'float' doesn't apply to type 'list'

In the reference implementation, only the first of these uses the new code. The other examples show that these kind of checks appear in multiple places, so it makes sense to add generic support for them.

Why CCALL_SELFARG?

The flag CCALL_SELFARG and the concept of self slicing are needed to support methods: the C function should not care whether it is called as unbound method or as bound method. In both cases, there should be a self argument and this is simply the first positional argument of an unbound method call.

For example, list.append is a METH_O method. Both the calls list.append([], 42) and [].append(42) should translate to the C call list_append([], 42).

Thanks to the proposed C call protocol, we can support this in such a way that both the unbound and the bound method share a PyCCallDef structure (with the CCALL_SELFARG flag set).

So, CCALL_SELFARG has two advantages: there is no extra layer of indirection for calling methods and constructing bound methods does not require setting up a PyCCallDef structure.

Another minor advantage is that we could make the error messages for a wrong call signature more uniform between Python methods and built-in methods. In the following example, Python is undecided whether a method takes 1 or 2 arguments:

>>> class List(list):
...     def myappend(self, item):
...         self.append(item)
>>> List().myappend(1, 2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: myappend() takes 2 positional arguments but 3 were given
>>> List().append(1, 2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: append() takes exactly one argument (2 given)

It is currently impossible for PyCFunction_Call to know the actual number of user-visible arguments since it cannot distinguish at runtime between a function (without self argument) and a bound method (with self argument). The CCALL_SELFARG flag makes this difference explicit.

Why CCALL_DEFARG?

The flag CCALL_DEFARG gives the callee access to the PyCCallDef *. There are various use cases for this:

1. The callee can use the cc_parent field, which is useful for PEP 573.
2. Applications are free to extend the PyCCallDef structure with user-defined fields, which can then be accessed analogously.
3. In the case where the PyCCallDef structure is part of the object structure (this is true for example for PyCFunctionObject), an appropriate offset can be subtracted from the PyCCallDef pointer to get a pointer to the callable object defining that PyCCallDef.

An earlier version of this PEP defined a flag CCALL_FUNCARG instead of CCALL_DEFARG which would pass the callable object to the callee. This had similar use cases, but there was some ambiguity for bound methods: should the “callable object” be the bound method object or the original function wrapped by the method? By passing the PyCCallDef * instead, this ambiguity is gone since the bound method uses the PyCCallDef * from the wrapped function.

Replacing tp_print

We repurpose tp_print as tp_ccalloffset because this makes it easier for external projects to backport the C call protocol to earlier Python versions. In particular, the Cython project has shown interest in doing that (see https://mail.python.org/pipermail/python-dev/2018-June/153927.html).