PEP 475 – Retry system calls failing with EINTR
- Retry system calls failing with EINTR
- Charles-François Natali <cf.natali at gmail.com>, Victor Stinner <vstinner at python.org>
- Antoine Pitrou <solipsis at pitrou.net>
- Standards Track
- Backward compatibility
System call wrappers provided in the standard library should be retried
automatically when they fail with
EINTR, to relieve application code
from the burden of doing so.
By system calls, we mean the functions exposed by the standard C library pertaining to I/O or handling of other system resources.
Interrupted system calls
On POSIX systems, signals are common. Code calling system calls must be prepared to handle them. Examples of signals:
- The most common signal is
SIGINT, the signal sent when CTRL+c is pressed. By default, Python raises a
KeyboardInterruptexception when this signal is received.
- When running subprocesses, the
SIGCHLDsignal is sent when a child process exits.
- Resizing the terminal sends the
SIGWINCHsignal to the applications running in the terminal.
- Putting the application in background (ex: press CTRL-z and then
bgcommand) sends the
Writing a C signal handler is difficult: only “async-signal-safe”
functions can be called (for example,
are not async-signal safe), and there are issues with reentrancy.
Therefore, when a signal is received by a process during the execution
of a system call, the system call can fail with the
EINTR error to
give the program an opportunity to handle the signal without the
restriction on signal-safe functions.
This behaviour is system-dependent: on certain systems, using the
SA_RESTART flag, some system calls are retried automatically instead
of failing with
EINTR. Regardless, Python’s
function clears the
SA_RESTART flag when setting the signal handler:
all system calls will probably fail with
EINTR in Python.
Since receiving a signal is a non-exceptional occurrence, robust POSIX code
must be prepared to handle
EINTR (which, in most cases, means
retry in a loop in the hope that the call eventually succeeds).
Without special support from Python, this can make application code
much more verbose than it needs to be.
Status in Python 3.4
In Python 3.4, handling the
InterruptedError exception (
dedicated exception class) is duplicated at every call site on a case-by-case
basis. Only a few Python modules actually handle this exception,
and fixes usually took several years to cover a whole module. Example of
while True: try: data = file.read(size) break except InterruptedError: continue
List of Python modules in the standard library which handle
Other programming languages like Perl, Java and Go retry system calls
EINTR at a lower level, so that libraries and applications
Use Case 1: Don’t Bother With Signals
In most cases, you don’t want to be interrupted by signals and you
don’t expect to get
InterruptedError exceptions. For example, do
you really want to write such complex code for a “Hello World”
while True: try: print("Hello World") break except InterruptedError: continue
InterruptedError can happen in unexpected places. For example,
FileIO.close() may raise
see the article close() and EINTR.
The Python issues related to EINTR section below gives examples of
bugs caused by
The expectation in this use case is that Python hides the
InterruptedError and retries system calls automatically.
Use Case 2: Be notified of signals as soon as possible
Sometimes yet, you expect some signals and you want to handle them as
soon as possible. For example, you may want to immediately quit a
program using the
CTRL+c keyboard shortcut.
Besides, some signals are not interesting and should not disrupt the application. There are two options to interrupt an application on only some signals:
- Set up a custom signal handler which raises an exception, such as
- Use a I/O multiplexing function like
select()together with Python’s signal wakeup file descriptor: see the function
The expectation in this use case is for the Python signal handler to be executed timely, and the system call to fail if the handler raised an exception – otherwise restart.
This PEP proposes to handle EINTR and retries at the lowest level, i.e. in the wrappers provided by the stdlib (as opposed to higher-level libraries and applications).
Specifically, when a system call fails with
EINTR, its Python wrapper
must call the given signal handler (using
If the signal handler raises an exception, the Python wrapper bails out
and fails with the exception.
If the signal handler returns successfully, the Python wrapper retries the system call automatically. If the system call involves a timeout parameter, the timeout is recomputed.
Example of standard library functions that need to be modified to comply with this PEP:
- functions of the
- special cases:
EINTRerror, the syscall is not retried
connect()(except for non-blocking sockets)
selector module already retries on
InterruptedError, but it
doesn’t recompute the timeout yet)
close() methods and
os.dup2() are a special case: they
EINTR instead of retrying. The reason is complex but involves
behaviour under Linux and the fact that the file descriptor may really be
closed even if EINTR is returned. See articles:
- Returning EINTR from close()
- (LKML) Re: [patch 7/7] uml: retry host close() on EINTR
- close() and EINTR
socket.socket.connect() method does not retry
non-blocking sockets if it is interrupted by a signal (fails with
The connection runs asynchronously in background. The caller is responsible
to wait until the socket becomes writable (ex: using
and then call
to check if the connection succeeded (
0) or failed.
Since interrupted system calls are automatically retried, the
InterruptedError exception should not occur anymore when calling those
system calls. Therefore, manual handling of
described in Status in Python 3.4 can be removed, which will simplify
standard library code.
Applications relying on the fact that system calls are interrupted
InterruptedError will hang. The authors of this PEP don’t
think that such applications exist, since they would be exposed to
other issues such as race conditions (there is an opportunity for deadlock
if the signal comes before the system call). Besides, such code would
In any case, those applications must be fixed to handle signals differently, to have a reliable behaviour on all platforms and all Python versions. A possible strategy is to set up a signal handler raising a well-defined exception, or use a wakeup file descriptor.
For applications using event loops,
signal.set_wakeup_fd() is the
recommended option to handle signals. Python’s low-level signal handler
will write signal numbers into the file descriptor and the event loop
will be awaken to read them. The event loop can handle those signals
without the restriction of signal handlers (for example, the loop can
be woken up in any thread, not just the main thread).
Wakeup file descriptor
Since Python 3.3,
signal.set_wakeup_fd() writes the signal number
into the file descriptor, whereas it only wrote a null byte before.
It becomes possible to distinguish between signals using the wakeup file
Linux has a
signalfd() system call which provides more information on
each signal. For example, it’s possible to know the pid and uid who sent
the signal. This function is not exposed in Python yet (see
On Unix, the
asyncio module uses the wakeup file descriptor to
wake up its event loop.
A C signal handler can be called from any thread, but Python signal handlers will always be called in the main Python thread.
Python’s C API provides the
PyErr_SetInterrupt() function which calls
SIGINT signal handler in order to interrupt the main Python thread.
Signals on Windows
Windows uses “control events”:
CTRL_BREAK_EVENT: Break (
CTRL_CLOSE_EVENT: Close event
CTRL_C_EVENT: CTRL+C (
The SetConsoleCtrlHandler() function can be used to install a control handler.
CTRL_BREAK_EVENT events can be sent to a
process using the GenerateConsoleCtrlEvent() function.
This function is exposed in Python as
The following signals are supported on Windows:
CTRL_BREAK_EVENT): signal only available on Windows
The default Python signal handler for
SIGINT sets a Windows event
time.sleep() is implemented with
waits for the
sigint_event object using
as the timeout. So the sleep can be interrupted by
_winapi.WaitForMultipleObjects() automatically adds
sigint_event to the list of watched handles, so it can also be
PyOS_StdioReadline() also used
failed to check if Ctrl-C or Ctrl-Z was pressed.
The implementation is tracked in issue 23285. It was committed on February 07, 2015.
This document has been placed in the public domain.
Last modified: 2021-09-17 18:18:24 GMT