Rejection Notice

From https://mail.python.org/pipermail/python-dev/2014-March/133118.html:

“”” I want to reject this PEP. I think the proposed syntax is acceptable given the desired semantics, although it’s still a bit jarring. It’s probably no worse than the colon used with lambda (which echoes the colon used in a def just like the colon here echoes the one in a try/except) and definitely better than the alternatives listed.

But the thing I can’t get behind are the motivation and rationale. I don’t think that e.g. dict.get() would be unnecessary once we have except expressions, and I disagree with the position that EAFP is better than LBYL, or “generally recommended” by Python. (Where do you get that? From the same sources that are so obsessed with DRY they’d rather introduce a higher-order-function than repeat one line of code? :-)

This is probably the most you can get out of me as far as a pronouncement. Given that the language summit is coming up I’d be happy to dive deeper in my reasons for rejecting it there (if there’s demand).

I do think that (apart from never explaining those dreadful acronyms :-) this was a well-written and well-researched PEP, and I think you’ve done a great job moderating the discussion, collecting objections, reviewing alternatives, and everything else that is required to turn a heated debate into a PEP. Well done Chris (and everyone who helped), and good luck with your next PEP! “””

Abstract

Just as PEP 308 introduced a means of value-based conditions in an expression, this system allows exception-based conditions to be used as part of an expression.

Motivation

A number of functions and methods have parameters which will cause them to return a specified value instead of raising an exception. The current system is ad-hoc and inconsistent, and requires that each function be individually written to have this functionality; not all support this.

• dict.get(key, default) - second positional argument in place of KeyError
• next(iter, default) - second positional argument in place of StopIteration
• list.pop() - no way to return a default
• seq[index] - no way to handle a bounds error
• min(sequence, default=default) - keyword argument in place of ValueError
• statistics.mean(data) - no way to handle an empty iterator

Had this facility existed early in Python’s history, there would have been no need to create dict.get() and related methods; the one obvious way to handle an absent key would be to respond to the exception. One method is written which signals the absence in one way, and one consistent technique is used to respond to the absence. Instead, we have dict.get(), and as of Python 3.4, we also have min(… default=default), and myriad others. We have a LBYL syntax for testing inside an expression, but there is currently no EAFP notation; compare the following:

# LBYL:
if key in dic:
    process(dic[key])
else:
    process(None)
# As an expression:
process(dic[key] if key in dic else None)

# EAFP:
try:
    process(dic[key])
except KeyError:
    process(None)
# As an expression:
process(dic[key] except KeyError: None)

Python generally recommends the EAFP policy, but must then proliferate utility functions like dic.get(key,None) to enable this.

Rationale

The current system requires that a function author predict the need for a default, and implement support for it. If this is not done, a full try/except block is needed.

Since try/except is a statement, it is impossible to catch exceptions in the middle of an expression. Just as if/else does for conditionals and lambda does for function definitions, so does this allow exception catching in an expression context.

This provides a clean and consistent way for a function to provide a default: it simply raises an appropriate exception, and the caller catches it.

With some situations, an LBYL technique can be used (checking if some sequence has enough length before indexing into it, for instance). This is not safe in all cases, but as it is often convenient, programmers will be tempted to sacrifice the safety of EAFP in favour of the notational brevity of LBYL. Additionally, some LBYL techniques (eg involving getattr with three arguments) warp the code into looking like literal strings rather than attribute lookup, which can impact readability. A convenient EAFP notation solves all of this.

There’s no convenient way to write a helper function to do this; the nearest is something ugly using either lambda:

def except_(expression, exception_list, default):
    try:
        return expression()
    except exception_list:
        return default()
value = except_(lambda: 1/x, ZeroDivisionError, lambda: float("nan"))

which is clunky, and unable to handle multiple exception clauses; or eval:

def except_(expression, exception_list, default):
    try:
        return eval(expression, globals_of_caller(), locals_of_caller())
    except exception_list as exc:
        l = locals_of_caller().copy()
        l['exc'] = exc
        return eval(default, globals_of_caller(), l)

def globals_of_caller():
    return sys._getframe(2).f_globals

def locals_of_caller():
    return sys._getframe(2).f_locals

value = except_("""1/x""",ZeroDivisionError,""" "Can't divide by zero" """)

which is even clunkier, and relies on implementation-dependent hacks. (Writing globals_of_caller() and locals_of_caller() for interpreters other than CPython is left as an exercise for the reader.)

Raymond Hettinger expresses a desire for such a consistent API. Something similar has been requested multiple times in the past.

Proposal

Just as the ‘or’ operator and the three part ‘if-else’ expression give short circuiting methods of catching a falsy value and replacing it, this syntax gives a short-circuiting method of catching an exception and replacing it.

This currently works:

lst = [1, 2, None, 3]
value = lst[2] or "No value"

The proposal adds this:

lst = [1, 2]
value = (lst[2] except IndexError: "No value")

Specifically, the syntax proposed is:

(expr except exception_list: default)

where expr, exception_list, and default are all expressions. First, expr is evaluated. If no exception is raised, its value is the value of the overall expression. If any exception is raised, exception_list is evaluated, and should result in either a type or a tuple, just as with the statement form of try/except. Any matching exception will result in the corresponding default expression being evaluated and becoming the value of the expression. As with the statement form of try/except, non-matching exceptions will propagate upward.

Parentheses are required around the entire expression, unless they would be completely redundant, according to the same rules as generator expressions follow. This guarantees correct interpretation of nested except-expressions, and allows for future expansion of the syntax - see below on multiple except clauses.

Note that the current proposal does not allow the exception object to be captured. Where this is needed, the statement form must be used. (See below for discussion and elaboration on this.)

This ternary operator would be between lambda and if/else in precedence.

Consider this example of a two-level cache:

for key in sequence:
    x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: f(key)))
    # do something with x

This cannot be rewritten as:

x = lvl1.get(key, lvl2.get(key, f(key)))

which, despite being shorter, defeats the purpose of the cache, as it must calculate a default value to pass to get(). The .get() version calculates backwards; the exception-testing version calculates forwards, as would be expected. The nearest useful equivalent would be:

x = lvl1.get(key) or lvl2.get(key) or f(key)

which depends on the values being nonzero, as well as depending on the cache object supporting this functionality.

Alternative Proposals

Discussion on python-ideas brought up the following syntax suggestions:

value = expr except default if Exception [as e]
value = expr except default for Exception [as e]
value = expr except default from Exception [as e]
value = expr except Exception [as e] return default
value = expr except (Exception [as e]: default)
value = expr except Exception [as e] try default
value = expr except Exception [as e] continue with default
value = default except Exception [as e] else expr
value = try expr except Exception [as e]: default
value = expr except default # Catches anything
value = expr except(Exception) default # Catches only the named type(s)
value = default if expr raise Exception
value = expr or else default if Exception
value = expr except Exception [as e] -> default
value = expr except Exception [as e] pass default

It has also been suggested that a new keyword be created, rather than reusing an existing one. Such proposals fall into the same structure as the last form, but with a different keyword in place of ‘pass’. Suggestions include ‘then’, ‘when’, and ‘use’. Also, in the context of the “default if expr raise Exception” proposal, it was suggested that a new keyword “raises” be used.

All forms involving the ‘as’ capturing clause have been deferred from this proposal in the interests of simplicity, but are preserved in the table above as an accurate record of suggestions.

The four forms most supported by this proposal are, in order:

value = (expr except Exception: default)
value = (expr except Exception -> default)
value = (expr except Exception pass default)
value = (expr except Exception then default)

All four maintain left-to-right evaluation order: first the base expression, then the exception list, and lastly the default. This is important, as the expressions are evaluated lazily. By comparison, several of the ad-hoc alternatives listed above must (by the nature of functions) evaluate their default values eagerly. The preferred form, using the colon, parallels try/except by using “except exception_list:”, and parallels lambda by having “keyword name_list: subexpression”; it also can be read as mapping Exception to the default value, dict-style. Using the arrow introduces a token many programmers will not be familiar with, and which currently has no similar meaning, but is otherwise quite readable. The English word “pass” has a vaguely similar meaning (consider the common usage “pass by value/reference” for function arguments), and “pass” is already a keyword, but as its meaning is distinctly unrelated, this may cause confusion. Using “then” makes sense in English, but this introduces a new keyword to the language - albeit one not in common use, but a new keyword all the same.

Left to right evaluation order is extremely important to readability, as it parallels the order most expressions are evaluated. Alternatives such as:

value = (expr except default if Exception)

break this, by first evaluating the two ends, and then coming to the middle; while this may not seem terrible (as the exception list will usually be a constant), it does add to the confusion when multiple clauses meet, either with multiple except/if or with the existing if/else, or a combination. Using the preferred order, subexpressions will always be evaluated from left to right, no matter how the syntax is nested.

Keeping the existing notation, but shifting the mandatory parentheses, we have the following suggestion:

value = expr except (Exception: default)
value = expr except(Exception: default)

This is reminiscent of a function call, or a dict initializer. The colon cannot be confused with introducing a suite, but on the other hand, the new syntax guarantees lazy evaluation, which a dict does not. The potential to reduce confusion is considered unjustified by the corresponding potential to increase it.

Example usage

For each example, an approximately-equivalent statement form is given, to show how the expression will be parsed. These are not always strictly equivalent, but will accomplish the same purpose. It is NOT safe for the interpreter to translate one into the other.

A number of these examples are taken directly from the Python standard library, with file names and line numbers correct as of early Feb 2014. Many of these patterns are extremely common.

Retrieve an argument, defaulting to None:

cond = (args[1] except IndexError: None)

# Lib/pdb.py:803:
try:
    cond = args[1]
except IndexError:
    cond = None

Fetch information from the system if available:

pwd = (os.getcwd() except OSError: None)

# Lib/tkinter/filedialog.py:210:
try:
    pwd = os.getcwd()
except OSError:
    pwd = None

Attempt a translation, falling back on the original:

e.widget = (self._nametowidget(W) except KeyError: W)

# Lib/tkinter/__init__.py:1222:
try:
    e.widget = self._nametowidget(W)
except KeyError:
    e.widget = W

Read from an iterator, continuing with blank lines once it’s exhausted:

line = (readline() except StopIteration: '')

# Lib/lib2to3/pgen2/tokenize.py:370:
try:
    line = readline()
except StopIteration:
    line = ''

Retrieve platform-specific information (note the DRY improvement); this particular example could be taken further, turning a series of separate assignments into a single large dict initialization:

# sys.abiflags may not be defined on all platforms.
_CONFIG_VARS['abiflags'] = (sys.abiflags except AttributeError: '')

# Lib/sysconfig.py:529:
try:
    _CONFIG_VARS['abiflags'] = sys.abiflags
except AttributeError:
    # sys.abiflags may not be defined on all platforms.
    _CONFIG_VARS['abiflags'] = ''

Retrieve an indexed item, defaulting to None (similar to dict.get):

def getNamedItem(self, name):
    return (self._attrs[name] except KeyError: None)

# Lib/xml/dom/minidom.py:573:
def getNamedItem(self, name):
    try:
        return self._attrs[name]
    except KeyError:
        return None

Translate numbers to names, falling back on the numbers:

g = (grp.getgrnam(tarinfo.gname)[2] except KeyError: tarinfo.gid)
u = (pwd.getpwnam(tarinfo.uname)[2] except KeyError: tarinfo.uid)

# Lib/tarfile.py:2198:
try:
    g = grp.getgrnam(tarinfo.gname)[2]
except KeyError:
    g = tarinfo.gid
try:
    u = pwd.getpwnam(tarinfo.uname)[2]
except KeyError:
    u = tarinfo.uid

Look up an attribute, falling back on a default:

mode = (f.mode except AttributeError: 'rb')

# Lib/aifc.py:882:
if hasattr(f, 'mode'):
    mode = f.mode
else:
    mode = 'rb'

return (sys._getframe(1) except AttributeError: None)
# Lib/inspect.py:1350:
return sys._getframe(1) if hasattr(sys, "_getframe") else None

Perform some lengthy calculations in EAFP mode, handling division by zero as a sort of sticky NaN:

value = (calculate(x) except ZeroDivisionError: float("nan"))

try:
    value = calculate(x)
except ZeroDivisionError:
    value = float("nan")

Calculate the mean of a series of numbers, falling back on zero:

value = (statistics.mean(lst) except statistics.StatisticsError: 0)

try:
    value = statistics.mean(lst)
except statistics.StatisticsError:
    value = 0

Looking up objects in a sparse list of overrides:

(overrides[x] or default except IndexError: default).ping()

try:
    (overrides[x] or default).ping()
except IndexError:
    default.ping()

try:
    ips.append(ip.ip)
except AttributeError:
    ips.append(ip.network_address)

ips.append(ip.ip except AttributeError: ip.network_address)

try:
    _ = ip.ip
except AttributeError:
    _ = ip.network_address
ips.append(_)

try:
    dirlist.append(_os.getcwd())
except (AttributeError, OSError):
    dirlist.append(_os.curdir)

dirlist.append(_os.getcwd() except (AttributeError, OSError): _os.curdir)

try:
    status.append('%s:%d' % self.addr)
except TypeError:
    status.append(repr(self.addr))

status.append('%s:%d' % self.addr except TypeError: repr(self.addr))

Comparisons with other languages

(With thanks to Andrew Barnert for compiling this section. Note that the examples given here do not reflect the current version of the proposal, and need to be edited.)

Ruby’s “begin…rescue…rescue…else…ensure…end” is an expression (potentially with statements inside it). It has the equivalent of an “as” clause, and the equivalent of bare except. And it uses no punctuation or keyword between the bare except/exception class/exception class with as clause and the value. (And yes, it’s ambiguous unless you understand Ruby’s statement/expression rules.)

x = begin computation() rescue MyException => e default(e) end;
x = begin computation() rescue MyException default() end;
x = begin computation() rescue default() end;
x = begin computation() rescue MyException default() rescue OtherException other() end;

In terms of this PEP:

x = computation() except MyException as e default(e)
x = computation() except MyException default(e)
x = computation() except default(e)
x = computation() except MyException default() except OtherException other()

Erlang has a try expression that looks like this

x = try computation() catch MyException:e -> default(e) end;
x = try computation() catch MyException:e -> default(e); OtherException:e -> other(e) end;

The class and “as” name are mandatory, but you can use “_” for either. There’s also an optional “when” guard on each, and a “throw” clause that you can catch, which I won’t get into. To handle multiple exceptions, you just separate the clauses with semicolons, which I guess would map to commas in Python. So:

x = try computation() except MyException as e -> default(e)
x = try computation() except MyException as e -> default(e), OtherException as e->other_default(e)

Erlang also has a “catch” expression, which, despite using the same keyword, is completely different, and you don’t want to know about it.

The ML family has two different ways of dealing with this, “handle” and “try”; the difference between the two is that “try” pattern-matches the exception, which gives you the effect of multiple except clauses and as clauses. In either form, the handler clause is punctuated by “=>” in some dialects, “->” in others.

To avoid confusion, I’ll write the function calls in Python style.

Here’s SML’s “handle”

let x = computation() handle MyException => default();;

Here’s OCaml’s “try”

let x = try computation() with MyException explanation -> default(explanation);;

let x = try computation() with

    MyException(e) -> default(e)
  | MyOtherException() -> other_default()
  | (e) -> fallback(e);;

In terms of this PEP, these would be something like:

x = computation() except MyException => default()
x = try computation() except MyException e -> default()
x = (try computation()
     except MyException as e -> default(e)
     except MyOtherException -> other_default()
     except BaseException as e -> fallback(e))

Many ML-inspired but not-directly-related languages from academia mix things up, usually using more keywords and fewer symbols. So, the Oz would map to Python as

x = try computation() catch MyException as e then default(e)

Many Lisp-derived languages, like Clojure, implement try/catch as special forms (if you don’t know what that means, think function-like macros), so you write, effectively

try(computation(), catch(MyException, explanation, default(explanation)))

try(computation(),
    catch(MyException, explanation, default(explanation)),
    catch(MyOtherException, explanation, other_default(explanation)))

In Common Lisp, this is done with a slightly clunkier “handler-case” macro, but the basic idea is the same.

The Lisp style is, surprisingly, used by some languages that don’t have macros, like Lua, where xpcall takes functions. Writing lambdas Python-style instead of Lua-style

x = xpcall(lambda: expression(), lambda e: default(e))

This actually returns (true, expression()) or (false, default(e)), but I think we can ignore that part.

Haskell is actually similar to Lua here (except that it’s all done with monads, of course):

x = do catch(lambda: expression(), lambda e: default(e))

You can write a pattern matching expression within the function to decide what to do with it; catching and re-raising exceptions you don’t want is cheap enough to be idiomatic.

But Haskell infixing makes this nicer:

x = do expression() `catch` lambda: default()
x = do expression() `catch` lambda e: default(e)

And that makes the parallel between the lambda colon and the except colon in the proposal much more obvious:

x = expression() except Exception: default()
x = expression() except Exception as e: default(e)

Tcl has the other half of Lua’s xpcall; catch is a function which returns true if an exception was caught, false otherwise, and you get the value out in other ways. And it’s all built around the implicit quote-and-exec that everything in Tcl is based on, making it even harder to describe in Python terms than Lisp macros, but something like

if {[ catch("computation()") "explanation"]} { default(explanation) }

Smalltalk is also somewhat hard to map to Python. The basic version would be

x := computation() on:MyException do:default()

… but that’s basically Smalltalk’s passing-arguments-with-colons syntax, not its exception-handling syntax.

Deferred sub-proposals

# Will catch any of the listed exceptions thrown by expr;
# any exception thrown by a default expression will propagate.
value = (expr
    except Exception1: default1
    except Exception2: default2
    # ... except ExceptionN: defaultN
)

# Will catch an Exception2 thrown by either expr or default1
value = (
    (expr except Exception1: default1)
    except Exception2: default2
)
# Will catch an Exception2 thrown by default1 only
value = (expr except Exception1:
    (default1 except Exception2: default2)
)

>>> (expr except Exception as e: e)

try: typ, dat = self._simple_command('LOGOUT')
except: typ, dat = 'NO', ['%s: %s' % sys.exc_info()[:2]]

typ, dat = (self._simple_command('LOGOUT')
    except BaseException as e: ('NO', '%s: %s' % (type(e), e)))

Rejected sub-proposals

value = expr (
    except Exception1 [as e]: default1
    except Exception2 [as e]: default2
    # ... except ExceptionN [as e]: defaultN
)

value = expr except ExceptionType: default
value = expr (except ExceptionType: default)

value = (expr) except ExceptionType: (default)

statement except Exception: pass

try:
    statement
except Exception:
    pass

os.unlink(some_file) except OSError: pass

from contextlib import suppress
with suppress(OSError): os.unlink(some_file)

Common objections

Copyright

This document has been placed in the public domain.