Abstract

This PEP proposes the addition of an optional given clause to several Python statements that do not currently have an associated code suite. This clause will create a statement local namespace for additional names that are accessible in the associated statement, but do not become part of the containing namespace.

Adoption of a new symbol, ?, is proposed to denote a forward reference to the namespace created by running the associated code suite. It will be a reference to a types.SimpleNamespace object.

The primary motivation is to enable a more declarative style of programming, where the operation to be performed is presented to the reader first, and the details of the necessary subcalculations are presented in the following indented suite. As a key example, this would elevate ordinary assignment statements to be on par with class and def statements where the name of the item to be defined is presented to the reader in advance of the details of how the value of that item is calculated. It also allows named functions to be used in a “multi-line lambda” fashion, where the name is used solely as a placeholder in the current expression and then defined in the following suite.

A secondary motivation is to simplify interim calculations in module and class level code without polluting the resulting namespaces.

The intent is that the relationship between a given clause and a separate function definition that performs the specified operation will be similar to the existing relationship between an explicit while loop and a generator that produces the same sequence of operations as that while loop.

The specific proposal in this PEP has been informed by various explorations of this and related concepts over the years (e.g. [1], [2], [3], [6], [8]), and is inspired to some degree by the where and let clauses in Haskell. It avoids some problems that have been identified in past proposals, but has not yet itself been subject to the test of implementation.

Proposal

This PEP proposes the addition of an optional given clause to the syntax for simple statements which may contain an expression, or may substitute for such a statement for purely syntactic purposes. The current list of simple statements that would be affected by this addition is as follows:

• expression statement
• assignment statement
• augmented assignment statement
• del statement
• return statement
• yield statement
• raise statement
• assert statement
• pass statement

The given clause would allow subexpressions to be referenced by name in the header line, with the actual definitions following in the indented clause. As a simple example:

sorted_data = sorted(data, key=?.sort_key) given:
    def sort_key(item):
        return item.attr1, item.attr2

The new symbol ? is used to refer to the given namespace. It would be a types.SimpleNamespace instance, so ?.sort_key functions as a forward reference to a name defined in the given clause.

A docstring would be permitted in the given clause, and would be attached to the result namespace as its __doc__ attribute.

The pass statement is included to provide a consistent way to skip inclusion of a meaningful expression in the header line. While this is not an intended use case, it isn’t one that can be prevented as multiple alternatives (such as ... and ()) remain available even if pass itself is disallowed.

The body of the given clause will execute in a new scope, using normal function closure semantics. To support early binding of loop variables and global references, as well as to allow access to other names defined at class scope, the given clause will also allow explicit binding operations in the header line:

# Explicit early binding via given clause
seq = []
for i in range(10):
    seq.append(?.f) given i=i in:
        def f():
            return i
assert [f() for f in seq] == list(range(10))

op(?.f, ?.g) given bound_a=a, bound_b=b in:
    def f():
        return bound_a + bound_b
    def g():
        return bound_a - bound_b

__arg1 = a
__arg2 = b
def __scope(bound_a, bound_b):
    def f():
        return bound_a + bound_b
    def g():
        return bound_a - bound_b
   return types.SimpleNamespace(**locals())
__ref = __scope(__arg1, __arg2)
__ref.__doc__ = __scope.__doc__
op(__ref.f, __ref.g)

expr_stmt: testlist_star_expr (augassign (yield_expr|testlist) |
             ('=' (yield_expr|testlist_star_expr))*)
del_stmt: 'del' exprlist
pass_stmt: 'pass'
return_stmt: 'return' [testlist]
yield_stmt: yield_expr
raise_stmt: 'raise' [test ['from' test]]
assert_stmt: 'assert' test [',' test]

expr_stmt: testlist_star_expr (augassign (yield_expr|testlist) |
             ('=' (yield_expr|testlist_star_expr))*) [given_clause]
del_stmt: 'del' exprlist [given_clause]
pass_stmt: 'pass' [given_clause]
return_stmt: 'return' [testlist] [given_clause]
yield_stmt: yield_expr [given_clause]
raise_stmt: 'raise' [test ['from' test]] [given_clause]
assert_stmt: 'assert' test [',' test] [given_clause]
given_clause: "given" [(NAME '=' test)+ "in"]":" suite

break given:
    a = b = 1

import sys given:
    a = b = 1

stmt: simple_stmt | given_stmt | compound_stmt
simple_stmt: small_stmt (';' (small_stmt | subexpr_stmt))* [';'] NEWLINE
small_stmt: (pass_stmt | flow_stmt | import_stmt |
             global_stmt | nonlocal_stmt)
flow_stmt: break_stmt | continue_stmt
given_stmt: subexpr_stmt (given_clause |
              (';' (small_stmt | subexpr_stmt))* [';']) NEWLINE
subexpr_stmt: expr_stmt | del_stmt | flow_subexpr_stmt | assert_stmt
flow_subexpr_stmt: return_stmt | raise_stmt | yield_stmt
given_clause: "given" (NAME '=' test)* ":" suite

stmt: simple_stmt | compound_stmt
simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE
small_stmt: (expr_stmt | del_stmt | pass_stmt | flow_stmt |
             import_stmt | global_stmt | nonlocal_stmt | assert_stmt)
flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt | yield_stmt

Rationale

Function and class statements in Python have a unique property relative to ordinary assignment statements: to some degree, they are declarative. They present the reader of the code with some critical information about a name that is about to be defined, before proceeding on with the details of the actual definition in the function or class body.

The name of the object being declared is the first thing stated after the keyword. Other important information is also given the honour of preceding the implementation details:

• decorators (which can greatly affect the behaviour of the created object, and were placed ahead of even the keyword and name as a matter of practicality more so than aesthetics)
• the docstring (on the first line immediately following the header line)
• parameters, default values and annotations for function definitions
• parent classes, metaclass and optionally other details (depending on the metaclass) for class definitions

This PEP proposes to make a similar declarative style available for arbitrary assignment operations, by permitting the inclusion of a “given” suite following any simple assignment statement:

TARGET = [TARGET2 = ... TARGETN =] EXPR given:
    SUITE

By convention, code in the body of the suite should be oriented solely towards correctly defining the assignment operation carried out in the header line. The header line operation should also be adequately descriptive (e.g. through appropriate choices of variable names) to give a reader a reasonable idea of the purpose of the operation without reading the body of the suite.

However, while they are the initial motivating use case, limiting this feature solely to simple assignments would be overly restrictive. Once the feature is defined at all, it would be quite arbitrary to prevent its use for augmented assignments, return statements, yield expressions, comprehensions and arbitrary expressions that may modify the application state.

The given clause may also function as a more readable alternative to some uses of lambda expressions and similar constructs when passing one-off functions to operations like sorted() or in callback based event-driven programming.

In module and class level code, the given clause will serve as a clear and reliable replacement for usage of the del statement to keep interim working variables from polluting the resulting namespace.

One potentially useful way to think of the proposed clause is as a middle ground between conventional in-line code and separation of an operation out into a dedicated function, just as an inline while loop may eventually be factored out into a dedicated generator.

Design Discussion

seq2 = [x for x in y if q(x) for y in seq if p(y)]

# would be equivalent to

seq2 = ?.result given seq=seq:
    result = []
    for y in seq:
        if p(y):
            for x in y:
                if q(x):
                    result.append(x)

seq2 = ?.g(seq) given:
    def g(seq):
        for y in seq:
            if p(y):
                for x in y:
                    if q(x):
                        yield x

seq2 = ? given seq=seq in:
    for y in seq:
        if p(y):
            for x in y:
                if q(x):
                    yield x

@classmethod
def classname(cls):
    return cls.__name__

classname = .d1(classname) given:
    d1 = classmethod
    def classname(cls):
        return cls.__name__

sorted_list = sorted(original, key=(lambda v: v.attr1, v.attr2))

sorted_list = sorted(original,
                     key=operator.attrgetter(v. 'attr1', 'attr2'))

def sort_key(item):
    return item.attr1, item.attr2

sorted_list = sorted(original, key=sort_key)

sorted_list = sorted(original, key=?.key) given:
    def key(item):
        return item.attr1, item.attr2

Open Questions

Examples

Defining callbacks for event driven programming:

# Current Python (definition before use)
def cb(sock):
    # Do something with socket
def eb(exc):
    logging.exception(
        "Failed connecting to %s:%s", host, port)
loop.create_connection((host, port), cb, eb) given:

# Becomes:
loop.create_connection((host, port), ?.cb, ?.eb) given:
    def cb(sock):
        # Do something with socket
    def eb(exc):
        logging.exception(
            "Failed connecting to %s:%s", host, port)

Defining “one-off” classes which typically only have a single instance:

# Current Python (instantiation after definition)
class public_name():
  ... # However many lines
public_name = public_name(*params)

# Current Python (custom decorator)
def singleton(*args, **kwds):
    def decorator(cls):
        return cls(*args, **kwds)
    return decorator

@singleton(*params)
class public_name():
  ... # However many lines

# Becomes:
public_name = ?.MeaningfulClassName(*params) given:
  class MeaningfulClassName():
    ... # Should trawl the stdlib for an example of doing this

Calculating attributes without polluting the local namespace (from os.py):

# Current Python (manual namespace cleanup)
def _createenviron():
  ... # 27 line function

environ = _createenviron()
del _createenviron

# Becomes:
environ = ?._createenviron() given:
    def _createenviron():
      ... # 27 line function

Replacing default argument hack (from functools.lru_cache):

# Current Python (default argument hack)
def decorating_function(user_function,
               tuple=tuple, sorted=sorted, len=len, KeyError=KeyError):
  ... # 60 line function
return decorating_function

# Becomes:
return ?.decorating_function given:
  # Cell variables rather than locals, but should give similar speedup
  tuple, sorted, len, KeyError = tuple, sorted, len, KeyError
  def decorating_function(user_function):
    ... # 60 line function

# This example also nicely makes it clear that there is nothing in the
# function after the nested function definition. Due to additional
# nested functions, that isn't entirely clear in the current code.

Possible Additions

• The current proposal allows the addition of a given clause only for simple statements. Extending the idea to allow the use of compound statements would be quite possible (by appending the given clause as an independent suite at the end), but doing so raises serious readability concerns (as values defined in the given clause may be used well before they are defined, exactly the kind of readability trap that other features like decorators and with statements are designed to eliminate)
• The “explicit early binding” variant may be applicable to the discussions on python-ideas on how to eliminate the default argument hack. A given clause in the header line for functions (after the return type annotation) may be the answer to that question.

Rejected Alternatives

• An earlier version of this PEP allowed implicit forward references to the names in the trailing suite, and also used implicit early binding semantics. Both of these ideas substantially complicated the proposal without providing a sufficient increase in expressive power. The current proposing with explicit forward references and early binding brings the new construct into line with existing scoping semantics, greatly improving the chances the idea can actually be implemented.
• In addition to the proposals made here, there have also been suggestions of two suite “in-order” variants which provide the limited scoping of names without supporting out-of-order execution. I believe these suggestions largely miss the point of what people are complaining about when they ask for multi-line lambda support - it isn’t that coming up with a name for the subexpression is especially difficult, it’s that naming the function before the statement that uses it means the code no longer matches the way the developer thinks about the problem at hand.
• I’ve made some unpublished attempts to allow direct references to the closure implicitly created by the given clause, while still retaining the general structure of the syntax as defined in this PEP (For example, allowing a subexpression like ?given or :given to be used in expressions to indicate a direct reference to the implied closure, thus preventing it from being called automatically to create the local namespace). All such attempts have appeared unattractive and confusing compared to the simpler decorator-inspired proposal in PEP 403.

Reference Implementation

None as yet. If you want a crash course in Python namespace semantics and code compilation, feel free to try ;)

TO-DO

• Mention PEP 359 and possible uses for locals() in the given clause
• Figure out if this can be used internally to make the implementation of zero-argument super() calls less awful

References

[1]

Explicitation lines in Python: https://mail.python.org/pipermail/python-ideas/2010-June/007476.html

[2]

‘where’ statement in Python: https://mail.python.org/pipermail/python-ideas/2010-July/007584.html

[3]

Where-statement (Proposal for function expressions): https://mail.python.org/pipermail/python-ideas/2009-July/005132.html

[4]

Name conflict with NumPy for ‘where’ keyword choice: https://mail.python.org/pipermail/python-ideas/2010-July/007596.html

[5]

The “Status quo wins a stalemate” design principle: http://www.boredomandlaziness.org/2011/02/status-quo-wins-stalemate.html

[6]

Assignments in list/generator expressions: https://mail.python.org/pipermail/python-ideas/2011-April/009863.html

[7]

Possible PEP 3150 style guidelines (#1): https://mail.python.org/pipermail/python-ideas/2011-April/009869.html

[8]

Discussion of PEP 403 (statement local function definition): https://mail.python.org/pipermail/python-ideas/2011-October/012276.html

[9]

Possible PEP 3150 style guidelines (#2): https://mail.python.org/pipermail/python-ideas/2011-October/012341.html

[10]

Multi-line lambdas (again!) https://mail.python.org/pipermail/python-ideas/2013-August/022526.html

Copyright

This document has been placed in the public domain.