Python Enhancement Proposals

PEP 3104 – Access to Names in Outer Scopes

PEP
3104
Title
Access to Names in Outer Scopes
Author
Ka-Ping Yee <ping at zesty.ca>
Status
Final
Type
Standards Track
Created
12-Oct-2006
Python-Version
3.0
Post-History

Contents

Abstract

In most languages that support nested scopes, code can refer to or rebind (assign to) any name in the nearest enclosing scope. Currently, Python code can refer to a name in any enclosing scope, but it can only rebind names in two scopes: the local scope (by simple assignment) or the module-global scope (using a global declaration).

This limitation has been raised many times on the Python-Dev mailing list and elsewhere, and has led to extended discussion and many proposals for ways to remove this limitation. This PEP summarizes the various alternatives that have been suggested, together with advantages and disadvantages that have been mentioned for each.

Rationale

Before version 2.1, Python’s treatment of scopes resembled that of standard C: within a file there were only two levels of scope, global and local. In C, this is a natural consequence of the fact that function definitions cannot be nested. But in Python, though functions are usually defined at the top level, a function definition can be executed anywhere. This gave Python the syntactic appearance of nested scoping without the semantics, and yielded inconsistencies that were surprising to some programmers – for example, a recursive function that worked at the top level would cease to work when moved inside another function, because the recursive function’s own name would no longer be visible in its body’s scope. This violates the intuition that a function should behave consistently when placed in different contexts. Here’s an example:

def enclosing_function():
    def factorial(n):
        if n < 2:
            return 1
        return n * factorial(n - 1)  # fails with NameError
    print factorial(5)

Python 2.1 moved closer to static nested scoping by making visible the names bound in all enclosing scopes (see PEP 227). This change makes the above code example work as expected. However, because any assignment to a name implicitly declares that name to be local, it is impossible to rebind a name in an outer scope (except when a global declaration forces the name to be global). Thus, the following code, intended to display a number that can be incremented and decremented by clicking buttons, doesn’t work as someone familiar with lexical scoping might expect:

def make_scoreboard(frame, score=0):
    label = Label(frame)
    label.pack()
    for i in [-10, -1, 1, 10]:
        def increment(step=i):
            score = score + step  # fails with UnboundLocalError
            label['text'] = score
        button = Button(frame, text='%+d' % i, command=increment)
        button.pack()
    return label

Python syntax doesn’t provide a way to indicate that the name score mentioned in increment refers to the variable score bound in make_scoreboard, not a local variable in increment. Users and developers of Python have expressed an interest in removing this limitation so that Python can have the full flexibility of the Algol-style scoping model that is now standard in many programming languages, including JavaScript, Perl, Ruby, Scheme, Smalltalk, C with GNU extensions, and C# 2.0.

It has been argued that such a feature isn’t necessary, because a rebindable outer variable can be simulated by wrapping it in a mutable object:

class Namespace:
    pass

def make_scoreboard(frame, score=0):
    ns = Namespace()
    ns.score = 0
    label = Label(frame)
    label.pack()
    for i in [-10, -1, 1, 10]:
        def increment(step=i):
            ns.score = ns.score + step
            label['text'] = ns.score
        button = Button(frame, text='%+d' % i, command=increment)
        button.pack()
    return label

However, this workaround only highlights the shortcomings of existing scopes: the purpose of a function is to encapsulate code in its own namespace, so it seems unfortunate that the programmer should have to create additional namespaces to make up for missing functionality in the existing local scopes, and then have to decide whether each name should reside in the real scope or the simulated scope.

Another common objection is that the desired functionality can be written as a class instead, albeit somewhat more verbosely. One rebuttal to this objection is that the existence of a different implementation style is not a reason to leave a supported programming construct (nested scopes) functionally incomplete. Python is sometimes called a “multi-paradigm language” because it derives so much strength, practical flexibility, and pedagogical power from its support and graceful integration of multiple programming paradigms.

A proposal for scoping syntax appeared on Python-Dev as far back as 1994 [1], long before PEP 227’s support for nested scopes was adopted. At the time, Guido’s response was:

This is dangerously close to introducing CSNS [classic static nested scopes]. If you were to do so, your proposed semantics of scoped seem alright. I still think there is not enough need for CSNS to warrant this kind of construct …

After PEP 227, the “outer name rebinding discussion” has reappeared on Python-Dev enough times that it has become a familiar event, having recurred in its present form since at least 2003 [2]. Although none of the language changes proposed in these discussions have yet been adopted, Guido has acknowledged that a language change is worth considering [12].

Other Languages

To provide some background, this section describes how some other languages handle nested scopes and rebinding.

JavaScript, Perl, Scheme, Smalltalk, GNU C, C# 2.0

These languages use variable declarations to indicate scope. In JavaScript, a lexically scoped variable is declared with the var keyword; undeclared variable names are assumed to be global. In Perl, a lexically scoped variable is declared with the my keyword; undeclared variable names are assumed to be global. In Scheme, all variables must be declared (with define or let, or as formal parameters). In Smalltalk, any block can begin by declaring a list of local variable names between vertical bars. C and C# require type declarations for all variables. For all these cases, the variable belongs to the scope containing the declaration.

Ruby (as of 1.8)

Ruby is an instructive example because it appears to be the only other currently popular language that, like Python, tries to support statically nested scopes without requiring variable declarations, and thus has to come up with an unusual solution. Functions in Ruby can contain other function definitions, and they can also contain code blocks enclosed in curly braces. Blocks have access to outer variables, but nested functions do not. Within a block, an assignment to a name implies a declaration of a local variable only if it would not shadow a name already bound in an outer scope; otherwise assignment is interpreted as rebinding of the outer name. Ruby’s scoping syntax and rules have also been debated at great length, and changes seem likely in Ruby 2.0 [28].

Overview of Proposals

There have been many different proposals on Python-Dev for ways to rebind names in outer scopes. They all fall into two categories: new syntax in the scope where the name is bound, or new syntax in the scope where the name is used.

New Syntax in the Binding (Outer) Scope

Scope Override Declaration

The proposals in this category all suggest a new kind of declaration statement similar to JavaScript’s var. A few possible keywords have been proposed for this purpose:

In all these proposals, a declaration such as var x in a particular scope S would cause all references to x in scopes nested within S to refer to the x bound in S.

The primary objection to this category of proposals is that the meaning of a function definition would become context-sensitive. Moving a function definition inside some other block could cause any of the local name references in the function to become nonlocal, due to declarations in the enclosing block. For blocks in Ruby 1.8, this is actually the case; in the following example, the two setters have different effects even though they look identical:

setter1 = proc { | x | y = x }      # y is local here
y = 13
setter2 = proc { | x | y = x }      # y is nonlocal here
setter1.call(99)
puts y                              # prints 13
setter2.call(77)
puts y                              # prints 77

Note that although this proposal resembles declarations in JavaScript and Perl, the effect on the language is different because in those languages undeclared variables are global by default, whereas in Python undeclared variables are local by default. Thus, moving a function inside some other block in JavaScript or Perl can only reduce the scope of a previously global name reference, whereas in Python with this proposal, it could expand the scope of a previously local name reference.

Required Variable Declaration

A more radical proposal [21] suggests removing Python’s scope-guessing convention altogether and requiring that all names be declared in the scope where they are to be bound, much like Scheme. With this proposal, var x = 3 would both declare x to belong to the local scope and bind it, where as x = 3 would rebind the existing visible x. In a context without an enclosing scope containing a var x declaration, the statement x = 3 would be statically determined to be illegal.

This proposal yields a simple and consistent model, but it would be incompatible with all existing Python code.

New Syntax in the Referring (Inner) Scope

There are three kinds of proposals in this category.

Outer Reference Expression

This type of proposal suggests a new way of referring to a variable in an outer scope when using the variable in an expression. One syntax that has been suggested for this is .x [7], which would refer to x without creating a local binding for it. A concern with this proposal is that in many contexts x and .x could be used interchangeably, which would confuse the reader [31]. A closely related idea is to use multiple dots to specify the number of scope levels to ascend [8], but most consider this too error-prone [17].

Rebinding Operator

This proposal suggests a new assignment-like operator that rebinds a name without declaring the name to be local [2]. Whereas the statement x = 3 both declares x a local variable and binds it to 3, the statement x := 3 would change the existing binding of x without declaring it local.

This is a simple solution, but according to PEP 3099 it has been rejected (perhaps because it would be too easy to miss or to confuse with =).

Scope Override Declaration

The proposals in this category suggest a new kind of declaration statement in the inner scope that prevents a name from becoming local. This statement would be similar in nature to the global statement, but instead of making the name refer to a binding in the top module-level scope, it would make the name refer to the binding in the nearest enclosing scope.

This approach is attractive due to its parallel with a familiar Python construct, and because it retains context-independence for function definitions.

This approach also has advantages from a security and debugging perspective. The resulting Python would not only match the functionality of other nested-scope languages but would do so with a syntax that is arguably even better for defensive programming. In most other languages, a declaration contracts the scope of an existing name, so inadvertently omitting the declaration could yield farther-reaching (i.e. more dangerous) effects than expected. In Python with this proposal, the extra effort of adding the declaration is aligned with the increased risk of non-local effects (i.e. the path of least resistance is the safer path).

Many spellings have been suggested for such a declaration:

  • scoped x [1]
  • global x in f [3] (explicitly specify which scope)
  • free x [5]
  • outer x [6]
  • use x [9]
  • global x [10] (change the meaning of global)
  • nonlocal x [11]
  • global x outer [18]
  • global in x [18]
  • not global x [18]
  • extern x [20]
  • ref x [22]
  • refer x [22]
  • share x [22]
  • sharing x [22]
  • common x [22]
  • using x [22]
  • borrow x [22]
  • reuse x [23]
  • scope f x [25] (explicitly specify which scope)

The most commonly discussed choices appear to be outer, global, and nonlocal. outer is already used as both a variable name and an attribute name in the standard library. The word global has a conflicting meaning, because “global variable” is generally understood to mean a variable with top-level scope [27]. In C, the keyword extern means that a name refers to a variable in a different compilation unit. While nonlocal is a bit long and less pleasant-sounding than some of the other options, it does have precisely the correct meaning: it declares a name not local.

Proposed Solution

The solution proposed by this PEP is to add a scope override declaration in the referring (inner) scope. Guido has expressed a preference for this category of solution on Python-Dev [14] and has shown approval for nonlocal as the keyword [19].

The proposed declaration:

nonlocal x

prevents x from becoming a local name in the current scope. All occurrences of x in the current scope will refer to the x bound in an outer enclosing scope. As with global, multiple names are permitted:

nonlocal x, y, z

If there is no pre-existing binding in an enclosing scope, the compiler raises a SyntaxError. (It may be a bit of a stretch to call this a syntax error, but so far SyntaxError is used for all compile-time errors, including, for example, __future__ import with an unknown feature name.) Guido has said that this kind of declaration in the absence of an outer binding should be considered an error [16].

If a nonlocal declaration collides with the name of a formal parameter in the local scope, the compiler raises a SyntaxError.

A shorthand form is also permitted, in which nonlocal is prepended to an assignment or augmented assignment:

nonlocal x = 3

The above has exactly the same meaning as nonlocal x; x = 3. (Guido supports a similar form of the global statement [24].)

On the left side of the shorthand form, only identifiers are allowed, not target expressions like x[0]. Otherwise, all forms of assignment are allowed. The proposed grammar of the nonlocal statement is:

nonlocal_stmt ::=
    "nonlocal" identifier ("," identifier)*
               ["=" (target_list "=")+ expression_list]
  | "nonlocal" identifier augop expression_list

The rationale for allowing all these forms of assignment is that it simplifies understanding of the nonlocal statement. Separating the shorthand form into a declaration and an assignment is sufficient to understand what it means and whether it is valid.

Note

The shorthand syntax was not added in the original implementation of the PEP. Later discussions [29] [30] concluded this syntax should not be implemented.

Backward Compatibility

This PEP targets Python 3000, as suggested by Guido [19]. However, others have noted that some options considered in this PEP may be small enough changes to be feasible in Python 2.x [26], in which case this PEP could possibly be moved to be a 2.x series PEP.

As a (very rough) measure of the impact of introducing a new keyword, here is the number of times that some of the proposed keywords appear as identifiers in the standard library, according to a scan of the Python SVN repository on November 5, 2006:

nonlocal    0
use         2
using       3
reuse       4
free        8
outer     147

global appears 214 times as an existing keyword. As a measure of the impact of using global as the outer-scope keyword, there are 18 files in the standard library that would break as a result of such a change (because a function declares a variable global before that variable has been introduced in the global scope):

cgi.py
dummy_thread.py
mhlib.py
mimetypes.py
idlelib/PyShell.py
idlelib/run.py
msilib/__init__.py
test/inspect_fodder.py
test/test_compiler.py
test/test_decimal.py
test/test_descr.py
test/test_dummy_threading.py
test/test_fileinput.py
test/test_global.py (not counted: this tests the keyword itself)
test/test_grammar.py (not counted: this tests the keyword itself)
test/test_itertools.py
test/test_multifile.py
test/test_scope.py (not counted: this tests the keyword itself)
test/test_threaded_import.py
test/test_threadsignals.py
test/test_warnings.py

References

[1] (1, 2)
Scoping (was Re: Lambda binding solved?) (Rafael Bracho) https://legacy.python.org/search/hypermail/python-1994q1/0301.html
[2] (1, 2)
Extended Function syntax (Just van Rossum) https://mail.python.org/pipermail/python-dev/2003-February/032764.html
[3]
Closure semantics (Guido van Rossum) https://mail.python.org/pipermail/python-dev/2003-October/039214.html
[4] (1, 2)
Better Control of Nested Lexical Scopes (Almann T. Goo) https://mail.python.org/pipermail/python-dev/2006-February/061568.html
[5]
PEP for Better Control of Nested Lexical Scopes (Jeremy Hylton) https://mail.python.org/pipermail/python-dev/2006-February/061602.html
[6]
PEP for Better Control of Nested Lexical Scopes (Almann T. Goo) https://mail.python.org/pipermail/python-dev/2006-February/061603.html
[7]
Using and binding relative names (Phillip J. Eby) https://mail.python.org/pipermail/python-dev/2006-February/061636.html
[8]
Using and binding relative names (Steven Bethard) https://mail.python.org/pipermail/python-dev/2006-February/061749.html
[9] (1, 2)
Lexical scoping in Python 3k (Ka-Ping Yee) https://mail.python.org/pipermail/python-dev/2006-July/066862.html
[10]
Lexical scoping in Python 3k (Greg Ewing) https://mail.python.org/pipermail/python-dev/2006-July/066889.html
[11]
Lexical scoping in Python 3k (Ka-Ping Yee) https://mail.python.org/pipermail/python-dev/2006-July/066942.html
[12]
Lexical scoping in Python 3k (Guido van Rossum) https://mail.python.org/pipermail/python-dev/2006-July/066950.html
[13]
Explicit Lexical Scoping (pre-PEP?) (Talin) https://mail.python.org/pipermail/python-dev/2006-July/066978.html
[14]
Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum) https://mail.python.org/pipermail/python-dev/2006-July/066991.html
[15]
Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum) https://mail.python.org/pipermail/python-dev/2006-July/066995.html
[16]
Lexical scoping in Python 3k (Guido van Rossum) https://mail.python.org/pipermail/python-dev/2006-July/066968.html
[17]
Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum) https://mail.python.org/pipermail/python-dev/2006-July/067004.html
[18] (1, 2, 3)
Explicit Lexical Scoping (pre-PEP?) (Andrew Clover) https://mail.python.org/pipermail/python-dev/2006-July/067007.html
[19] (1, 2)
Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum) https://mail.python.org/pipermail/python-dev/2006-July/067067.html
[20]
Explicit Lexical Scoping (pre-PEP?) (Matthew Barnes) https://mail.python.org/pipermail/python-dev/2006-July/067221.html
[21]
Sky pie: a “var” keyword (a thread started by Neil Toronto) https://mail.python.org/pipermail/python-3000/2006-October/003968.html
[22] (1, 2, 3, 4, 5, 6, 7)
Alternatives to ‘outer’ (Talin) https://mail.python.org/pipermail/python-3000/2006-October/004021.html
[23]
Alternatives to ‘outer’ (Jim Jewett) https://mail.python.org/pipermail/python-3000/2006-November/004153.html
[24]
Draft PEP for outer scopes (Guido van Rossum) https://mail.python.org/pipermail/python-3000/2006-November/004166.html
[25]
Draft PEP for outer scopes (Talin) https://mail.python.org/pipermail/python-3000/2006-November/004190.html
[26]
Draft PEP for outer scopes (Nick Coghlan) https://mail.python.org/pipermail/python-3000/2006-November/004237.html
[27]
Global variable (version 2006-11-01T01:23:16) http://en.wikipedia.org/wiki/Global_variable
[28]
Ruby 2.0 block local variable http://redhanded.hobix.com/inspect/ruby20BlockLocalVariable.html
[29]
Issue 4199: combining assignment with global & nonlocal (Guido van Rossum) https://mail.python.org/pipermail/python-dev/2013-June/127142.html
[30]
Whatever happened to ‘nonlocal x = y’? (Guido van Rossum) https://mail.python.org/pipermail/python-dev/2018-January/151627.html
[31]
Using and binding relative names (Almann T. Goo) https://mail.python.org/pipermail/python-dev/2006-February/061761.html

Acknowledgements

The ideas and proposals mentioned in this PEP are gleaned from countless Python-Dev postings. Thanks to Jim Jewett, Mike Orr, Jason Orendorff, and Christian Tanzer for suggesting specific edits to this PEP.

Source: https://github.com/python-discord/peps/blob/main/pep-3104.txt

Last modified: 2022-01-21 11:03:51 GMT