Python decorators, the right way: the 4 audiences of programming languages

Python decorators are a useful but flawed language feature. Intended to make source code easier to write, and a little more readable, they neglect to address another use case: that of the programmer who will be calling the decorated code.

If you’re a Python programmer, the following post will show you why decorators exist, and how to compensate for their limitations. And even if you’re not a Python a programmer, I hope to demonstrate the importance of keeping in mind all of the different audiences for the code you write.

Why decorators exists: authoring and reading code

A programming language needs to satisfy four different audiences:

1. The computer which will run the source code.
2. The author, the programmer writing the source code.
3. A future reader of the source code.
4. A future caller of the source code, a programmer who will write code that calls functions and classes in the source code.

Python decorators were created for authors and readers, but neglect the needs of callers. Let’s start by seeing what decorators are, and how they make it easier to author and read code.

Imagine you want to emulate the Java synchronized keyword: you want to run a method of a class with a lock held, so only one thread can call the method at a time. You can do so with the following code, where the synchronized functions creates a new, replacement method that wraps the given one:

from threading import Lock

def synchronized(function):
    """
    Given a method, return a new method that acquires a
    lock, calls the given method, and then releases the
    lock.
    """
    def wrapper(self, *args, **kwargs):
        """A synchronized wrapper."""
        with self._lock:
            return function(self, *args, **kwargs)
    return wrapper

You can then use the synchronized utility like so:

class ExampleSynchronizedClass:
    def __init__(self):
        self._lock = Lock()
        self._items = []

    # Problematic usage:
    def add(self, item):
        """Add a new item."""
        self._items.append(item)
    add = synchronized(add)

As an author this usage is problematic: you need to type “add” twice, leading to a potential for typos. As a reader of the code you also only learn that add() is synchronized at the end, rather than the beginning. Python therefore provides the decorator syntax, which does the exact same thing as the above but more succinctly:

class ExampleSynchronizedClass:
    def __init__(self):
        self._lock = Lock()
        self._items = []

    # Nicer decorator usage:
    @synchronized
    def add(self, item):
        """Add a new item."""
        self._items.append(item)

Where decorators fail: calling code

The problem with decorators is that they fail to address the needs of programmers calling the decorated functions. As a user of ExampleSynchronizedClass you likely want your editor or IDE to show the docstring for add, and to detect the appropriate signature. Likewise if you’re writing documentation and want to automatically generate an API reference from the source code.

But in fact, what you get is the signature, name and docstring for the wrapper function:

>>> help(ExampleSynchronizedClass.add)
Help on method wrapper in module synchronized:

wrapper(self, *args, **kwargs) unbound synchronized.ExampleSynchronizedClass method
    A synchronized wrapper.

To solve this Python provides a utility decorator called functools.wraps, that copies attributes like name and docstring from the wrapped function. We change the implementation of the decorator:

from threading import Lock
from functools import wraps

def synchronized(function):
    """
    Given a method, return a new method that acquires a
    lock, calls the given method, and then releases the
    lock.
    """
    @wraps(function)
    def wrapper(self, *args, **kwargs):
        """A synchronized wrapper."""
        with self._lock:
            return function(self, *args, **kwargs)
    return wrapper

And now we get better help:

Help on method add in module synchronized:

add(self, item) unbound synchronized.ExampleSynchronizedClass method
    Add a new item.

In versions of Python less than 3.4 signature will still be wrong: it’s still the signature of the wrapper, not the underlying function. If you want to support older versions of Python, one solution is to use a 3rd party library called wrapt. We redefine our decorator once more, this time using wrapt instead of functools.wraps:

import wrapt
from threading import Lock

@wrapt.decorator
def synchronized(function, self, args, kwargs):
    """
    Given a method, return a new method that acquires a
    lock, calls the given method, and then releases the
    lock.
    """
    with self._lock:
        return function(*args, **kwargs)

Beyond supporting older versions of Python, wrapt also has the benefit of being more succinct.

Addressing all audiences

While functols.wraps and wrapt do the trick, they still require you to remember to use them every time you define a new decorator. Arguably this is a failure in the Python language: it would’ve been more elegant to do the equivalent functionality as part of the @ syntax in the language itself, rather than relying on library code to fix it.

When you are writing a library, or perhaps even designing a programming language, it’s always worth keeping in mind that you need to support four distinct audiences: the computer, authors, readers and callers. And if you’re a Python programmer creating a decorator, do use wrapt: it’ll make your callers happier, and since it’s also more succinct it will also make life a little easier for your readers.

Updated: Noted Python 3.4 does do signatures, and tried to make the issue with flaw more explicit. Thanks to Kevin Granger and hwayne for suggestions.