Decorator

City University of Hong Kong

CS1302 Introduction to Computer Programming

%reload_ext divewidgets

Optional Arguments¶

Recall the generator for Fibonacci numbers:

%%optlite -l -h 450
def fibonacci_sequence(Fn, Fnn, stop):
    """Return a generator that generates Fibonacci numbers
    starting from Fn and Fnn until stop (exclusive)."""
    while Fn < stop:
        yield Fn  # return Fn and pause execution
        Fn, Fnn = Fnn, Fnn + Fn


for fib in fibonacci_sequence(0, 1, 5):
    print(fib)

Fibonacci sequence normally starts with 0 and 1 by default. Is it possible to make arguments Fn and Fnn optional with default values?

How to give arguments default values?

def fibonacci_sequence(Fn=0, Fnn=1, stop=None):
    while stop is None or Fn < stop:
        value = yield Fn
        Fn, Fnn = Fnn, Fnn + Fn

for fib in fibonacci_sequence(stop=5):
    print(fib)  # with default Fn=0, Fnn=1

for fib in fibonacci_sequence(5):
    print(fib)
    if fib > 10:  
        break  # Will this be executed?

With fibonacci_sequence(5), Fn=5 while Fnn=1 and stop=None by default. The while loop in the definition of fibonacci_sequence becomes an infinite loop. The generator keeps generating the next Fibonacci number without raising StopIteration exception, and so the for loop will be an infinite loop unless it is terminated by the break statement.

E.g., the following results in an error:

fibonacci_sequence(stop=10, 1)

fibonacci_sequence(1, Fn=1)

The following shows that the behavior of range is different.

for count in range(1, 10, 2):
    print(count, end=" ")  # counts from 1 to 10 in steps of 2
print()
for count in range(1, 10):
    print(count, end=" ")  # default step=1
print()
for count in range(10):
    print(count, end=" ")  # default start=0, step=1
range(stop=10)  # fails

range takes only positional arguments.
However, the first positional argument has different intepretations (start or stop) depending on the number of arguments (2 or 1).

range is indeed NOT a generator.

print(type(range), type(range(10)))

How is range implemented?

Variable number of arguments¶

The implementation of range uses a variable number of arguments.

def print_arguments(*args, **kwargs):
    """Take any number of arguments and prints them"""
    print("args ({}): {}".format(type(args), args))
    print("kwargs ({}): {}".format(type(kwargs), kwargs))


print_arguments(0, 10, 2, start=1, stop=2)

args is a tuple of positional arguments.
kwargs is a dictionary of keyword arguments, which is a list of values indexed by unique keys that are not necessary integers.

d = {'start': 1, 'stop': 2}
d['start'], d['stop'], d.keys(), d.values(), d.items()

* and ** are unpacking operators for tuple/list and dictionary respectively:

args = (0, 10, 2)
kwargs = {"start": 1, "stop": 2}
print_arguments(*args, **kwargs)

The following function converts all the arguments to a string, which will be useful later on.

def argument_string(*args, **kwargs):
    """Return the string representation of the list of arguments."""
    return "({})".format(
        ", ".join(
            [
                *(f"{v!r}" for v in args),  # arguments
                *(f"{k}={v!r}" for k, v in kwargs.items()),  # keyword arguments
            ]
        )
    )


argument_string(0, 10, 2, start=1, stop=2)

def fibonacci_sequence(*args):
    """Return a generator that generates Fibonacci numbers
    starting from Fn and Fnn to stop (exclusive).
    generator.send(value) sets next number to value.

    fibonacci_sequence(stop)
    fibonacci_sequence(Fn,Fnn)
    fibonacci_sequence(Fn,Fnn,stop)
    """
    Fn, Fnn, stop = 0, 1, None  # default values

    # handle different number of arguments
    if len(args) == 1:
        ### BEGIN SOLUTION
        stop = args[0]
        ### END SOLUTION
    elif len(args) == 2:
        Fn, Fnn = args[0], args[1]
    elif len(args) > 2:
        Fn, Fnn, stop = args[0], args[1], args[2]

    while stop is None or Fn < stop:
        value = yield Fn
        if value is not None:
            Fnn = value  # set next number to the value of yield expression
        Fn, Fnn = Fnn, Fnn + Fn

for fib in fibonacci_sequence(5):  # default Fn=0, Fn=1
    print(fib)

for fib in fibonacci_sequence(1, 2):  # default stop=None
    print(fib)
    if fib > 5:
        break

args = (1, 2, 5)
for fib in fibonacci_sequence(*args):  # default stop=None
    print(fib)

Decorator¶

What is function decoration?
Why decorate a function?

def fibonacci(n):
    """Returns the Fibonacci number of order n."""
    global count, depth
    count += 1
    depth += 1
    print("{:>3}: {}fibonacci({!r})".format(count, "|" * depth, n))

    value = fibonacci(n - 1) + fibonacci(n - 2) if n > 1 else 1 if n == 1 else 0

    depth -= 1
    if depth == -1:  # recursion done
        print("Done")
        count = 0  # reset count for subsequent recursions
    return value


count, depth = 0, -1
for n in range(6):
    print(fibonacci(n))

The code decorates the fibonacci function by printing each recursive call and the depth of the call stack.
The decoration is useful in showing the efficiency of the function, but it rewrites the function definition.

How to decorate a function without changing its implementation?

Decorations are often temporary. Is it possible to avoid

going through the source codes to remove decorations?
switching back and forth between the original and decorated codes?

def fibonacci(n):
    """Returns the Fibonacci number of order n."""
    return fibonacci(n - 1) + fibonacci(n - 2) if n > 1 else 1 if n == 1 else 0


def fibonacci_decorated(n):
    """Returns the Fibonacci number of order n."""
    global count, depth
    count += 1
    depth += 1
    print("{:>3}: {}fibonacci({!r})".format(count, "|" * depth, n))

    value = fibonacci(n)

    depth -= 1
    if depth == -1:  # recursion done
        print("Done")
        count = 0  # reset count for subsequent recursions
    return value


count, depth = 0, -1
for n in range(6):
    print(fibonacci_decorated(n))

Solution to Exercise 3

The recursive calls to fibonacci does not print the function calls, because fibonacci does not call fibonacci_decorated.

Solution to Exercise 4

The call leads to an infinite recursion. The desired recursion fibonacci points to originally is no longer invoked.

An elegant solution is to

capture the function to be decorated in the closure of the decorated function, and
rename the decorated function to the same name as the function to be decorated.

def print_function_call(f):
    def wrapper(*args, **kwargs):
        nonlocal count, depth
        count += 1
        depth += 1
        call = "{}{}".format(f.__name__, argument_string(*args, **kwargs))
        print("{:>3}:{}{}".format(count, "|" * depth, call))
        value = f(*args, **kwargs)  # calls f
        depth -= 1
        if depth == -1:
            print("Done")
            count = 0
        return value
    count, depth = 0, -1
    return wrapper  # return the decorated function

The above defines a decorator print_function_call that takes in a function f to be decorated and returns the decorated function wrapper that captures and decorates f:

wrapper expects the same set of arguments for f,
returns the same value returned by f on the arguments, but
can execute additional codes before and after calling f to print the function call.

By redefining fibonacci as the returned wrapper, the original fibonacci captured by wrapper calls wrapper as desired.

def fibonacci(n):
    return fibonacci(n - 1) + fibonacci(n - 2) if n > 1 else 1 if n == 1 else 0


fibonacci = print_function_call(fibonacci)  # so original fibonnacci calls wrapper
fibonacci(5)

The redefinition does not change the original fibonacci captured by wrapper.

import inspect

for cell in fibonacci.__closure__:
    if callable(cell.cell_contents):
        print(inspect.getsource(cell.cell_contents))

Python provides the syntatic sugar below to simplify the redefinition.

@print_function_call
def fibonacci(n):
    return fibonacci(n - 1) + fibonacci(n - 2) if n > 1 else 1 if n == 1 else 0


fibonacci(5)

The following visualizes the execution step-by-step:

%%optlite -l
def print_function_call(f):
    def wrapper(*args, **kwargs):
        nonlocal count, depth
        count += 1
        depth += 1
        call = "{}{}".format(f.__name__, "({})".format(", ".join([*(f"{v!r}" for v in args), *(f"{k}={v!r}" for k, v in kwargs.items())])))
        print("{:>3}:{}{}".format(count, "|" * depth, call))
        value = f(*args, **kwargs)  # calls f
        depth -= 1
        if depth == -1:
            print("Done")
            count = 0
        return value

    count, depth = 0, -1
    return wrapper  # return the decorated function

@print_function_call
def fibonacci(n):
    return fibonacci(n - 1) + fibonacci(n - 2) if n > 1 else 1 if n == 1 else 0

fibonacci(5)

Solution to Exercise 5

To decorate any function with possibly different number of arguments.

Applications of decorators¶

Decorator is a software design pattern that are reused in different scenarios for different applications.

For example, the following uses decoration to improves decoration:

%%optlite -r -l
import functools

def print_function_call(f):
    @functools.wraps(f)  # give wrapper the identity of f and more
    def wrapper(*args, **kwargs):
        nonlocal count, depth
        count += 1
        depth += 1
        call = "{}{}".format(f.__name__, "({})".format(", ".join([*(f"{v!r}" for v in args), *(f"{k}={v!r}" for k, v in kwargs.items())])))
        print("{:>3}:{}{}".format(count, "|" * depth, call))
        value = f(*args, **kwargs)  # calls f
        depth -= 1
        if depth == -1:
            print("Done")
            count = 0
        return value

    count, depth = 0, -1
    return wrapper  # return the decorated function

@print_function_call
def fibonacci(n):
    return fibonacci(n - 1) + fibonacci(n - 2) if n > 1 else 1 if n == 1 else 0

fibonacci(5)

What is the difference?

The decoration @functools.wraps(f) of the decorated function wrapper

makes some attributes (such as __name__) of the decorated function the same as those of original function, and
adds some useful attributes such as __wrapped__ that points to the original function.

In the optlite visualization, notice that wrapper points to a function named fibonacci instead of wrapper.

We can also undo the decoration using __wrapped__.

fibonacci, fibonacci_decorated = fibonacci.__wrapped__, fibonacci  # recover
print("original fibonacci:")
print(fibonacci(5))

fibonacci = fibonacci_decorated  # decorate
print("decorated fibonacci:")
print(fibonacci(5))

How to use decorator to improve recursion?

We can also use a decorator to make recursion more efficient by caching the return values.
cache is a dictionary where cache[n] stores the computed value of $F_n$ to avoid redundant computations.

def caching(f):
    """Cache the return value of a function that takes a single argument.

    Parameters
    ----------
    f: Callable
        A function that takes a single argument.

    Returns
    -------
    Callable:
        The function same as f but has its return valued automatically cached
        when called. It has a method clear_cache to clear its cache.
    """

    @functools.wraps(f)
    def wrapper(n):
        if n not in cache:
            cache[n] = f(n)
        else:
            print("read from cache")
        return cache[n]

    cache = {}
    wrapper.clear_cache = lambda: cache.clear()  # add method to clear cache
    return wrapper


@print_function_call
@caching
def fibonacci(n):
    return fibonacci(n - 1) + fibonacci(n - 2) if n > 1 else 1 if n == 1 else 0

fibonacci(5)
fibonacci(5)
fibonacci.clear_cache()
fibonacci(5)

A method clear_cache is added to the wrapper to clear the cache.
lambda <argument list> : <expression>is called a lambda expression, which conveniently defines an anonymous function.

type(fibonacci.clear_cache), fibonacci.clear_cache.__name__

Module¶

How to create a module?

To create a module, simply put the code in a python source file <module name>.py in

the current directory, or
a python site-packages directory in system path.

import sys

print(sys.path)

For example, recurtools.py in the current directory defines the module recurtools.

from IPython.display import Code

Code(filename="recurtools.py", language="python")

The module provides the decorators print_function_call and caching defined earlier.

import recurtools as rc


@rc.print_function_call
@rc.caching
def factorial(n):
    return factorial(n - 1) if n > 1 else 1

factorial(5)
factorial(5)
factorial.clear_cache()
factorial(5)