Decorator¶

Optional Arguments¶

Recall the generator for Fibonacci numbers:

%%mytutor -h 450
def fibonacci_sequence(Fn, Fnn, stop):
    """Return a generator that generates Fibonacci numbers
    starting from Fn and Fnn until stop (exclusive)."""
    while Fn < stop:
        yield Fn  # return Fn and pause execution
        Fn, Fnn = Fnn, Fnn + Fn


for fib in fibonacci_sequence(0, 1, 5):
    print(fib)

Fibonacci sequence normally starts with 0 and 1 by default. Is it possible to make arguments Fn and Fnn optional?

How to give arguments default values?

def fibonacci_sequence(Fn=0, Fnn=1, stop=None):
    while stop is None or Fn < stop:
        value = yield Fn
        Fn, Fnn = Fnn, Fnn + Fn

Arguments with default values specified by =... are called default arguments. They are optional in the function call:

for fib in fibonacci_sequence(stop=5):
    print(fib)  # with default Fn=0, Fnn=1

stop=5 in the function call is called a keyword argument. As supposed to positional arguments, it specifies the name of the argument explicitly.

Exercise Is fibonacci_sequence(stop=5) the same as fibonacci_sequence(5)? In particular, what is the behavior of the following code?

for fib in fibonacci_sequence(5):
    print(fib)
    if fib > 10:  
        break  # Will this be executed?

With fibonacci_sequence(5), Fn=5 while Fnn=1 and stop=None by default. The while loop in the definition of fibonacci_sequence becomes an infinite loop. The generator keeps generating the next Fibonacci number without raising StopIteration exception, and so the for loop will be an infinite loop unless it is terminated by the break statement.

Rules for specifying arguments:

Default (Keyword) arguments must be after all non-default (positional) arguments in a function definition (call).
The value for an argument can not be specified more than once in a function definition (call).

E.g., the following results in an error:

fibonacci_sequence(stop=10, 1)

  File "<ipython-input-6-c4b4809b18c1>", line 1
    fibonacci_sequence(stop=10, 1)
                                ^
SyntaxError: positional argument follows keyword argument

fibonacci_sequence(1, Fn=1)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-2db5e024912c> in <module>
----> 1 fibonacci_sequence(1, Fn=1)

TypeError: fibonacci_sequence() got multiple values for argument 'Fn'

The following shows that the behavior of range is different.

for count in range(1, 10, 2):
    print(count, end=" ")  # counts from 1 to 10 in steps of 2
print()
for count in range(1, 10):
    print(count, end=" ")  # default step=1
print()
for count in range(10):
    print(count, end=" ")  # default start=0, step=1
range(stop=10)  # fails

3 5 7 9 
2 3 4 5 6 7 8 9 
1 2 3 4 5 6 7 8 9 

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-6358db32db1e> in <module>
      7 for count in range(10):
      8     print(count, end=" ")  # default start=0, step=1
----> 9 range(stop=10)  # fails

TypeError: range() takes no keyword arguments

range takes only positional arguments.
However, the first positional argument has different intepretations (start or stop) depending on the number of arguments (2 or 1).

range is indeed NOT a generator.

print(type(range), type(range(10)))

<class 'type'> <class 'range'>

How is range implemented?

Variable number of arguments¶

The implementation of range uses a variable number of arguments.

def print_arguments(*args, **kwargs):
    """Take any number of arguments and prints them"""
    print("args ({}): {}".format(type(args), args))
    print("kwargs ({}): {}".format(type(kwargs), kwargs))


print_arguments(0, 10, 2, start=1, stop=2)

args (<class 'tuple'>): (0, 10, 2)
kwargs (<class 'dict'>): {'start': 1, 'stop': 2}

args is a tuple of positional arguments.
kwargs is a dictionary of keyword arguments.

* and ** are unpacking operators for tuple/list and dictionary respectively:

args = (0, 10, 2)
kwargs = {"start": 1, "stop": 2}
print_arguments(*args, **kwargs)

args (<class 'tuple'>): (0, 10, 2)
kwargs (<class 'dict'>): {'start': 1, 'stop': 2}

The following function converts all the arguments to a string.
It will be useful later on.

def argument_string(*args, **kwargs):
    """Return the string representation of the list of arguments."""
    return "({})".format(
        ", ".join(
            [
                *["{!r}".format(v) for v in args],  # arguments
                *[
                    "{}={!r}".format(k, v) for k, v in kwargs.items()
                ],  # keyword arguments
            ]
        )
    )


argument_string(0, 10, 2, start=1, stop=2)

'(0, 10, 2, start=1, stop=2)'

Exercise Why use "{!r}".format(v) to format the argument?

Hint: See token conversion and the following code:

v = 'a'
"{!r}".format(v), repr(v)

("'a'", "'a'")

!r will convert \(v\) to the string representation that can be evaluated by python eval. In particular, 'a' will be converted to "'a'", which has the quotation needed for the string literal.

Exercise Redefine fibonacci_sequence so that the positional arguments depend on the number of arguments:

def fibonacci_sequence(*args):
    """Return a generator that generates Fibonacci numbers
    starting from Fn and Fnn to stop (exclusive).
    generator.send(value) sets next number to value.

    fibonacci_sequence(stop)
    fibonacci_sequence(Fn,Fnn)
    fibonacci_sequence(Fn,Fnn,stop)
    """
    Fn, Fnn, stop = 0, 1, None  # default values

    # handle different number of arguments
    if len(args) == 1:
        ### BEGIN SOLUTION
        stop = args[0]
        ### END SOLUTION
    elif len(args) == 2:
        Fn, Fnn = args[0], args[1]
    elif len(args) > 2:
        Fn, Fnn, stop = args[0], args[1], args[2]

    while stop is None or Fn < stop:
        value = yield Fn
        if value is not None:
            Fnn = value  # set next number to the value of yield expression
        Fn, Fnn = Fnn, Fnn + Fn

for fib in fibonacci_sequence(5):  # default Fn=0, Fn=1
    print(fib)

for fib in fibonacci_sequence(1, 2):  # default stop=None
    print(fib)
    if fib > 5:
        break

args = (1, 2, 5)
for fib in fibonacci_sequence(*args):  # default stop=None
    print(fib)

1
2
3

Decorator¶

What is function decoration?
Why decorate a function?

def fibonacci(n):
    """Returns the Fibonacci number of order n."""
    global count, depth
    count += 1
    depth += 1
    print("{:>3}: {}fibonacci({!r})".format(count, "|" * depth, n))

    value = fibonacci(n - 1) + fibonacci(n - 2) if n > 1 else 1 if n == 1 else 0

    depth -= 1
    if depth == -1:  # recursion done
        print("Done")
        count = 0  # reset count for subsequent recursions
    return value


count, depth = 0, -1
for n in range(6):
    print(fibonacci(n))

fibonacci(0)
Done
0
fibonacci(1)
Done
1
fibonacci(2)
|fibonacci(1)
|fibonacci(0)
Done
1
fibonacci(3)
|fibonacci(2)
||fibonacci(1)
||fibonacci(0)
|fibonacci(1)
Done
2
fibonacci(4)
|fibonacci(3)
||fibonacci(2)
|||fibonacci(1)
|||fibonacci(0)
||fibonacci(1)
|fibonacci(2)
||fibonacci(1)
||fibonacci(0)
Done
3
fibonacci(5)
|fibonacci(4)
||fibonacci(3)
|||fibonacci(2)
||||fibonacci(1)
||||fibonacci(0)
|||fibonacci(1)
||fibonacci(2)
|||fibonacci(1)
|||fibonacci(0)
|fibonacci(3)
||fibonacci(2)
|||fibonacci(1)
|||fibonacci(0)
||fibonacci(1)
Done
5

The code decorates the fibonacci function by printing each recursive call and the depth of the call stack.
The decoration is useful in showing the efficiency of the function, but it rewrites the function definition.

How to decorate a function without changing its code?

What if the decorations are temporary and should be removed later?
Go through the source codes of all decorated functions to remove the decorations?
When updating a piece of code, switch back and forth between original and decorated codes?

What about defining a new function that calls and decorates the original function?

def fibonacci(n):
    """Returns the Fibonacci number of order n."""
    return fibonacci(n - 1) + fibonacci(n - 2) if n > 1 else 1 if n == 1 else 0


def fibonacci_decorated(n):
    """Returns the Fibonacci number of order n."""
    global count, depth
    count += 1
    depth += 1
    print("{:>3}: {}fibonacci({!r})".format(count, "|" * depth, n))

    value = fibonacci(n)

    depth -= 1
    if depth == -1:  # recursion done
        print("Done")
        count = 0  # reset count for subsequent recursions
    return value


count, depth = 0, -1
for n in range(6):
    print(fibonacci_decorated(n))

  1: fibonacci(0)
Done
0
  1: fibonacci(1)
Done
1
  1: fibonacci(2)
Done
1
  1: fibonacci(3)
Done
2
  1: fibonacci(4)
Done
3
  1: fibonacci(5)
Done
5

We want fibonacci to call fibonacci_decorated instead.
What about renaming fibonacci_decorated to fibonacci?

fibonacci = fibonacci_decorated
count, depth = 0, -1
fibonacci_decorated(10)

(If you are faint-hearted, don’t run the above code.)

We want fibonacci_decorated to call the original fibonacci.

The solution is to capture the original fibonacci in a closure:

import functools


def print_function_call(f):
    """Decorate a recursive function to print the call stack.

    The decorator also keep tracks of the number and depth of a recursive call to print the call stack.

    Parameters
    ----------
    f: Calla76ble
        A recursive function.

    Returns
    -------
    Callable:j
        The decorated function that also prints the function call
        when called.
    """

    @functools.wraps(f)  # give wrapper the identity of f and more
    def wrapper(*args, **kwargs):
        nonlocal count, depth
        count += 1
        depth += 1
        call = "{}{}".format(f.__name__, argument_string(*args, **kwargs))
        print("{:>3}:{}{}".format(count, "|" * depth, call))

        value = f(*args, **kwargs)  # calls f

        depth -= 1
        if depth == -1:
            print("Done")
            count = 0
        return value

    count, depth = 0, -1
    return wrapper  # return the decorated function

print_function_call takes in f and returns wrapper, which captures and decorates f:

wrapper expects the same set of arguments for f,
returns the same value returned by f on the arguments, but
can execute additional codes before and after calling f to print the function call.

By redefining fibonacci as the returned wrapper, the original fibonacci captured by wrapper calls wrapper as desired.

def fibonacci(n):
    return fibonacci(n - 1) + fibonacci(n - 2) if n > 1 else 1 if n == 1 else 0


fibonacci = print_function_call(fibonacci)  # so original fibonnacci calls wrapper
fibonacci(5)

fibonacci(5)
|fibonacci(4)
||fibonacci(3)
|||fibonacci(2)
||||fibonacci(1)
||||fibonacci(0)
|||fibonacci(1)
||fibonacci(2)
|||fibonacci(1)
|||fibonacci(0)
|fibonacci(3)
||fibonacci(2)
|||fibonacci(1)
|||fibonacci(0)
||fibonacci(1)
Done

The redefinition does not change the original fibonacci captured by wrapper.

import inspect

for cell in fibonacci.__closure__:
    if callable(cell.cell_contents):
        print(inspect.getsource(cell.cell_contents))

def fibonacci(n):
    return fibonacci(n - 1) + fibonacci(n - 2) if n > 1 else 1 if n == 1 else 0

Python provides the syntatic sugar below to simplify the redefinition.

@print_function_call
def fibonacci(n):
    return fibonacci(n - 1) + fibonacci(n - 2) if n > 1 else 1 if n == 1 else 0


fibonacci(5)

fibonacci(5)
|fibonacci(4)
||fibonacci(3)
|||fibonacci(2)
||||fibonacci(1)
||||fibonacci(0)
|||fibonacci(1)
||fibonacci(2)
|||fibonacci(1)
|||fibonacci(0)
|fibonacci(3)
||fibonacci(2)
|||fibonacci(1)
|||fibonacci(0)
||fibonacci(1)
Done

There are many techniques used in the above decorator.

Why use a variable number of arguments in wrapper

To decorate any function with possibly different number of arguments.

Why decorate the wrapper with @functools.wraps(f)?

Ensures some attributes (such as __name__) of the wrapper function is the same as those of f.
Add useful attributes. E.g., __wrapped__ stores the original function so we can undo the decoration.

fibonacci, fibonacci_decorated = fibonacci.__wrapped__, fibonacci  # recover
print("original fibonacci:")
print(fibonacci(5))

fibonacci = fibonacci_decorated  # decorate
print("decorated fibonacci:")
print(fibonacci(5))

original fibonacci:
5
decorated fibonacci:
fibonacci(5)
|fibonacci(4)
||fibonacci(3)
|||fibonacci(2)
||||fibonacci(1)
||||fibonacci(0)
|||fibonacci(1)
||fibonacci(2)
|||fibonacci(1)
|||fibonacci(0)
|fibonacci(3)
||fibonacci(2)
|||fibonacci(1)
|||fibonacci(0)
||fibonacci(1)
Done
5

How to use decorator to improve recursion?

We can also use a decorator to make recursion more efficient by caching the return values.
cache is a dictionary where cache[n] stores the computed value of \(F_n\) to avoid redundant computations.

def caching(f):
    """Cache the return value of a function that takes a single argument.

    Parameters
    ----------
    f: Callable
        A function that takes a single argument.

    Returns
    -------
    Callable:
        The function same as f but has its return valued automatically cached
        when called. It has a method clear_cache to clear its cache.
    """

    @functools.wraps(f)
    def wrapper(n):
        if n not in cache:
            cache[n] = f(n)
        else:
            print("read from cache")
        return cache[n]

    cache = {}
    wrapper.clear_cache = lambda: cache.clear()  # add method to clear cache
    return wrapper


@print_function_call
@caching
def fibonacci(n):
    return fibonacci(n - 1) + fibonacci(n - 2) if n > 1 else 1 if n == 1 else 0

fibonacci(5)
fibonacci(5)
fibonacci.clear_cache()
fibonacci(5)

  1:fibonacci(5)
  2:|fibonacci(4)
  3:||fibonacci(3)
  4:|||fibonacci(2)
  5:||||fibonacci(1)
  6:||||fibonacci(0)
  7:|||fibonacci(1)
read from cache
  8:||fibonacci(2)
read from cache
  9:|fibonacci(3)
read from cache
Done
  1:fibonacci(5)
read from cache
Done
  1:fibonacci(5)
  2:|fibonacci(4)
  3:||fibonacci(3)
  4:|||fibonacci(2)
  5:||||fibonacci(1)
  6:||||fibonacci(0)
  7:|||fibonacci(1)
read from cache
  8:||fibonacci(2)
read from cache
  9:|fibonacci(3)
read from cache
Done

A method clear_cache is added to the wrapper to clear the cache.
lambda <argument list> : <expression>is called a lambda expression, which conveniently defines an anonymous function.

type(fibonacci.clear_cache), fibonacci.clear_cache.__name__

(function, '<lambda>')

Module¶

How to create a module?

To create a module, simply put the code in a python source file <module name>.py in

the current directory, or
a python site-packages directory in system path.

import sys

print(sys.path)

['/home/cs1302/21a/source/Lecture6', '/home/cs1302/my-conda-envs/jb/lib/python38.zip', '/home/cs1302/my-conda-envs/jb/lib/python3.8', '/home/cs1302/my-conda-envs/jb/lib/python3.8/lib-dynload', '', '/home/cs1302/.local/lib/python3.8/site-packages', '/home/cs1302/my-conda-envs/jb/lib/python3.8/site-packages', '/home/cs1302/my-conda-envs/jb/lib/python3.8/site-packages/IPython/extensions', '/home/cs1302/.ipython']

For example, recurtools.py in the current directory defines the module recurtools.

from IPython.display import Code

Code(filename="recurtools.py", language="python")

'''
Recursion Tools
===============

Contain tools to print the call stack and cache the return values
of a recursive function.
'''

import functools


def _argument_string(*args, **kwargs):
    """Return the string representation of the list of arguments."""
    return "({})".format(
        ", ".join(
            [
                *["{!r}".format(v) for v in args],  # arguments
                *[
                    "{}={!r}".format(k, v) for k, v in kwargs.items()
                ],  # keyword arguments
            ]
        )
    )


def print_function_call(f):
    """Decorate a recursive function to print the call stack.

    The decorator also keep tracks of the number and depth of a recursive call to print the call stack.

    Parameters
    ----------
    f: Callable
        A recursive function.

    Returns
    -------
    Callable:
        The decorated function that also prints the function call
        when called.

    Examples
    --------
    >>> @print_function_call
    ... def fibonacci(n):
    ...     return fibonacci(n - 1) + fibonacci(n - 2) if n > 1 else 1 if n == 1 else 0
    ...
    >>> fibonacci(5)
      1:fibonacci(5)
      2:|fibonacci(4)
      3:||fibonacci(3)
      4:|||fibonacci(2)
      5:||||fibonacci(1)
      6:||||fibonacci(0)
      7:|||fibonacci(1)
      8:||fibonacci(2)
      9:|||fibonacci(1)
     10:|||fibonacci(0)
     11:|fibonacci(3)
     12:||fibonacci(2)
     13:|||fibonacci(1)
     14:|||fibonacci(0)
     15:||fibonacci(1)
    Done
    5
    """

    @functools.wraps(f)  # give wrapper the identity of f and more
    def wrapper(*args, **kwargs):
        nonlocal count, depth
        count += 1
        depth += 1
        call = "{}{}".format(f.__name__, _argument_string(*args, **kwargs))
        print("{:>3}:{}{}".format(count, "|" * depth, call))

        value = f(*args, **kwargs)  # calls f

        depth -= 1
        if depth == -1:
            print("Done")
            count = 0
        return value

    count, depth = 0, -1
    return wrapper  # return the decorated function


def caching(f):
    """Cache the return value of a function that takes a single argument.

    Parameters
    ----------
    f: Callable
        A function that takes a single argument.

    Returns
    -------
    Callable:
        The function same as f but has its return valued automatically cached
        when called. It has a method clear_cache to clear its cache.

    Examples
    --------
    >>> @print_function_call
    ... @caching
    ... def fibonacci(n):
    ...     return fibonacci(n - 1) + fibonacci(n - 2) if n > 1 else 1 if n == 1 else 0
    ...
    >>> fibonacci(5)
      1:fibonacci(5)
      2:|fibonacci(4)
      3:||fibonacci(3)
      4:|||fibonacci(2)
      5:||||fibonacci(1)
      6:||||fibonacci(0)
      7:|||fibonacci(1)
    read from cache
      8:||fibonacci(2)
    read from cache
      9:|fibonacci(3)
    read from cache
    Done
    5
    >>> fibonacci(5)
      1:fibonacci(5)
    read from cache
    Done
    5
    >>> fibonacci.clear_cache()
    >>> fibonacci(5)
      1:fibonacci(5)
      2:|fibonacci(4)
      3:||fibonacci(3)
      4:|||fibonacci(2)
      5:||||fibonacci(1)
      6:||||fibonacci(0)
      7:|||fibonacci(1)
    read from cache
      8:||fibonacci(2)
    read from cache
      9:|fibonacci(3)
    read from cache
    Done
    5
    """

    @functools.wraps(f)
    def wrapper(n):
        if n not in cache:
            cache[n] = f(n)
        else:
            print("read from cache")
        return cache[n]

    cache = {}
    wrapper.clear_cache = lambda: cache.clear()  # add method to clear cache
    return wrapper

The module provides the decorators print_function_call and caching defined earlier.

import recurtools as rc


@rc.print_function_call
@rc.caching
def factorial(n):
    return factorial(n - 1) if n > 1 else 1

factorial(5)
factorial(5)
factorial.clear_cache()
factorial(5)

factorial(5)
|factorial(4)
||factorial(3)
|||factorial(2)
||||factorial(1)
Done
factorial(5)
read from cache
Done
factorial(5)
|factorial(4)
||factorial(3)
|||factorial(2)
||||factorial(1)
Done

CS1302

Decorator¶

Optional Arguments¶

Variable number of arguments¶

Decorator¶

Module¶