Python 'with' Statement

Introduction

There are situations when resources need to be properly managed, such as, thread locks, file handles, network sockets, etc. For example, if a lock is acquired, it needs to be properly released. Similarly, if a file is opened, it needs to be properly closed.

The resource management is typically done using the try-except-finally statements, which can result in lot of boilerplate code.

The following is a simple python program named LocknFileOne.py that demonstrates resource management using the try-except-finally statements:

#
# Name: LocknFileOne.py
#

import threading
import time

# ----- Global variables -----

seqno = 0
test_file = "/tmp/test_file"
lock = threading.Lock()

# ----- Method get_next -----

def get_next():
    global seqno
    lock.acquire()
    seqno += 1
    seq = seqno
    lock.release()
    return seq

# ----- Method read_test_file -----

def read_test_file():
    name = threading.currentThread().getName()
    fp = None
    try:
        fp = open(test_file)
        fp.readline()
    except IOError:
        print "LocknFileOne:: <" + name + "> :: File " + test_file + " not found !!!"
    finally:
        if fp is not None:
            fp.close()

# ----- Method thread_func -----

def thread_func():
    name = threading.currentThread().getName()
    for i in range(1, 6):
        print "LocknFileOne:: <" + name + "> :: seqno = " + str(get_next())
        time.sleep(1)
    read_test_file()
    return

# ----- Main -----

if __name__ == '__main__':
    thr1 = threading.Thread(name="Thread-1", target=thread_func)
    thr2 = threading.Thread(name="Thread-2", target=thread_func)
    thr1.start()
    thr2.start()

Executing LocknFileOne.py results in the following output:

Output.1

LocknFileOne:: <Thread-1> :: seqno = 1
LocknFileOne:: <Thread-2> :: seqno = 2
LocknFileOne:: <Thread-2> :: seqno = 3
LocknFileOne:: <Thread-1> :: seqno = 4
LocknFileOne:: <Thread-1> :: seqno = 5
LocknFileOne:: <Thread-2> :: seqno = 6
LocknFileOne:: <Thread-1> :: seqno = 7
LocknFileOne:: <Thread-2> :: seqno = 8
LocknFileOne:: <Thread-1> :: seqno = 9
LocknFileOne:: <Thread-2> :: seqno = 10
LocknFileOne:: <Thread-1> :: File /tmp/test_file not found !!!
LocknFileOne:: <Thread-2> :: File /tmp/test_file not found !!!

By using Python's with statement, one can streamline and automate resource management.

The following is a simple python program named LocknFileTwo.py that demonstrates resource management using the with statement:

#
# Name: LocknFileTwo.py
#

import threading
import time

# ----- Global variables -----

seqno = 0
test_file = "/tmp/test_file"
lock = threading.Lock()

# ----- Method get_next -----

def get_next():
    global seqno
    with lock:
        seqno += 1
        seq = seqno
    return seq

# ----- Method read_test_file -----

def read_test_file():
    name = threading.currentThread().getName()
    try:
        with open(test_file) as fp:
            fp.readline()
    except IOError:
        print "LocknFileTwo:: <" + name + "> :: File " + test_file + " not found !!!"

# ----- Method thread_func -----

def thread_func():
    name = threading.currentThread().getName()
    for i in range(1, 6):
        print "LocknFileTwo:: <" + name + "> :: seqno = " + str(get_next())
        time.sleep(1)
    read_test_file()
    return

# ----- Main -----

if __name__ == '__main__':
    thr1 = threading.Thread(name="Thread-1", target=thread_func)
    thr2 = threading.Thread(name="Thread-2", target=thread_func)
    thr1.start()
    thr2.start()

Executing LocknFileTwo.py results in the following output:

Output.2

LocknFileTwo:: <Thread-1> :: seqno = 1
LocknFileTwo:: <Thread-2> :: seqno = 2
LocknFileTwo:: <Thread-1> :: seqno = 3
LocknFileTwo:: <Thread-2> :: seqno = 4
LocknFileTwo:: <Thread-1> :: seqno = 5
LocknFileTwo:: <Thread-2> :: seqno = 6
LocknFileTwo:: <Thread-1> :: seqno = 7
LocknFileTwo:: <Thread-2> :: seqno = 8
LocknFileTwo:: <Thread-1> :: seqno = 9
LocknFileTwo:: <Thread-2> :: seqno = 10
LocknFileTwo:: <Thread-1> :: File /tmp/test_file not found !!!
LocknFileTwo:: <Thread-2> :: File /tmp/test_file not found !!!

The general usage format of the with statement is as follows:

with <expression> [as <variable>]:
code-block

If the <expression> returns a value, then we need the optional as keyword, followed by a <variable> to which the return value can be assigned.

Ex:

with lock:
seqno += 1

In the above example, before executing the expression seqno += 1, Python automatically acquires the thread lock (internally calls lock.acquire()). Once the expression seqno += 1 is executed, Python automatically releases the thread lock (internally calls lock.release()).

Ex:

with open(test_file) as fp:
fp.readline()

In the above example, Python executes the expression open(test_file) and assigns the return value (file pointer) to the variable fp. After executing fp.readline(), Python automatically closes the file (internally calls fp.close()) regardless of whether an exception is raised or not.

Seems like some kind of a MAGIC being performed by Python here ???

When the with statement is executed, the following sequence of events happens:

• Python evaluates the <expression>

• The <expression> could be an object (like lock) or returns an object (like open(test_file)). Let us refer to this value (lock or fp) as the OBJECT

• Python automatically invokes the method __enter()__ on the OBJECT

• Python executes the code-block

• Regardless of whether or not an exception is raised, Python automatically invokes the method __exit()__ on the OBJECT

This is often referred to as the context management protocol in Python documentation.

The OBJECT implementing the two methods __enter__() and __exit__() is referred to as a context manager.

References

Python Quick Notes :: Part - 1

Python Quick Notes :: Part - 2

Python Quick Notes :: Part - 3

Python Quick Notes :: Part - 4

Python Quick Notes :: Part - 5

Python Quick Notes :: Part - 6