What is the with statement in Python?

Definition of the with statement in Python

The with statement in Python is a powerful mechanism that allows you to automatically manage resources such as files, network connections, or locks. It guarantees that a resource is properly initialized before use and cleanly released after use, even if an error occurs during the execution of the code block.

Concretely, with relies on the concept of a context manager. When you use this statement, Python automatically calls two special methods: __enter__ when entering the block and __exit__ when leaving it, whether the exit is normal or caused by an exception. This mechanism makes your code safer, more readable, and more concise.

If you want to master Python in depth, including advanced concepts like context managers, we recommend following our complete Python course available on Believemy.

Basic syntax of with

The syntax of the with statement is simple and elegant:

with expression as variable:
    # code block using the variable
    pass

Here is how it works in detail:

• expression: an expression that returns an object supporting the context manager protocol (i.e., having the __enter__ and __exit__ methods).
• as variable: (optional) assigns the value returned by __enter__ to a variable that you can use within the block.
• The indented block: the code that executes within the managed context.

At the end of the block (or in case of an exception), the __exit__ method is automatically called, ensuring the clean release of the resource.

Why use with rather than try/finally?

Before the introduction of with (Python 2.5), resource management systematically relied on try/finally blocks. Let's compare both approaches:

Without with (classic approach)

file = open("data.txt", "r")
try:
    content = file.read()
    print(content)
finally:
    file.close()

With with (modern approach)

with open("data.txt", "r") as file:
    content = file.read()
    print(content)

The difference is striking. With with, you no longer need to think about closing the file manually. The file is automatically closed when leaving the block, even if an exception is raised. The code is shorter, more readable, and less error-prone.

Practical examples

Reading and writing files

The most common use of with is file manipulation. Here is how to read and write files safely:

# Writing to a file
with open("journal.txt", "w") as file:
    file.write("First line\n")
    file.write("Second line\n")
    file.write("Third line\n")

# Reading the file
with open("journal.txt", "r") as file:
    for line in file:
        print(line.strip())

In this example, even if an error occurs during writing or reading, the file will be properly closed thanks to with.

Managing multiple resources simultaneously

Since Python 3.1, you can manage multiple context managers in a single with statement:

# Copy the content of one file to another
with open("source.txt", "r") as source, open("destination.txt", "w") as destination:
    for line in source:
        destination.write(line.upper())

This syntax is equivalent to nesting two with blocks, but it is much more compact:

# Equivalent with nested blocks
with open("source.txt", "r") as source:
    with open("destination.txt", "w") as destination:
        for line in source:
            destination.write(line.upper())

Using with locks (threading)

The with statement is particularly useful in concurrent programming for managing locks safely:

import threading

lock = threading.Lock()
counter = 0

def increment():
    global counter
    with lock:
        # The lock is automatically acquired here
        current_value = counter
        counter = current_value + 1
    # The lock is automatically released here

# Create and start multiple threads
threads = [threading.Thread(target=increment) for _ in range(100)]
for t in threads:
    t.start()
for t in threads:
    t.join()

print(f"Final counter: {counter}")  # Prints: Final counter: 100

Notice that in the case of locks, the as clause is not necessary because we do not need a return variable.

Creating your own context managers

You can create your own context managers in two ways: by using a class with special methods, or by using the contextlib.contextmanager decorator.

Method 1: with a class

To create a context manager from a class, you need to implement the __enter__ and __exit__ methods:

class Timer:
    """Context manager to measure the execution time of a code block."""
    
    def __init__(self, name="Block"):
        self.name = name
        self.start = None
    
    def __enter__(self):
        import time
        self.start = time.time()
        print(f"⏱️ Starting measurement for '{self.name}'")
        return self  # Value assigned to the variable after 'as'
    
    def __exit__(self, exc_type, exc_value, exc_traceback):
        import time
        duration = time.time() - self.start
        print(f"⏱️ '{self.name}' completed in {duration:.4f} seconds")
        return False  # Do not suppress exceptions


# Usage
with Timer("Intensive calculation") as timer:
    total = sum(range(1_000_000))
    print(f"Result: {total}")

The __exit__ method receives three parameters related to a potential exception:

• exc_type: the type of the exception (or None if no exception).
• exc_value: the value of the exception.
• exc_traceback: the traceback of the exception.

If __exit__ returns True, the exception is suppressed. If it returns False (or None), the exception is propagated normally.

Method 2: with contextlib and a generator

The contextlib module offers a decorator that allows you to create a context manager from a simple generator function using yield:

from contextlib import contextmanager

@contextmanager
def temporary_directory(prefix="tmp"):
    import tempfile
    import shutil
    
    # Entry code (__enter__)
    path = tempfile.mkdtemp(prefix=prefix)
    print(f"📁 Temporary directory created: {path}")
    
    try:
        yield path  # Value assigned to the variable after 'as'
    finally:
        # Exit code (__exit__)
        shutil.rmtree(path)
        print(f"🗑️ Temporary directory deleted: {path}")


# Usage
with temporary_directory("my_app_") as folder:
    print(f"Working in: {folder}")
    # The folder exists here
# The folder is automatically deleted here

The context manager protocol in detail

To better understand how with works internally, here is the precise execution flow:

Common built-in context managers

Python provides many built-in context managers that you can use directly with with:

Exception suppression with contextlib.suppress

from contextlib import suppress

# Ignore a specific error
with suppress(FileNotFoundError):
    import os
    os.remove("nonexistent_file.txt")
    # No error raised if the file does not exist

print("The program continues normally")

Redirecting standard output

from contextlib import redirect_stdout
import io

# Capture print output into a string
buffer = io.StringIO()
with redirect_stdout(buffer):
    print("This text is captured")
    print("This one too")

result = buffer.getvalue()
print(f"Captured text: {result}")  # Prints the captured text

Temporary directory change

import os
from contextlib import contextmanager

@contextmanager
def change_directory(path):
    previous = os.getcwd()
    os.chdir(path)
    try:
        yield
    finally:
        os.chdir(previous)

# Usage
print(f"Current directory: {os.getcwd()}")
with change_directory("/tmp"):
    print(f"Temporary directory: {os.getcwd()}")
print(f"Back to directory: {os.getcwd()}")

Asynchronous context managers (async with)

Since Python 3.5, it is possible to use with in an asynchronous context thanks to async with. The corresponding methods are __aenter__ and __aexit__:

import asyncio

class HTTPSession:
    """Simplified example of an asynchronous context manager."""
    
    async def __aenter__(self):
        print("🌐 Opening HTTP session")
        # Simulate opening a connection
        await asyncio.sleep(0.1)
        return self
    
    async def __aexit__(self, exc_type, exc_value, exc_traceback):
        print("🌐 Closing HTTP session")
        # Simulate closing the connection
        await asyncio.sleep(0.1)
        return False
    
    async def get(self, url):
        print(f"📡 GET request to {url}")
        await asyncio.sleep(0.2)
        return {"status": 200, "data": "OK"}


async def main():
    async with HTTPSession() as session:
        response = await session.get("https://api.example.com/data")
        print(f"Response: {response}")

asyncio.run(main())

Best practices

Here are the best practices to follow when using the with statement in Python:

• Always prefer with for files: never use open() without with. This is the recommended way by the Python community and official documentation.
• Use with for any resource requiring cleanup: database connections, network sockets, locks, temporary files, etc.
• Keep with blocks short: release resources as early as possible by only including code that actually needs the resource within the block.
• Prefer @contextmanager for simple cases: if your context manager is simple, the decorator is more concise than a full class.
• Don't forget try/finally in @contextmanager: without it, the cleanup code will not execute in case of an exception.
• Document your context managers: use docstring to explain what your context manager does, what it returns, and what exceptions it can handle.
• Return False in __exit__ by default: only suppress exceptions if you have a very good reason to do so.

Frequently asked questions

Both approaches allow you to guarantee the execution of cleanup code, but with is more concise and less error-prone. With try/finally, you must manually manage the acquisition and release of the resource, which can lead to oversights. The with statement encapsulates this logic into a reusable object (the context manager), making the code cleaner and more maintainable. Additionally, the context manager can be shared across multiple parts of your code, promoting reusability.

Yes, the as clause is optional. It is only useful when you need the value returned by __enter__. For example, with a lock (threading.Lock), you generally do not need the variable: with lock: is sufficient. However, with open(), the as clause is essential to access the file object.

Yes, you can nest with blocks or use the compact syntax with commas: with open("a.txt") as a, open("b.txt") as b:. Since Python 3.10, you can also use parentheses for better readability across multiple lines: with (open("a.txt") as a, open("b.txt") as b):. Both approaches are functionally equivalent.

The with statement is one of the essential Python features that distinguish amateur code from professional code. To master this statement and the entire Python language, we recommend following our Python course on Believemy. You will learn not only the basics of the language, but also advanced concepts like context managers, asynchronous programming, and development best practices.