What is threading in Python?

Definition of threading in Python

Threading is a mechanism that allows multiple instruction flows (called threads) to execute simultaneously within a single process. In Python, the threading module from the standard library provides all the tools needed to create, manage, and synchronize threads. If you want to master concurrent programming in Python, our comprehensive Python course will guide you step by step through these advanced concepts.

A thread (or thread of execution) is the smallest unit of execution that an operating system can schedule. Unlike processes, threads within the same program share the same memory space, which makes them lightweight and fast to create, but also more delicate to manage due to potential concurrent data access issues.

Why use threading?

Threading is particularly well-suited in the following situations:

• I/O-bound operations: HTTP requests, file reading/writing, database interactions
• Graphical interfaces: maintaining interface responsiveness during long-running tasks
• Servers: handling multiple client connections simultaneously
• Background tasks: logging, monitoring, periodic updates

On the other hand, for CPU-bound tasks (intensive computations), you should instead turn to the multiprocessing module which bypasses the GIL by using separate processes.

Creating a thread in Python

Method 1: with a target function

The simplest way to create a thread is to pass a function as the target argument to the threading.Thread constructor:

import threading
import time

def download_file(filename):
    print(f"Starting download of {filename}...")
    time.sleep(2)  # Simulates a download
    print(f"Download of {filename} completed!")

# Creating threads
thread1 = threading.Thread(target=download_file, args=("image.png",))
thread2 = threading.Thread(target=download_file, args=("video.mp4",))

# Starting threads
thread1.start()
thread2.start()

# Waiting for threads to finish
thread1.join()
thread2.join()

print("All downloads are completed.")

In this example, both downloads execute in parallel instead of sequentially. The total execution time will be approximately 2 seconds instead of 4 seconds sequentially. Notice the use of f-string to format the output messages.

Method 2: by inheriting from Thread

You can also create a thread by defining a class that inherits from threading.Thread and overriding the run() method:

import threading
import time

class MyThread(threading.Thread):
    """Custom thread to perform a task."""
    
    def __init__(self, name, duration):
        super().__init__()
        self.name = name
        self.duration = duration
    
    def run(self):
        print(f"Thread '{self.name}' started.")
        time.sleep(self.duration)
        print(f"Thread '{self.name}' finished after {self.duration}s.")

# Creation and start
t1 = MyThread("Task A", 3)
t2 = MyThread("Task B", 1)

t1.start()
t2.start()

t1.join()
t2.join()

print("All tasks are completed.")

This approach is ideal when you need to encapsulate complex logic inside the thread, taking advantage of object-oriented programming. Note the use of a docstring to document the class.

Thread synchronization

When multiple threads access the same data, concurrency problems (race conditions) can occur. Python provides several synchronization mechanisms to avoid these issues.

The Lock

A Lock is the simplest synchronization mechanism. It ensures that only one thread at a time can execute a critical section:

import threading

counter = 0
lock = threading.Lock()

def increment(n):
    global counter
    for _ in range(n):
        lock.acquire()
        try:
            counter += 1
        finally:
            lock.release()

# Creating two threads
t1 = threading.Thread(target=increment, args=(100000,))
t2 = threading.Thread(target=increment, args=(100000,))

t1.start()
t2.start()
t1.join()
t2.join()

print(f"Final counter: {counter}")  # Always 200000

Without the lock, the final result could be less than 200000 because both threads could read and write the counter variable at the same time. Note the use of the global keyword to modify the variable defined outside the function.

Lock with context manager

The with syntax makes using locks much cleaner and safer:

import threading

counter = 0
lock = threading.Lock()

def increment(n):
    global counter
    for _ in range(n):
        with lock:
            counter += 1

threads = []
for i in range(5):
    t = threading.Thread(target=increment, args=(50000,))
    threads.append(t)
    t.start()

for t in threads:
    t.join()

print(f"Final counter: {counter}")  # Always 250000

This approach is recommended because it guarantees that the lock will be released automatically, even if an exception occurs in the protected block. We use a list here to store the created threads.

RLock (reentrant lock)

An RLock (Reentrant Lock) can be acquired multiple times by the same thread without causing a deadlock:

import threading

lock = threading.RLock()

def outer_function():
    with lock:
        print("Lock acquired in outer_function")
        inner_function()

def inner_function():
    with lock:
        print("Lock acquired in inner_function")

t = threading.Thread(target=outer_function)
t.start()
t.join()

With a regular Lock, this code would cause a deadlock because the thread would try to acquire a lock it already holds. The RLock solves this problem by counting the number of acquisitions.

Events

An Event allows a thread to signal other threads that an event has occurred:

import threading
import time

event = threading.Event()

def server():
    print("Server: initializing...")
    time.sleep(3)
    print("Server: ready!")
    event.set()  # Signals that the server is ready

def client(name):
    print(f"Client {name}: waiting for server...")
    event.wait()  # Waits for the event to be signaled
    print(f"Client {name}: connected to server!")

# Starting server and clients
threading.Thread(target=server).start()
threading.Thread(target=client, args=("Alice",)).start()
threading.Thread(target=client, args=("Bob",)).start()

The clients wait for the server to be ready before connecting. The event serves as a synchronization signal between threads.

Daemon threads

A daemon thread is a thread that runs in the background and terminates automatically when all non-daemon threads have finished:

import threading
import time

def background_task():
    while True:
        print("Monitoring in progress...")
        time.sleep(1)

# Daemon thread: terminates when the main program stops
monitor = threading.Thread(target=background_task, daemon=True)
monitor.start()

# Main program
print("Main program running...")
time.sleep(3)
print("Main program finished.")
# The daemon thread terminates automatically here

Daemon threads are very useful for monitoring tasks, logging, or any processing that should run as long as the program is active, without preventing its shutdown.

ThreadPoolExecutor: modern threading

Since Python 3.2, the concurrent.futures module offers a high-level interface for managing thread pools. This is the recommended way to work with threading in modern projects:

from concurrent.futures import ThreadPoolExecutor, as_completed
import time

def process_request(url):
    """Simulates processing an HTTP request."""
    print(f"Request to {url}...")
    time.sleep(1)  # Simulates network latency
    return f"Response from {url}"

urls = [
    "https://api.example.com/users",
    "https://api.example.com/posts",
    "https://api.example.com/comments",
    "https://api.example.com/albums",
    "https://api.example.com/photos",
]

# Using a pool of 3 maximum threads
with ThreadPoolExecutor(max_workers=3) as executor:
    futures = {executor.submit(process_request, url): url for url in urls}
    
    for future in as_completed(futures):
        url = futures[future]
        try:
            result = future.result()
            print(f"Success: {result}")
        except Exception as e:
            print(f"Error for {url}: {e}")

print("All requests have been processed.")

The ThreadPoolExecutor automatically handles thread creation, reuse, and destruction. We use a dict comprehension here to associate each future with its original URL.

Best practices

Here are the best practices to follow when using threading in Python:

• Prefer ThreadPoolExecutor: use concurrent.futures.ThreadPoolExecutor rather than manually creating threads for cleaner and safer code.
• Minimize shared data: the more data you share between threads, the higher the risk of concurrency bugs. Pass data as arguments and retrieve results.
• Always use locks: if you must share mutable data between threads, systematically protect it with a Lock or RLock.
• Avoid deadlocks: always acquire locks in the same order throughout your program, and use timeouts when possible.
• Use daemon threads with caution: daemon threads are terminated abruptly. Do not entrust them with critical tasks requiring proper cleanup.
• Handle exceptions: exceptions in a thread do not propagate to the main thread. Always catch them within the thread itself.
• Limit the number of threads: creating too many threads incurs scheduling overhead. Use a reasonably sized pool.

Frequently asked questions

Threading uses threads within a single process, sharing the same memory space. It is limited by the GIL for CPU-bound tasks but excellent for I/O-bound operations. Multiprocessing creates separate processes with their own memory spaces, thus bypassing the GIL and offering true parallelism for intensive computations. The choice depends on the nature of your task: I/O-bound → threading, CPU-bound → multiprocessing.

The GIL (Global Interpreter Lock) prevents simultaneous execution of Python bytecode by multiple threads across multiple cores. However, the GIL is released during I/O operations (network, files, etc.) and when calling certain C libraries (like NumPy). Thus, threading remains very performant for I/O-bound tasks. Note that since Python 3.13, an experimental "free-threaded" mode is under development to eventually remove the GIL.

To avoid race conditions, you must protect access to shared resources using synchronization mechanisms: Lock, RLock, Semaphore, or Condition. The best approach is to minimize shared data between threads by passing data through arguments and retrieving results via Future objects (with ThreadPoolExecutor) or queues (queue.Queue).

Threading is an advanced topic that requires a solid foundation in Python. We recommend starting by mastering the fundamentals of the language (functions, classes, exception handling) before tackling concurrency. Our comprehensive Python course covers these concepts progressively, from the basics to advanced topics like threading, allowing you to build solid and directly applicable skills.