Threads and concurrency basics

Concurrency means doing several pieces of work at the same time. A thread is one path of running code; a program can have several. In Python interviews the questions almost always start with the GIL, then move to the practical side: when do threads help, how do you avoid race conditions (bugs that depend on which thread runs first), and how do you run a test suite (the whole set of tests) in parallel without tests getting in each other's way?

Step 1: the GIL

CPython (the standard Python interpreter, the program that runs your code) has a Global Interpreter Lock (GIL): only one thread can run Python bytecode (the instructions Python compiles your code into) at a time. It keeps the interpreter's own data, such as reference counts (the number of references to each object), simple and safe. Two things follow from it:

• Threads do not make CPU-heavy Python code faster across several cores.
• Threads do help with waiting. A thread that waits for the network, the disk or time.sleep releases the GIL, so other threads run in the meantime.

Python 3.13 added an experimental build without the GIL; the standard 3.12 interpreter always has it.

Step 2: threads, processes or asyncio?

• Threads (threading, ThreadPoolExecutor, a pool of reusable threads) suit I/O-bound work, meaning work that mostly waits for input or output, such as many API calls (web requests) while preparing test data. Threads share data easily, so shared data needs protection.
• Processes (multiprocessing, ProcessPoolExecutor; a process is a separate running copy of Python) suit CPU-bound work (work that mostly calculates). Each process has its own interpreter and its own GIL, so all cores are used, but data must be passed between them.
• asyncio runs many tasks on one thread; each task pauses at await while it waits. Calling an async def function without await only creates a coroutine object (a paused piece of work that has not run), which is a classic bug.

import time
from concurrent.futures import ThreadPoolExecutor

def fake_api_call(user_id):
    time.sleep(0.2)                  # waiting releases the GIL
    return f"user-{user_id}"

start = time.perf_counter()
with ThreadPoolExecutor(max_workers=5) as pool:
    results = list(pool.map(fake_api_call, range(5)))
elapsed = time.perf_counter() - start
print(results)
print("faster than one by one:", elapsed < 0.8)

with ThreadPoolExecutor(max_workers=2) as pool:
    future = pool.submit(pow, 2, 10)
    print(future.result())

['user-0', 'user-1', 'user-2', 'user-3', 'user-4']
faster than one by one: True
1024

pool.map returns results in input order, and leaving the with block waits for every task. future.result() waits for one task and raises its exception again if it failed.

Step 3: race conditions despite the GIL

The GIL protects the interpreter, not your data. Python can switch threads between two bytecode steps, so total = total + amount can lose updates when two threads mix their reads and writes (one thread reads the old total, another writes a new one, then the first overwrites it). This is a race condition. "Check, then act" code such as if key not in cache: cache[key] = load() has the same problem. Guard shared updates with a threading.Lock (a rule that lets one thread in at a time) in a with block, pass work between threads through a queue.Queue (a waiting line that is safe for threads), or avoid sharing by letting each task return its own result.

import threading
from concurrent.futures import ThreadPoolExecutor

lock = threading.Lock()
shared = {"done": 0}

def process(batch):
    local = 0
    for _ in batch:
        local += 1                    # private to this task: no race
    with lock:                        # one thread at a time
        shared["done"] += local
    return local

batches = [range(1000)] * 8
with ThreadPoolExecutor(max_workers=4) as pool:
    per_task = list(pool.map(process, batches))
print(sum(per_task), shared["done"])

local_data = threading.local()
local_data.browser = "chrome-main"
def show():
    print("worker sees:", getattr(local_data, "browser", "nothing"))
t = threading.Thread(target=show)
t.start()
t.join()

8000 8000
worker sees: nothing

threading.local() gives each thread its own attributes, so the worker does not see the main thread's value. The other classic risk is a deadlock: two threads each hold one lock and wait for the other's. Take locks in a fixed order and hold them briefly.

Step 4: running tests in parallel

The usual tool is pytest-xdist (a plugin for pytest, the usual Python test tool): pytest -n 4 starts four worker processes and shares the tests between them, so the GIL does not matter. The workers share no Python memory, but they do share everything outside Python: test accounts, database rows, files and ports. So give each test its own browser in a function-scoped fixture (a pytest function that prepares things and runs once per test), create unique data per test (or per worker with the worker_id fixture), write files to tmp_path (a fresh temporary folder pytest gives each test), and never depend on test order.

@pytest.fixture
def driver():
    d = webdriver.Chrome()
    yield d
    d.quit()          # runs even when the test fails

How to answer in an interview

Start with the GIL in one sentence, then give the rule of thumb: "Threads for waiting, processes for CPU work, asyncio for very many network calls." Then add the warning: "The GIL does not make my code thread-safe, so I lock shared updates or avoid sharing." For parallel tests, explain that xdist uses processes and that outside resources such as accounts and files still need to be unique per test.

Common mistakes

• Saying the GIL makes Python code thread-safe.
• Using threads to speed up CPU-heavy work.
• Forgetting await in async code.
• Keeping one driver in a module-level variable.
• Letting parallel tests use the same test account or file.

Recap

The GIL lets one thread run Python code at a time, so threads help with waiting and processes help with CPU work. Shared data still needs a lock, a queue, or no sharing at all. pytest-xdist runs tests in separate processes, so make browsers, data and files unique per test.