ThreadPoolExecutor Memory Blowup When You Submit a Million Futures

Overview

You have a million items to process, and a pool of 16 threads, so you loop and call executor.submit for each item. Memory climbs steadily, sometimes until the process is killed, even though only 16 tasks run at a time. The workers are not the problem. The queue and the futures are.

1. Why Memory Grows

ThreadPoolExecutor.submit never blocks. It places a work item in an unbounded internal queue and returns a Future. Submitting a million tasks creates a million work items, each holding the function, its arguments, and a Future. If you also keep every future in a list, those objects, and their eventual results, stay alive until you drop them.

from concurrent.futures import ThreadPoolExecutor

def fetch(item):
    ...

with ThreadPoolExecutor(max_workers=16) as pool:
    # All million work items are queued immediately.
    futures = [pool.submit(fetch, item) for item in items]
    results = [f.result() for f in futures]

This holds every argument, future, and result in memory at once. If items is itself a generator, you have also forced it to be fully materialized in the queue.

2. Fix One: Bound the Number of In-Flight Tasks

The core idea is backpressure: do not submit the next task until a slot frees up. A semaphore is a simple way to do it.

import threading
from concurrent.futures import ThreadPoolExecutor

MAX_IN_FLIGHT = 64


def run(items, fn, handle_result):
    sem = threading.BoundedSemaphore(MAX_IN_FLIGHT)

    def release(fut):
        sem.release()
        try:
            handle_result(fut.result())
        except Exception as exc:
            print(f"task failed: {exc}")

    with ThreadPoolExecutor(max_workers=16) as pool:
        for item in items:
            sem.acquire()  # blocks when too many tasks are pending
            fut = pool.submit(fn, item)
            fut.add_done_callback(release)

items can now be a lazy iterator, and memory is bounded by MAX_IN_FLIGHT plus whatever your result handler retains. Pick the bound relative to worker count, a small multiple is common, and measure.

Note that done callbacks run in the worker thread (or the submitting thread if the future is already done), so keep them short and thread-safe.

3. Fix Two: Process in Chunks

If you prefer simpler code, submit a batch, wait for it, then move on.

from itertools import islice


def batched(iterable, n):
    it = iter(iterable)
    while batch := list(islice(it, n)):
        yield batch


with ThreadPoolExecutor(max_workers=16) as pool:
    for batch in batched(items, 1000):
        for result in pool.map(fetch, batch):
            handle(result)

Python 3.12 added itertools.batched, which does the same thing. The downside of chunking is idle workers at the end of each batch while the slowest task finishes. The semaphore approach keeps workers busier.

Also be aware that Executor.map submits all inputs eagerly, so never call it on the full million-item iterable.

4. Fix Three: Consume Results as They Complete

Do not wait for the whole run to read results. With a bounded set of pending futures, use concurrent.futures.wait with FIRST_COMPLETED:

from concurrent.futures import ThreadPoolExecutor, wait, FIRST_COMPLETED

def run(items, fn, limit=64):
    pending = set()
    with ThreadPoolExecutor(max_workers=16) as pool:
        for item in items:
            pending.add(pool.submit(fn, item))
            if len(pending) >= limit:
                done, pending = wait(pending, return_when=FIRST_COMPLETED)
                for fut in done:
                    yield fut.result()
        for fut in pending:
            yield fut.result()

Each result is yielded and then released, so the caller can write it to disk or a database and let it go. This is a generator, so memory stays flat as long as the consumer does not accumulate results.

5. Check Whether Threads Are the Right Tool

Threads help with I/O-bound work. For CPU-bound Python code, the GIL limits parallelism, so use ProcessPoolExecutor, where the same unbounded-queue concern applies and arguments are pickled across processes. For very large I/O fan-out, an async approach with a semaphore-limited set of tasks may use less memory than thousands of threads. Also check whether the downstream service can tolerate the request rate you are generating.

6. Practical Checklist

• Never create all futures up front for a large input.
• Bound in-flight work, and stream results out.
• Do not store results you can write incrementally.
• Prefer passing small identifiers rather than large objects as task arguments.
• Measure peak memory with a representative input, using tools such as tracemalloc or your process monitor.

Conclusion

The executor accepts work faster than it processes it and queues the rest without limit. Add backpressure with a semaphore, batches, or a bounded wait loop, and consume results as they complete.

If you have a data pipeline whose memory use does not match its workload, I can help profile it and design a bounded processing model. Contact me.

Additional Resources

• Python docs: concurrent.futures
• Python docs: threading.BoundedSemaphore
• Python docs: tracemalloc