Processes: Best for CPU-Bound Tasks

In a GIL-enabled CPython interpreter, separate processes can run Python computations on different CPU cores. Each worker has its own memory, so arguments and results must be transferred. The cost can outweigh the benefit for small jobs.

ProcessPoolExecutor manages a pool of workers. Its map sends work to them and returns results in input order. Define worker functions at module level so child processes can import them.

Count Complete Words

Save the complete example as process_word_count.py and run python process_word_count.py in a terminal. The if __name__ == "__main__" guard keeps child processes from starting another pool when they import the file. This pattern works with spawn-based startup on Windows and macOS; do not paste this pool definition directly into a notebook and assume its functions are importable.

Split the text into words before making chunks. Splitting arbitrary character ranges could cut a word in half and produce incorrect counts.

from concurrent.futures import ProcessPoolExecutor
from collections import Counter
import time

def count_words(words):
    return Counter(word.lower() for word in words)

def merge(counters):
    total = Counter()
    for counter in counters:
        total.update(counter)
    return total

if __name__ == "__main__":
    words = ("To be or not to be " * 20_000).split()
    chunks = [words[i:i + 20_000] for i in range(0, len(words), 20_000)]
    start = time.perf_counter()
    with ProcessPoolExecutor(max_workers=2) as pool:
        partials = list(pool.map(count_words, chunks))
    total = merge(partials)
    print(total.most_common(3))
    print(f"Finished in {time.perf_counter() - start:.2f}s")

Output — the elapsed time varies:

[('to', 40000), ('be', 40000), ('or', 20000)]
Finished in ...s

Each chunk contains complete words. Workers count them independently; merge adds their Counters. Tied counts follow first appearance in the merged data, which is deterministic here because map retains chunk order.

Try it: time the same chunks with a direct loop. Then import ThreadPoolExecutor and use it in place of ProcessPoolExecutor. Compare counts before comparing time. In this example, Python computation, worker startup and data transfer all affect the result; no particular speedup is guaranteed.

With threads and processes established, asynchronous I/O offers another way to coordinate tasks that wait.