Project: Parallel Word Counter

Combine files, regular expressions and a process pool to count words across several documents. The pool distributes files among two workers; it does not dedicate one CPU core to every file.

Define the Counting Rule

For this project, a word is a sequence matched by \b\w+\b after lowercasing. This includes Unicode letters, digits and underscores, and separates punctuation. It is a simple tokenization rule, not a linguistic definition for every language. The : Path and -> Counter annotations document the argument and return types; they do not validate them automatically.

Save the complete program as parallel_word_counter.py in a new practice folder. It creates three demonstration files, overwriting those same files if run again. The explicit file list keeps unrelated files out of the results.

from concurrent.futures import ProcessPoolExecutor
from pathlib import Path
from collections import Counter
import re
import time

def count_file(path: Path) -> Counter:
    text = path.read_text(encoding="utf-8")
    return Counter(re.findall(r"\b\w+\b", text.lower()))

def merge(counters):
    total = Counter()
    for counter in counters:
        total.update(counter)
    return total

if __name__ == "__main__":
    demo_dir = Path("demo_corpus")
    demo_dir.mkdir(exist_ok=True)
    (demo_dir / "a.txt").write_text("To be, or not to be.\n", encoding="utf-8")
    (demo_dir / "b.txt").write_text("To be! To learn. To build.\n", encoding="utf-8")
    (demo_dir / "c.txt").write_text("Build, learn, repeat.\n", encoding="utf-8")
    files = [demo_dir / name for name in ("a.txt", "b.txt", "c.txt")]
    start = time.perf_counter()
    with ProcessPoolExecutor(max_workers=2) as pool:
        results = list(pool.map(count_file, files))
    total = merge(results)
    print("Top 5 words:", total.most_common(5))
    print(f"Completed in {time.perf_counter() - start:.2f}s")

Output — timing varies:

Top 5 words: [('to', 5), ('be', 3), ('learn', 2), ('build', 2), ('or', 1)]
Completed in ...s

The words be,, be. and be! all become be. Counts for learn and build tie; their order follows first occurrence as the file results are merged. or is the first of the words with count 1.

Check Before Measuring

Count the same files sequentially and assert that the merged Counter is identical. Test an empty file, mixed capitalization and punctuation. A decoding error should be reported rather than silently discarding text.

Change it: add more documents explicitly to the file list. Measure sequential and process-pool versions using the same inputs. These tiny files may run faster sequentially because worker startup costs more than counting; performance is a result to investigate, not a promised benefit.

The next chapter uses similar records and text-processing ideas with data obtained through HTTP and HTML parsing.