Guided project: build a traceable document table
On this page
A digital-humanities researcher wants to inspect the vocabulary and completeness of catalogue titles before beginning NLP. Build a document table that preserves source object IDs, raw titles and normalized representations. The source is the same 100-object museum snapshot; you are analyzing titles, not artwork images or full catalogue descriptions.
State the document contract
Define one title record per source object ID. Check ID uniqueness and distinguish missing, empty and whitespace-only text. Preserve raw titles. Two different objects may share a title, so text equality alone cannot establish duplicate objects.
Normalize and measure
Create a comparison representation using NFKC Unicode normalization, case folding and whitespace normalization. Explain the reason for each change and retain the original text. Measure normalized character length and a simple token count using contiguous Unicode letter runs. State that this regular expression is an illustrative representation, not a language-independent linguistic tokenizer.
Calculate token occurrences and vocabulary size. Produce a table of repeated normalized titles with IDs, rather than deleting them automatically. Distinguish occurrence frequency from document frequency. A common token can appear repeatedly in one title and across many titles; those quantities answer different questions.
Investigate limitations
Inspect at least five raw/normalized pairs and one title containing punctuation, accents or non-English text if such a case exists in the selected snapshot. If no such record is present, demonstrate the transformation on a separately labelled invented fixture; do not claim it was observed in the museum data.
Create a horizontal bar display of the ten most frequent tokens and label it as a descriptive view of selected titles. Do not claim that it measures artistic importance, language prevalence or topic quality. Stopword removal and language detection would require additional decisions and verification.
Deliver and transfer
Export document features, token counts, repeated-title report, audit and plot. Acceptance evidence includes 100 source IDs, unchanged raw titles, one explicit missingness flag per document, and a documented normalization/tokenization policy. Verify one token count by hand. Reconcile the sum of per-document token counts with total occurrences.
For transfer, compute document frequency for a chosen token and compare it with its occurrence count. Propose a future evaluation split grouped by source object, parent document or author when appropriate. Explain why related or near-duplicate content crossing a split can make NLP evaluation optimistic.
Begin the experiment
Run from the extracted experiment folder, or use the notebook’s first path-setup cell. Inspect the source before implementing the analysis.
from pathlib import Path
import pandas as pd
source = Path('data/teaching/museum.csv')
records = pd.read_csv(source)
print(records.shape)
print(records.head())Check your work
Attempt the project first. Then compare with the complete worked solution. Preserve your own implementation and explain differences.