Prepare documents, text and multimodal records
On this page
A document collection is a dataset whose observations happen to contain language. Before tokenization or modelling, you need identifiers, provenance, missingness rules and a clear connection between text and labels. This chapter prepares data for the later NLP book. It does not assume that cleaning all punctuation or removing common words is always beneficial.
Define the document and preserve the raw text
A document might be a title, paragraph, review, email or full book. These units produce different counts and different dependence structures. Several paragraphs from the same book should not automatically be treated as independent documents when evaluating performance on unseen books. Keep source identifiers and any parent-document or author grouping needed for a split.
The museum project uses titles from a saved, CC0 metadata snapshot. A title is not a complete description of an artwork, and title vocabulary is not evidence of a work’s historical influence. Store raw text separately from any normalized representation so that transformations remain inspectable.
Distinguish missing, empty and whitespace-only values
import pandas as pd
text = pd.Series(['Café', ' ', None, 'CAFE\u0301', 'Red House'], dtype='string')
missing = text.isna()
blank = text.fillna('').str.strip().eq('')
print(missing.tolist(), blank.tolist())
assert missing.sum() == 1 and blank.sum() == 2[False, False, True, False, False] [False, True, True, False, False]
The empty-looking entries have different origins even if you later exclude both from a text analysis. Record the exclusion rule. Do not convert a missing value into the literal word “None” or “nan” and then count it as language.
Normalize only for a stated purpose
Unicode can encode visually similar text in different ways. Normalization can make particular comparisons more consistent. Case folding supports case-insensitive comparison; whitespace normalization can remove layout variation. These operations can also change information relevant to a task. Keep accents, punctuation or original case when they carry meaning.
normalized = text.str.normalize('NFKC').str.casefold().str.replace(r'\s+', ' ', regex=True).str.strip()
print(normalized.tolist())
assert normalized.iloc[0] == normalized.iloc[3] == 'café'['café', '', <NA>, 'café', 'red house']
The Pandas text guide documents the string interface. Use an explicit string dtype and check the behaviour of your installed version. Normalizing the two café spellings helps comparison; removing their accent would be a further, different decision.
Make tokenization assumptions visible
A token is a unit chosen for analysis: perhaps a word-like segment, character or subword. A simple regular expression can illustrate counts without being a language-independent tokenizer. It may treat an apostrophe, hyphen or unspaced script in ways unsuitable for an NLP application.
tokens = normalized.fillna('').str.findall(r'[^\W\d_]+')
lengths = tokens.map(len)
frequency = tokens.explode().dropna().value_counts()
print(lengths.tolist())
print(frequency)
assert lengths.iloc[-1] == 2[1, 0, 0, 1, 2]
café 2
red 1
house 1
Name: count, dtype: int64
Count occurrences and document frequency separately. A word repeated ten times in one document has ten occurrences but appears in one document. Stopword removal changes the representation; negation and domain-specific terms can make a generic list unsuitable. The later NLP book develops tokenization, bag-of-words, TF–IDF, embeddings and transformer representations with task-specific evaluation.
Find duplicates before defining evaluation boundaries
records = pd.DataFrame({'document_id': ['d1','d2','d3'], 'text': ['One title', 'one title', 'Another work']})
records['comparison_text'] = records.text.str.casefold().str.replace(r'\s+', ' ', regex=True).str.strip()
records['duplicate_text'] = records.comparison_text.duplicated(keep=False)
print(records[['document_id', 'duplicate_text']])
assert records.duplicate_text.sum() == 2 document_id duplicate_text
0 d1 True
1 d2 True
2 d3 False
Repeated text may be a legitimate reused title rather than a duplicate observation. Inspect the source IDs and task. Near-duplicate content across training and evaluation can make performance optimistic even when record IDs differ. Labels copied from a filename or text field can also leak the answer. Group related documents before splitting when the intended task concerns unseen sources.
Connect text with images and sequences
A multimodal dataset connects several representations of the same observation. Keep a manifest containing the observation ID, source, file path, label, collection time and split membership. Verify that files exist and that the same observation is not assigned conflicting labels. Do not assume that alphabetical filenames align with a separately sorted target array.
Images commonly use height, width and channel dimensions; batches add a sample dimension. A sequence can use sample, step and feature dimensions. Framework conventions vary, so record the convention. Pixel scaling changes a numeric range; it does not validate image content, rights, labels or representativeness. The existing tensor-handoff experiment lets you verify shapes without downloading images or requiring a neural-network framework.
Practice and explanation
Create three document pairs: one differing only in case, one differing by punctuation, and one with genuinely different meaning. State which transformations preserve the comparison you intend. Explain why duplicate titles are not sufficient evidence for deleting museum records. Then write a manifest for three hypothetical images and describe how you would verify alignment before loading a batch.
For optional AI help, ask: “Critique this text-normalization policy for my stated task. Give a counterexample where a transformation would remove useful meaning.” Verify the counterexample yourself. Your deliverable is an explicit representation policy and a traceable document table, not a claim that a generic cleaner understands language.