Worked solution: build a traceable document table
On this page
The solution treats entity identity and text representation separately. It checks source IDs, keeps raw titles, records unresolved text, then computes normalized comparison fields. Repeated normalized titles are exported for investigation, not removed.
The regular expression deliberately illustrates its own limits. It retains runs of Unicode letters and excludes digits and underscores. An apostrophe or hyphen can split a familiar expression; an unspaced script may remain one long run. Explain those consequences before using the features for another task.
The occurrence table and per-document counts reconcile the representation. The plot shows which tokens occur often in this selection. It does not infer themes or classify texts. Before later NLP modelling, define labels, related-document grouping and a representation learned only from training documents.
Complete implementation
The following function is the implementation supplied in code/book2lab.py. It receives an explicit source path and output folder. The companion script imports the required libraries and saves selectable SVG plots.
from pathlib import Path
import json
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from book2lab import save_plot
def text_analysis(path,out):
out=Path(out);out.mkdir(parents=True,exist_ok=True);df=pd.read_csv(path,dtype={'id':'string','title':'string'})
if df.id.duplicated().any():raise ValueError('Repeated document identifier')
clean=df[['id','title']].copy();clean['missing_text']=clean.title.isna() | clean.title.str.strip().eq('')
clean['normalized']=clean.title.str.normalize('NFKC').str.casefold().str.replace(r'\s+',' ',regex=True).str.strip()
clean['characters']=clean.normalized.str.len().astype('Int64')
clean['tokens']=clean.normalized.fillna('').str.findall(r'[^\W\d_]+')
clean['token_count']=clean.tokens.map(len)
# Simple Unicode letter runs, not a language-independent linguistic tokenizer.
frequency=clean.tokens.explode().dropna().value_counts().rename_axis('token').reset_index(name='count')
duplicate_groups=clean.loc[~clean.missing_text & clean.normalized.duplicated(keep=False),['id','title','normalized']]
result={'documents':len(clean),'missing_text':int(clean.missing_text.sum()),'duplicate_title_records':len(duplicate_groups),'token_occurrences':int(clean.token_count.sum()),'vocabulary':len(frequency),'tokenization':'Unicode letter runs; no stopword removal or language identification'}
clean.drop(columns='tokens').to_csv(out/'documents.csv',index=False);frequency.to_csv(out/'token_counts.csv',index=False);duplicate_groups.to_csv(out/'duplicate_titles.csv',index=False);(out/'audit.json').write_text(json.dumps(result,indent=2))
top=frequency.head(10).iloc[::-1];fig,ax=plt.subplots(figsize=(7,4));ax.barh(top.token,top['count'],color='#315b84');ax.set(xlabel='Occurrences in selected titles',ylabel='Letter-run token',title='Title vocabulary: a descriptive view');save_plot(fig,out/'vocabulary.svg')
return clean,frequency,result
result = text_analysis(Path('data/teaching/museum.csv'), Path('outputs/text'))
print(result[-1] if isinstance(result, tuple) else result)Explain and extend
The occurrence table and per-document counts reconcile the representation. The plot shows which tokens occur often in this selection. It does not infer themes or classify texts. Before later NLP modelling, define labels, related-document grouping and a representation learned only from training documents.
Return to the project brief and compare the acceptance evidence with your own outputs. Do not replace a valid alternative merely because its variable names differ.
Inspect the verified output
{
"documents": 100,
"missing_text": 0,
"duplicate_title_records": 0,
"token_occurrences": 394,
"vocabulary": 260,
"tokenization": "Unicode letter runs; no stopword removal or language identification"
}