Chapter 8 worked answers: Cleaning and Validating Data

Compare your reasoning after an attempt. Different valid implementations are welcome when they preserve the observation set and the stated meaning.

B2-08-E01 · Separate parsing and validation

Parse the numeric values and flag both unresolved and negative counts.

A negative number parses successfully but violates the count rule.

s=pd.Series(["12","unknown","-2"])
numeric=pd.to_numeric(s,errors="coerce")
invalid=numeric.isna()|numeric.lt(0)
assert invalid.tolist()==[False,True,True]
print(invalid.tolist())

[False, True, True]

B2-08-E02 · Investigate key conflicts

Identify all conflicting ID rows without silently choosing one.

These are conflicting records, not exact duplicate rows; a source rule is required to choose or reconcile them.

df=pd.DataFrame({"id":["a","a","b"],"value":[2,3,4]})
conflicts=df.loc[df.id.duplicated(keep=False)]
assert len(conflicts)==2 and conflicts.value.tolist()==[2,3]
print(conflicts)

  id  value
0  a      2
1  a      3

B2-08-E03 · Preserve missingness semantics

Normalize institution spacing and retain the distinction between missing visits and zero visits.

Whitespace normalization is a label policy. It does not justify replacing an unresolved count with zero.

df=pd.DataFrame({"institution":[" N ","S",None],"visits":[2,None,0]})
clean=df.copy();clean["institution"]=clean.institution.astype("string").str.strip()
assert pd.isna(clean.loc[1,"visits"]) and clean.loc[2,"visits"]==0
print(clean)

  institution  visits
0           N     2.0
1           S     NaN
2        <NA>     0.0

Return to practice

Change one supplied input, predict the effect and rerun your own implementation. Return to the exercise prompts to identify a task you can now explain independently.

Chapter exercisesWorked answers after an attempt