From an interesting question to a reproducible investigation

A table can contain thousands of rows and still be unable to answer the question that matters. Before calculating anything, write down what you want to learn, what one observation represents, and which observations would make an answer possible. This lesson connects the introduction to the practical work that follows. It assumes that you can run Python, use lists and dictionaries, call functions, and read a simple error message. If those skills feel uncertain, revisit the corresponding Book 1 examples; you do not need to repeat a full programming course here.

Define the question and the observation

Suppose a library manager asks whether a weekend programme attracted more readers. “More” needs a comparison: more than weekdays, more than last month, or more than the number of seats available? The observation might be a registration, a visit, a person or a programme session. These are different units. One person can attend several sessions, and a registration does not guarantee attendance. An identifier should distinguish the unit you chose, rather than silently changing its meaning halfway through the investigation.

Write a bounded question: “For the three sessions in this teaching example, which had the highest attendance proportion among registered participants?” This question defines its scope and denominator. It does not ask whether the programme caused an increase in reading. Description, comparison, prediction and causal explanation require different evidence; a comparison of recorded groups alone does not establish the effect of an intervention. For the formal distinction, see Hernán and Robins, Causal Inference: What If (2020), available from the authors’ book page.

Follow the lifecycle, including its return paths

Use the sequence question → acquire → inspect → prepare → explore → communicate. Return to an earlier stage when a check fails. If an identifier is not unique, revisit the observation unit. If an important group is absent, revisit acquisition. If a proposed conclusion depends on an unverified assumption, revisit the question. Numerical foundations support the whole sequence; they are tools you will reuse, rather than a separate investigation to finish once and forget.

Keep raw inputs separate from derived tables. Record a source URL or file identity, acquisition time, permission or licence, units and any selection rule. A saved snapshot lets another reader investigate the same observations even when a live service changes. Your analysis log should explain what changed and why, not merely list the functions you called.

Work through a small investigation

The following records are invented teaching data, not a survey or a public library’s operational records. Predict which session wins by count, then which wins by attendance proportion.

sessions = [
    {"session": "A", "registered": 20, "attended": 15},
    {"session": "B", "registered": 10, "attended": 9},
    {"session": "C", "registered": 30, "attended": 18},
]
for row in sessions:
    assert 0 <= row["attended"] <= row["registered"]
    row["attendance_rate"] = row["attended"] / row["registered"]
print(max(sessions, key=lambda r: r["attended"])["session"])
print(max(sessions, key=lambda r: r["attendance_rate"])["session"])
print(sum(r["attended"] for r in sessions) / sum(r["registered"] for r in sessions))

C
B
0.7

Session C has the largest count, 18; B has the highest proportion, 0.9. Across all registrations the proportion is 42/60 = 0.7. These statements can all be true. The difference is the question, not an inconsistency in Python. If the same person registered several times, the pooled result describes registrations, not the fraction of distinct people reached.

A zero denominator needs an explicit policy. It may mean that a session was cancelled, that registration was unavailable, or that the data are incomplete. Replacing the denominator with one would manufacture a result. Retain the unresolved case and explain why a rate cannot be calculated.

Make the workspace reproducible

Use folders such as data/raw, data/derived, notebooks, code and reports. These names are a convention; the important separation is between evidence, transformations and communication. Record Python and package versions, the commands needed to run the work, and any input selection. Restart a notebook’s kernel and run its cells in order before treating the displayed output as reproducible. A value left over from an earlier experiment can otherwise make a broken analysis appear to work.

Optional AI support comes after your attempt. Ask for an explanation of one error or a critique of your denominator, then verify the response against the records. Keep credentials and non-public personal records out of prompts. An assistant’s fluent summary is not evidence that the rows or arithmetic are correct.

Practice and independent check

Add a fourth session with 12 registrations and 6 attendances. Before running the code, predict whether either winning session changes and calculate the new pooled fraction by hand. Explain why averaging the four session rates would answer a different question from dividing total attendances by total registrations. Then propose a data dictionary with a type, meaning and unit for every field.

For an independent task, choose a question about your own publicly shareable data. Submit a one-page analysis brief: decision or question, observation unit, comparison, denominator, required fields, source, limitations and expected output. Do not begin with a favourite library or model. Begin with the evidence an answer would require.

Transition

The next numerical lessons teach how to represent observations in arrays. Keep the observation unit in mind: a shape describes storage, while the analysis brief explains what that storage means.

Chapter exercisesWorked answers after an attempt