Explore distributions, groups and defensible comparisons

The GroupBy and operations lessons give you the mechanics of summaries. This lesson asks what those summaries mean. It assumes that you can filter, group and aggregate a table. The constructed examples are small enough to check by hand; they illustrate reasoning rather than provide evidence about actual institutions.

Count observations and measure values separately

A group’s row count and the number of non-missing values in a column can differ. Report support for a summary, especially when missingness is uneven. A mean based on two observations should not appear to have the same evidential support as a mean based on two thousand.

import pandas as pd
records = pd.DataFrame({"group": ["A", "A", "B", "B"], "score": [8., None, 5., 7.]})
summary = records.groupby("group").agg(rows=("score", "size"), observed=("score", "count"), mean=("score", "mean"), median=("score", "median"))
print(summary)
assert summary.loc["A", "rows"] == 2 and summary.loc["A", "observed"] == 1

       rows  observed  mean  median
group                              
A         2         1   8.0     8.0
B         2         2   6.0     6.0

Before comparing means, ask whether the variable has the same unit, measurement process and observation period in both groups. Inspect extremes and quantiles alongside the average. A later visualization can reveal structure that a table hides; for now, use these checks to decide what a figure must show rather than choosing a chart before understanding the data.

Define the denominator and weights

A count answers “how many recorded events?” A rate answers “how many events relative to a stated exposure or eligible group?” An average of subgroup rates weights each subgroup equally. A pooled rate weights each underlying eligible observation equally, provided that the observations and denominators are compatible and non-overlapping.

attendance = pd.DataFrame({"group": ["small", "large"], "attended": [9, 30], "registered": [10, 60]})
attendance["rate"] = attendance["attended"] / attendance["registered"]
print(attendance["rate"].mean())
pooled = attendance["attended"].sum() / attendance["registered"].sum()
print(pooled)
assert attendance["rate"].mean() == 0.7
assert abs(pooled - 39 / 70) < 1e-12

0.7
0.5571428571428572

The two results differ because the small group receives different weight. Neither is universally correct. State the target quantity and choose the weighting that matches it. A rate without a denominator description is incomplete communication.

Check whether aggregation hides a different pattern

The following example deliberately assigns different mixes of easier and harder tasks. In each task category, A has the higher success rate; after pooling, B has the higher rate because B has many more easy tasks. This is an arithmetic construction showing how composition can reverse a comparison.

trial = pd.DataFrame({
    "method": ["A", "A", "B", "B"],
    "task": ["easy", "hard", "easy", "hard"],
    "success": [9, 20, 80, 1],
    "total": [10, 100, 100, 10],
})
trial["rate"] = trial["success"] / trial["total"]
print(trial.pivot(index="task", columns="method", values="rate"))
pooled = trial.groupby("method")[["success", "total"]].sum()
print(pooled["success"] / pooled["total"])
assert 9/10 > 80/100 and 20/100 > 1/10
assert 29/110 < 81/110

method    A    B
task            
easy    0.9  0.8
hard    0.2  0.1
method
A    0.263636
B    0.736364
dtype: float64

This does not identify which method would improve outcomes under intervention. It tells you to inspect composition and define the target comparison. The appropriate adjustment depends on the scientific question and data-generating process, not on which summary gives the preferred ranking.

Investigate relationships and uncertainty

Use cross-tabulations for joint category counts, proportions with stated normalization, and correlation for a clearly defined numeric association. Record sample size and missing-value handling. Correlation is sensitive to the observations included, and many exploratory comparisons increase the opportunity to find a striking pattern by chance. A surprising result should motivate verification, not an immediate causal story.

When reporting uncertainty, identify what varies: individual observations, sampling of observations, measurement, or a model’s assumptions. A standard deviation is not interchangeable with a confidence interval. Reuse the mathematical lesson’s resampling example only when its sampling assumptions are justified. Separate exploratory findings from questions specified before examining the outcome.

Practice and independent check

Produce a cross-tabulation of method and task, then calculate the share of easy tasks within each method. Explain the pooled reversal in words without saying that one table is “wrong.” Change the task counts while retaining each task-specific rate and observe which pooled comparison changes.

For independent work, write a short exploratory memo using your project data. Include one overall summary, one subgroup comparison, missingness and support counts, one alternative explanation, and one limitation. Suggest a figure that Book 3 could use to check the proposed pattern. The deliverable is a defensible question and summary, not a claim that exploratory arithmetic has established a real-world mechanism.

Chapter exercisesWorked answers after an attempt