Use diagnostic plots to investigate data

A diagnostic plot helps you check a question about the observations. It may reveal a skewed distribution, a subgroup, an unusual value or a missing period that a single summary hides. This chapter introduces five useful views with Matplotlib. Book 3 develops chart design, statistical displays and visual communication in greater depth.

Begin with Figure, Axes and a question

A Figure contains the drawing; an Axes provides a plotting region and its coordinate system. plt.subplots() gives you both. Use named axes methods so that the labels, title and marks belong to the intended panel. The Matplotlib quick-start guide documents this object interface.

The measured penguin snapshot contains species, island and physical measurements. Gorman and colleagues collected the observations through Palmer Station LTER; the simplified dataset is distributed by Horst, Hill and Gorman’s palmerpenguins project. Preserve their attribution. Missing measurements are not zero-sized animals.

from pathlib import Path
import pandas as pd
import matplotlib.pyplot as plt
penguins = pd.read_csv(Path('data/teaching/penguins.csv'))
measured = penguins.dropna(subset=['body_mass_g', 'flipper_length_mm'])
assert len(penguins) == 344 and len(measured) == 342

Inspect a distribution with a histogram

A histogram divides a numeric axis into intervals and counts observations in them. Its appearance depends on the intervals. A peak in a pooled distribution can reflect several species rather than one homogeneous population. Change the bins and compare subgroup views before interpreting a small bump as a meaningful structure.

fig, ax = plt.subplots(figsize=(6, 4))
ax.hist(measured['body_mass_g'], bins=12, color='#315b84', edgecolor='white')
ax.set(xlabel='Body mass (g)', ylabel='Measured animals', title='Body mass in the saved penguin sample')
fig.tight_layout()
plt.show()

The count denominator is the 342 observations with both chosen measurements, because that was the declared analysis subset. A separate body-mass-only analysis could choose its own inclusion rule. Always connect the plotted observations to the table that supplied them.

Compare groups with a box plot

A box plot compresses a distribution into a median, quartiles and a specified whisker convention. Marks outside the whiskers are candidates for investigation; the display does not prove they are errors. Read group sizes alongside the plot and consider whether sex, island or collection year changes the comparison.

species = sorted(measured['species'].unique())
fig, ax = plt.subplots(figsize=(6, 4))
ax.boxplot([measured.loc[measured.species.eq(s), 'body_mass_g'] for s in species])
ax.set_xticks(range(1, len(species)+1), species)
ax.set(xlabel='Species', ylabel='Body mass (g)', title='Distribution within species')
fig.tight_layout()
plt.show()

With the default Matplotlib box-plot convention, whiskers extend to the farthest observed value within 1.5 interquartile ranges of a box edge. State that convention if it matters to an interpretation. Do not describe it as a universal definition of an outlier.

Investigate a relationship with a scatterplot

Each point represents one observation, placed using two numeric values. Colour identifies a documented group; it does not establish causality. Inspect whether a pooled relationship also appears within groups and whether the plotted rows have different missingness from the full table.

fig, ax = plt.subplots(figsize=(6, 4))
for name, group in measured.groupby('species'):
    ax.scatter(group.flipper_length_mm, group.body_mass_g, label=name, alpha=0.6, s=20)
ax.set(xlabel='Flipper length (mm)', ylabel='Body mass (g)', title='Measurements with species shown')
ax.legend()
fig.tight_layout()
plt.show()

Compare counts with bars and ordered values with lines

Bars support comparisons of discrete categories. Begin their quantitative axis at zero when bar length encodes magnitude. A time line communicates order and spacing; connecting a missing period without qualification can imply observations you do not possess.

counts = penguins['species'].value_counts().sort_index()
fig, ax = plt.subplots(figsize=(6, 4))
ax.bar(counts.index, counts.values, color='#315b84')
ax.set(xlabel='Species', ylabel='Recorded animals', title='Support for each species comparison')
fig.tight_layout()
plt.show()

series = pd.Series([10., 12., None, 18.], index=pd.date_range('2023-01-01', periods=4, freq='D'))
fig, ax = plt.subplots(figsize=(6, 4))
ax.plot(series.index, series.values, marker='o')
ax.set(xlabel='Date', ylabel='Illustrative visits', title='Retain an unresolved day as a gap')
fig.autofmt_xdate()
fig.tight_layout()
plt.show()

Export and critique the result

A useful figure includes units, an intelligible title, readable labels and a source or caption. Export SVG when selectable vector elements and text are useful. fig.savefig('outputs/figure.svg') writes an output; create the output folder first and keep it separate from raw evidence. The book’s experiment runner saves the diagnostic figures for inspection.

Before accepting a figure, identify its observation unit, exclusions, axes, visual encoding and main limitation. Ask whether a table or a different chart would make the comparison clearer. In the chapter exercises, change a bin count, identify the relevant denominator, and explain why a subgroup pattern deserves investigation before fitting a line.

Chapter exercisesWorked answers after an attempt