Missing Data
On this page
Data analysis in the real world rarely begins with a perfect dataset. Researchers and analysts often encounter missing or incomplete data — gaps caused by errors in data entry, inconsistent file formats, sensor failures, or simply because some information was never collected in the first place.
In the context of Dynamic Humanities, missing data can occur in digitized archives, metadata, or text corpora. For instance, a historical record may lack a publication year, an author’s gender might not be documented, or a survey may include partial responses. These gaps must be handled carefully: deleting too much can remove valuable context, while filling data incorrectly can distort analysis.
In this section, we explore strategies for detecting, understanding, and handling missing data in Pandas. We will cover:
- Identifying missing data
- Removing incomplete entries
- Filling or imputing missing values
- Replacing missing data with computed statistics
By the end, the reader will understand how to prepare data so that it can be analyzed safely and meaningfully — an essential step before applying grouping, summarization, or statistical modeling.
Creating and Exploring Missing Data
To understand how Pandas handles missing values, let’s begin by constructing a simple dataset that includes a few empty cells. In Pandas, missing values are typically represented as NaN, which stands for Not a Number.
import numpy as np
import pandas as pd
# Create a small dataset with missing values
df = pd.DataFrame({
'A': [1, 2, np.nan],
'B': [5, np.nan, np.nan],
'C': [1, 2, 3]
})
print(df) A B C
0 1.0 5.0 1
1 2.0 NaN 2
2 NaN NaN 3
Here, columns A and B contain missing values (NaN), while C is complete. This small table represents a typical situation: partial information across multiple variables.
Before deciding what to do with missing data, it’s important to identify where it appears. Pandas provides the method .isnull() for this purpose.
print(df.isnull()) A B C
0 False False False
1 False True False
2 True True False
Each True indicates a missing entry. We can also quickly summarize how many missing values exist per column using .isnull().sum():
print(df.isnull().sum())A 1
B 2
C 0
dtype: int64
This summary reveals that column B is most affected, with two missing values.
Removing Missing Data
The simplest strategy is to remove rows or columns that contain missing values. A small missing fraction alone does not justify deletion: even a few systematically missing observations can change a comparison. Define which fields the question needs, record exclusions and consider what caused the gaps.
Dropping Rows with Missing Values
We can remove any row containing at least one NaN using .dropna():
cleaned_rows = df.dropna()
print(cleaned_rows) A B C
0 1.0 5.0 1
Only the first row remains, because it was the only one without missing data. While this ensures a complete dataset, it can also lead to significant data loss — so it should be used with caution.
Dropping Columns with Missing Values
If a specific column contains too many missing values, it may be more practical to remove the entire column instead of entire rows. To do so, we use the argument axis=1:
cleaned_columns = df.dropna(axis=1)
print(cleaned_columns) C
0 1
1 2
2 3
Column C is kept because it contains no missing values. This approach is common when certain variables are incomplete and contribute little to the analysis.
Setting a Threshold
Sometimes, it’s useful to retain rows that have at least a minimum number of valid (non-missing) values. The thresh parameter lets us define this requirement:
thresholded = df.dropna(thresh=2)
print(thresholded) A B C
0 1.0 5.0 1
1 2.0 NaN 2
Here, only rows with two or more valid entries are preserved. The threshold is a mechanical row-selection rule, not a quality guarantee. A retained row may still lack the particular measurement your question requires. Prefer an explicit required-column check when that is the intended contract.
Filling Missing Values
Instead of removing missing data, another approach is to fill it — that is, replace missing entries with substitute values. This is especially useful when each missing value represents a small gap that doesn’t compromise the overall dataset.
Pandas offers the .fillna() method for this task. Let’s start with a simple example:
filled_df = df.fillna(value='FILL VALUE')
print(filled_df) A B C
0 1.0 5.0 1
1 2.0 FILL VALUE 2
2 FILL VALUE FILL VALUE 3
Here, all missing values were replaced with the string “FILL VALUE”. In real datasets, we often fill missing values using numerical estimates, previous observations, or default constants depending on the context.
Filling with Column Means
A common statistical approach is to replace missing numerical data with the mean (average) of the column. Mean filling retains rows but does not preserve the observed distribution: it inserts extra values at the mean and changes spread and relationships. The code demonstrates an operation, not a default recommendation.
df['A'] = df['A'].fillna(value=df['A'].mean())
print(df) A B C
0 1.0 5.0 1
1 2.0 NaN 2
2 1.5 NaN 3
The missing value in column A was replaced with the mean of the other two values, (1 + 2) / 2 = 1.5. Whether this is appropriate depends on the variable, missingness process and analytical purpose. The inserted value is an estimate, not a recovered observation. Preserve a missingness flag and compare conclusions under defensible alternatives. In predictive work, estimate an imputation rule from training data only; see the imputation guide.
Filling with Forward or Backward Values
When data represents a sequence over time, missing values can sometimes be filled using nearby observations. This is known as forward filling (ffill) or backward filling (bfill).
df['B'] = df['B'].ffill()
print(df) A B C
0 1.0 5.0 1
1 2.0 5.0 2
2 1.5 5.0 3
Here, each missing value in column B was replaced by the last valid value above it. Forward filling carries a past value into later rows. It needs a meaningful sort order and a defensible persistence assumption; it should not cross unrelated entities or unbounded gaps. Backward filling can introduce future information. The toy table has no time column, so this example shows the mechanics only.
Replacing Missing Data with Calculated Values
More sophisticated techniques use derived or modeled values to fill gaps. For example, missing data can be replaced by median, mode, or even predictions from regression models. More elaborate methods require their own assumptions and validation. Filling cells does not automatically preserve statistical relationships or account for uncertainty in the estimated values.
Summary
This section introduced how to identify, remove, and fill missing data using Pandas. Missing values (NaN) can be handled with .dropna() to remove them or .fillna() to replace them with constants or statistics such as the mean. Documented handling makes the assumptions visible; it cannot guarantee an unbiased or accurate analysis. Keep the original values and justify the policy for the question being answered.
Next, audit types, identifiers and unresolved records in the cleaning-and-validation lesson. Later, GroupBy connects these decisions to group sizes, denominators and comparisons.
Experiment: filling at the mean changes spread
Before running, compare the observed values 1 and 2 with the filled values 1, 2 and 1.5. Which mean stays fixed, and which variance changes?
raw = pd.Series([1.0, 2.0, np.nan])
was_missing = raw.isna()
filled = raw.fillna(raw.mean())
print(raw.mean(), filled.mean())
print(raw.var(ddof=0), filled.var(ddof=0))
assert was_missing.sum() == 1 and raw.isna().sum() == 1
assert np.isclose(raw.mean(), filled.mean())
assert np.isclose(raw.var(ddof=0), 0.25)
assert np.isclose(filled.var(ddof=0), 1/6)1.5 1.5
0.25 0.16666666666666666
Both means are 1.5, but the descriptive variance falls from 0.25 to about 0.1667. The added midpoint changes the distribution without adding an observed measurement. Save the missingness mask and compare observed-only with filled results before interpreting a downstream statistic.