Operations with Pandas

In the previous chapters, we learned how to structure and combine datasets using Pandas. Now, the next step in real-world data analysis is to explore, transform, and summarize that data — turning it from raw information into usable insight.

These day-to-day tasks fall under what we call Pandas operations. They include identifying unique values, filtering rows, applying functions, sorting, cleaning, and creating quick summaries such as pivot tables.

In this chapter, we will work through these operations using a small but realistic dataset inspired by literary and cultural analytics.

An Illustrative Literary Dataset

Suppose a digital humanities researcher studies book publications by different authors across years. They collect a dataset containing each author’s name, the number of books published, and their average reader rating:

These rows are an invented teaching fixture, including ratings; they are not a verified bibliography or empirical study. The fixed reference year 2025 makes the years-ago calculation reproducible.

import pandas as pd
import numpy as np

data = {
    'Author': ['Austen', 'Tolstoy', 'Orwell', 'Austen', 'Orwell', 'Shakespeare', 'Tolstoy', 'Orwell'],
    'Year': [1813, 1877, 1945, 1811, 1949, 1603, 1869, 1933],
    'Books_Published': [1, 1, 1, 1, 1, 1, 1, 1],
    'Avg_Rating': [4.3, 4.1, 4.5, np.nan, 4.7, 4.8, np.nan, 4.4],
    'Genre': ['Romance', 'Historical', 'Dystopian', 'Romance', 'Political', 'Tragedy', 'Historical', 'Political']
}

df = pd.DataFrame(data)
df.head()

This dataset mirrors the kind of information used in literary data analysis, where researchers compare authors, genres, and publication trends.

Exploring Unique Values

The first step in exploration is to check what unique categories exist in our data. For example, how many different genres are present?

df['Genre'].unique()

array(['Romance', 'Historical', 'Dystopian', 'Political', 'Tragedy'],
      dtype=object)

We can also count how many unique authors appear and how often each one occurs:

print(df['Author'].nunique(), "\n\n", df['Author'].value_counts())

4 

 Author
Orwell         3
Austen         2
Tolstoy        2
Shakespeare    1
Name: count, dtype: int64

Orwell appears in three of these eight invented rows. That describes the fixture. The table contains no edition identifier or selection evidence from which to infer why an author appears more often.

Filtering Data with Conditions

Next, analysts often want to focus on a subset of the data — for example, books published after 1900 with ratings above 4.5.

high_rated_modern = df[(df['Year'] > 1900) & (df['Avg_Rating'] > 4.5)]
high_rated_modern

Filtering this way allows researchers to isolate specific cultural trends, for instance, high-rated works in modern political literature.

Applying Functions to Data

Sometimes data needs to be transformed or enriched using functions. Pandas’ .apply() method lets us apply a function to each value in a column. For example, compute elapsed calendar years from the recorded publication year to the fixed reference year 2025. This is neither a century count nor evidence of the year of composition:

def years_ago(year):
    return 2025 - year

print()
df['Years_Ago'] = df['Year'].apply(years_ago)
df[['Author', 'Year', 'Years_Ago']]

This kind of derived variable helps contextualize data — here showing how distant in time each publication is.

Aggregation and Summarization

To summarize numerical columns, we can use aggregation functions like sum(), mean(), or count(). For instance, calculate the unweighted mean of non-missing row-level ratings. This is not an equal-author average, and no underlying reviewer counts are supplied:

print(df['Avg_Rating'].mean())

4.466666666666666

Or count non-missing Books_Published entries per author. Every supplied row has value 1, so counting rows and summing books happen to agree here. For rows containing several books, use sum() to total the recorded quantities:

print(df.groupby('Author')['Books_Published'].count())

Author
Austen         2
Orwell         3
Shakespeare    1
Tolstoy        2
Name: Books_Published, dtype: int64

This describes how many selected rows contribute per author in the invented table, not an author’s complete publishing productivity.

Sorting and Ordering Data

Sorting helps uncover patterns, for example, finding the most highly rated works.

df.sort_values(by='Avg_Rating', ascending=False)

Sorting places the Shakespeare row with rating 4.8 first, followed by one Orwell row at 4.7. This is a row-level ordering of invented ratings, not a supported ranking of authors or literary quality.

Handling Missing Data

Missing values are a normal part of real-world datasets. In our example, some average ratings are missing:

df['Avg_Rating'].isnull()

0    False
1    False
2    False
3     True
4    False
5    False
6     True
7    False
Name: Avg_Rating, dtype: bool

For a mechanical demonstration, fill the missing ratings with the mean of the observed ratings. This is an assumed replacement policy, not evidence that those books received that rating. The following pivot includes these inserted values:

observed_ratings = df['Avg_Rating'].copy()
df['Rating_Was_Missing'] = observed_ratings.isna()
df.fillna({'Avg_Rating': df['Avg_Rating'].mean()}, inplace=True)
df

Every row now has a numeric entry, but two entries are estimates. Numeric completeness does not establish reliable averages or correlations. Preserve an observed-versus-filled flag and report the policy; never describe an inserted value as an observed rating.

Creating Pivot Tables

Pivot tables are a powerful way to summarize data across multiple dimensions. For example, let’s explore the average rating per author and genre:

pivot = df.pivot_table(values='Avg_Rating', index='Author', columns='Genre', aggfunc='mean')
pivot

The pivot summarizes this particular invented, partly filled table. It does not identify reader preferences or differences between literary traditions. A missing author–genre combination is an absent combination, not a zero rating.

Summary

This chapter showed how Pandas operations turn raw data into insight — through filtering, transformation, aggregation, sorting, and summarization. Using a literary dataset, we uncovered trends in publication, ratings, and genres, illustrating how data becomes interpretation.

Next, connect these operations to exploratory reasoning: inspect denominators, weights and subgroup relationships before defending a comparison.

Experiment: a count is different from a quantity

Suppose a row summarizes several books rather than one. Predict the row count and the sum before running.

quantities = pd.DataFrame({"author": ["A", "A", "B"], "books": [2, 3, 4]})
summary = quantities.groupby("author")["books"].agg(rows="count", books="sum")
print(summary)
assert summary.loc["A", "rows"] == 2
assert summary.loc["A", "books"] == 5

        rows  books
author             
A          2      5
B          1      4

Author A has two non-missing rows representing five books. Replacing one quantity with a missing value makes count() fall, while size() still counts both rows. Neither missing quantity nor an absent combination should silently become a recorded zero.

Chapter exercisesWorked answers after an attempt