Understand a machine-learning question before choosing a model
On this page
Machine learning uses observations to learn a rule whose performance must be assessed on information not used to fit that rule. Before the later ML book introduces algorithms, you need to distinguish the task, the target, the information available at prediction time and the way success will be measured. This chapter uses hand-checkable predictions so that evaluation does not depend on an unexplained model.
Name the task and target
Regression predicts a numeric quantity such as a future duration. Classification predicts a category such as a document label. Clustering groups observations according to a chosen representation and similarity; it does not discover an objectively correct category system merely by producing clusters. A label must have a stated meaning, source and observation time.
A feature is an input measurement or representation. The target is what the task asks you to predict. Identifiers, collection batches and information recorded after the outcome may create shortcuts that will not exist in the intended setting. Write the availability rule before inspecting which feature seems most predictive.
Compare with a simple baseline
A baseline makes a task measurable before model complexity enters. For a numeric target, a training mean or median is a simple constant prediction. A classification baseline might predict the most common training label. Fit even these simple choices on training observations, then evaluate on separate observations.
import numpy as np
train_minutes = np.array([8., 10., 12., 14.])
test_minutes = np.array([9., 15., 18.])
baseline = np.repeat(train_minutes.mean(), len(test_minutes))
errors = test_minutes - baseline
mae = np.abs(errors).mean()
rmse = np.sqrt((errors**2).mean())
print(baseline, mae, rmse)
assert np.allclose(baseline, [11., 11., 11.])
assert np.isclose(mae, 13/3)[11. 11. 11.] 4.333333333333333 4.795831523312719
Mean absolute error is . Root mean squared error is . Both retain the target unit; squared error gives larger deviations more influence. Select the measure from the consequences and question, not the one that makes a chosen model look strongest. See the scikit-learn evaluation guide for metric definitions and conventions.
Understand classification errors
For a binary task, define the positive label first. A true positive is a positive observation predicted positive; a false positive is a negative observation predicted positive. A false negative is a missed positive. Accuracy counts all correct predictions, while precision and recall focus on different denominators.
actual = np.array([1, 1, 0, 0, 0, 0])
predicted = np.array([1, 0, 1, 0, 0, 0])
tp = int(((actual == 1) & (predicted == 1)).sum())
fp = int(((actual == 0) & (predicted == 1)).sum())
fn = int(((actual == 1) & (predicted == 0)).sum())
accuracy = (actual == predicted).mean()
precision = tp / (tp + fp)
recall = tp / (tp + fn)
print(accuracy, precision, recall)
assert tp == fp == fn == 1
assert precision == recall == .50.6666666666666666 0.5 0.5
These constructed labels do not represent a clinical, legal or other operational decision system. They expose the arithmetic. If a denominator is zero, choose and report a defined handling policy rather than concealing the undefined quantity. For several classes, report how class-specific measures are combined and inspect support for each class.
Separate fitting, selection and final evaluation
Training observations fit a rule. Validation observations support choices such as a threshold or hyperparameter. A final test set estimates performance after those choices have been settled. Reusing a test set to select the best experiment turns it into part of the selection process. Repeated subjects, groups and time order require appropriate partitions rather than a default shuffle.
The data-handoff lesson supplies concrete examples of disjoint time splits, training-fitted scaling, imputation and category vocabularies. Carry those boundaries into a model pipeline. A low training error alone cannot distinguish a useful pattern from memorization, leakage or excessive flexibility. Learning curves and residual analyses become important in the next books.
Investigate errors and distribution differences
Report which observations were evaluated and whether their distribution resembles the intended use. Inspect errors by meaningful groups when support allows it. A difference in group performance can motivate investigation; it does not explain its own cause. Preserve the outcome definition, measurement process and missingness when comparing evaluations.
A model can be repeatable and inappropriate for a decision. The data-analysis responsibility is to expose those limits and supply a transparent evaluation contract. A suitable handoff lists the question, feature availability, target timing, partition IDs, preprocessing parameters, baseline, chosen metrics and unresolved risks.
Practice and explanation
For the regression example, calculate one absolute and one squared error by hand. Explain why RMSE exceeds MAE here. For classification, compare an always-negative prediction with the supplied predictions: calculate accuracy and recall and explain why the preferred system cannot be selected from accuracy alone without a stated use.
An optional tutor prompt is: “I have defined my positive class, baseline and evaluation split. Challenge one denominator or availability assumption before recommending an algorithm.” You should finish this chapter able to explain an evaluation problem independently, before studying how a regression, classifier or neural network is fitted.