Prepare a trustworthy handoff for machine learning and deep learning
On this page
A model receives a representation of observations, not the meaning you intended. The handoff must therefore preserve row identity, feature meaning, target timing, partition membership and numerical structure. This lesson assumes that you can validate a table, work with arrays and explain an observation key. It does not train a predictive model; that belongs to the next books.
Define the prediction-time question
Suppose a later project will estimate reading time from information available before a session begins. Pages planned might be available; minutes actually spent cannot be a predictor of itself. A review written after a session could reveal the outcome and would not be available at prediction time. List each proposed feature, when it becomes available and why it is permitted.
A target is the outcome to predict. An identifier distinguishes observations; it is not automatically a useful quantitative feature. An apparent shortcut may encode a collection batch, a future outcome or a memorized entity. Exclude such information when it would make evaluation answer a different question from deployment.
Choose evaluation partitions before learning transformations
A random split can be appropriate for approximately independent observations from the intended setting. Repeated entities may require grouped partitions; future prediction requires time-aware evaluation. The scikit-learn cross-validation guide describes alternatives for grouped and temporal data. Choose the split from the intended use, not from whichever method produces the strongest score.
The following values are synthetic. The final two observations deliberately differ strongly from the first four so that leakage is easy to see.
import numpy as np
import pandas as pd
frame = pd.DataFrame({
"record_id": ["s1", "s2", "s3", "s4", "s5", "s6"],
"date": pd.date_range("2023-01-01", periods=6, freq="D"),
"pages": [100., 120., 140., 160., 1000., 1200.],
"minutes": [10., 12., 14., 16., 100., 120.],
}).set_index("record_id")
train = frame.loc[frame["date"] < "2023-01-05"].copy()
test = frame.loc[frame["date"] >= "2023-01-05"].copy()
assert set(train.index).isdisjoint(test.index)
assert train["date"].max() < test["date"].min()
print(len(train), len(test))4 2An evaluation set should not influence the fitted preprocessing used to assess generalization. This includes learned imputation, scaling, feature selection and category vocabularies. The scikit-learn pitfalls guide explains this form of leakage and how pipelines help avoid it. Here we demonstrate the arithmetic directly so you can inspect what is learned.
location = train["pages"].mean()
scale = train["pages"].std(ddof=0)
assert scale > 0
train_scaled = (train["pages"] - location) / scale
test_scaled = (test["pages"] - location) / scale
print(location, round(scale, 3))
print(test_scaled.round(3).tolist())
assert location == 130.0
assert not np.isclose(location, frame["pages"].mean())130.0 22.361
[38.908, 47.852]The large transformed test values reveal a distribution difference in this constructed example. Recomputing the scaler on the test set would hide part of that difference and change the evaluation procedure. A constant training feature needs an explicit policy because its standard deviation is zero. Unknown future categories also need a documented treatment; do not silently renumber categories separately in training and evaluation.
Learn missing-value and category rules from training data
A missing numeric feature needs a declared treatment. In this demonstration, learn a median from observed training values and reuse it when an evaluation value is missing. Do not recompute the replacement from evaluation outcomes or evaluation feature distributions.
observed_train = pd.Series([100.0, np.nan, 140.0, 160.0])
replacement = observed_train.median()
filled_train = observed_train.fillna(replacement)
filled_test = pd.Series([np.nan, 1000.0]).fillna(replacement)
assert replacement == 140.0 and filled_test.iloc[0] == 140.0
print(filled_train.tolist(), filled_test.tolist())[100.0, 140.0, 140.0, 160.0] [140.0, 1000.0]Categorical encoding also needs a stable vocabulary. Build it from training observations, preserve column order, and explicitly identify unknown evaluation categories. In this example an unseen category has all-zero known-category indicators and a separate unknown flag; this is a documented policy, not an assumption that the unseen category equals one of the known ones.
train_genre = pd.Series(["history", "poetry", "history", "drama"], index=train.index)
test_genre = pd.Series(["poetry", "essay"], index=test.index)
vocabulary = sorted(train_genre.unique())
train_encoded = pd.get_dummies(train_genre).reindex(columns=vocabulary, fill_value=False)
test_encoded = pd.get_dummies(test_genre).reindex(columns=vocabulary, fill_value=False)
unknown_genre = ~test_genre.isin(vocabulary)
print(vocabulary, unknown_genre.tolist())
assert test_encoded.iloc[1].sum() == 0 and unknown_genre.iloc[1]
assert train_encoded.columns.tolist() == test_encoded.columns.tolist()['drama', 'history', 'poetry'] [False, True]Different applications may reject unknown categories or learn a dedicated unknown category; choose the policy before evaluation. Preserve the vocabulary and missing-value rule alongside the numeric scaler. Neither one-hot column numbers nor category codes define a meaningful ordering unless the scientific variable itself is ordered.
Preserve feature-target alignment
Select columns by an explicit list and select targets using the same observation order. Converting a mixed table directly to an array can produce an object array or include an unintended target column. Before discarding labels, verify exactly which rows and fields will be transferred.
feature_names = ["pages"]
X_train = train.loc[:, feature_names].to_numpy(dtype="float32")
y_train = train.loc[:, "minutes"].to_numpy(dtype="float32")
# In normal use, retain IDs explicitly rather than relying only on equal lengths.
ids = train.index.to_numpy()
y_aligned = frame.loc[ids, "minutes"].to_numpy(dtype="float32")
assert np.array_equal(y_train, y_aligned)
assert X_train.shape == (4, 1) and y_aligned.shape == (4,)
assert np.isfinite(X_train).all() and np.isfinite(y_aligned).all()
X_train_scaled = train_scaled.to_numpy(dtype="float32")[:, None]
assert X_train_scaled.shape == X_train.shape
print(X_train.shape, y_aligned.shape, feature_names)(4, 1) (4,) ['pages']Here X_train retains raw feature units and X_train_scaled uses the earlier training-fitted scaling. Record which representation a later model receives. The transferable rule is that IDs establish alignment; equal lengths alone do not. If you shuffle feature rows, apply the same ID order to the target. A later implementation should package that rule into its data-loading and validation functions.
Read tensor dimensions without guessing
A tensor is a multidimensional array in this introductory handoff. A common sequence representation is (samples, time_steps, features); a common image representation is (samples, height, width, channels). Frameworks can use different conventions. State the convention instead of treating an axis number as self-explanatory.
sequence_batch = np.arange(12, dtype="float32").reshape(2, 3, 2)
image_batch = np.zeros((2, 3, 4, 1), dtype="float32")
print(sequence_batch.shape, image_batch.shape)
assert sequence_batch[1, 0, 0] == 6(2, 3, 2) (2, 3, 4, 1)These arrays illustrate shapes only; they are not measured sequences or real images. Reshaping cannot repair a mistaken association between an observation and its label. Sequence windows also require a prediction-time boundary: no input window may contain information from after the outcome it is intended to predict. Whether earlier context may be shared across evaluation boundaries depends on the declared deployment setting.
Deliver a data contract
A useful handoff records observation IDs, ordered feature names, target definition, units, dtypes, shapes, missing-value policy, source snapshot, split IDs and preprocessing parameters fitted on training data. Include checks for disjoint partitions, temporal ordering where relevant, finite required values and target alignment. Save uncertainty about coverage and representativeness rather than treating a numerical array as proof of readiness.
Practice and independent check
Shuffle the feature rows and show how ID-based target selection preserves alignment. Add a repeated person ID and design a grouped split for prediction on unseen people. Add a predictor that becomes available after the outcome and reject it with a written availability rule. Finally, label every dimension of a sequence and image batch.
Your handoff is ready for the next learning stage when another reader can reproduce its arrays and explain what every row, column, partition and dimension means. Model choice, neural-network architecture, optimization and performance claims remain later tasks.