Probability, sampling and uncertainty before modelling

A random number generator can produce values without making the uncertainty of an investigation clear. This lesson introduces the language needed to distinguish variation in observations, uncertainty about a summary, and assumptions about sampling. You should already understand a NumPy array, a mean and a fraction.

Events and conditional probability

An event is a specified set of outcomes, such as drawing an even number from a fair six-sided die. Under that stated model its probability is 3/6. Probability belongs to a model or an explicitly described sampling process; it is not created merely by calling a random function.

Conditional probability changes the information available. If you know the die result is greater than three, only 4, 5 and 6 remain possible under the model. Two of these are even, so the conditional probability is 2/3. In symbols, when $P(B)>0$, $P(A\mid B)=P(A\cap B)/P(B)$.

import numpy as np
outcomes = np.arange(1, 7)
even = outcomes % 2 == 0
above_three = outcomes > 3
print(even.mean(), even[above_three].mean())
assert even.sum() == 3 and even[above_three].sum() == 2

0.5 0.6666666666666666

In observational data, conditioning on a recorded group is a descriptive restriction. It does not itself tell you what intervening on that group would do. Compare like observation processes and consider selection and confounding before interpreting a conditional comparison as an effect.

Expected value and variability

The expected value is a probability-weighted average of possible values: $E[X]=\sum_x xP(X=x)$ for a discrete variable. A fair die has expected value 3.5 even though no single roll equals 3.5. Variance measures squared deviation from that expectation: $\operatorname{Var}(X)=E[(X-E[X])^2]$. Its square root has the original unit.

probabilities = np.repeat(1/6, 6)
expectation = (outcomes * probabilities).sum()
variance = ((outcomes - expectation)**2 * probabilities).sum()
print(expectation, variance)
assert np.isclose(expectation, 3.5)
assert np.isclose(variance, 35/12)

3.5 2.9166666666666665

A simulation approximates features of the specified model. More simulated rolls can reduce Monte Carlo variation around the model’s expectation; they cannot establish that a real die is fair or that a convenient observational dataset is representative.

Sampling is part of the analysis

A population is the set of units your question concerns; a sample is the observed subset. Random selection can support particular inferential arguments when the design and assumptions are respected. Convenience records, missing groups and measurement errors can limit those arguments. A large biased sample can still give a precise answer to the wrong question.

Repeated observations from the same person, institution or time period may be dependent. More rows do not necessarily mean proportionally more independent information. Define the sampling unit before constructing uncertainty intervals. Keep a record of nonresponse, exclusions and coverage.

Distinguish spread from uncertainty of a mean

Standard deviation describes variation among observations. Standard error describes variation of an estimator under a specified repetition of sampling. For independent, identically distributed observations with finite variance, the familiar estimated standard error of a mean is $s/\sqrt{n}$. Those assumptions are substantive; temporal or group dependence can invalidate the simple calculation.

sample = np.array([4., 6., 8., 10.])
s = sample.std(ddof=1)
se = s / np.sqrt(sample.size)
print(sample.mean(), s, se)
assert sample.mean() == 7.0 and np.isclose(se, s/2)

7.0 2.581988897471611 1.2909944487358056

A confidence procedure concerns repeated sampling under its assumptions. It is not a guarantee that the parameter lies inside one realized interval, nor a statement about the proportion of individual observations inside it. The earlier resampling experiment lets you investigate an interval procedure; it does not remove its assumptions.

Investigate rather than ritualize a threshold

Before a formal test, define the scientific question, outcome, comparison and sampling design. A small p-value is not an effect size, and a large p-value does not prove equality. Repeated exploration can produce striking results by chance. Report the size, direction, support and limitations of a comparison rather than treating one threshold as a substitute for reasoning. See the American Statistical Association statement for the scope and limits of p-value interpretation.

Practice and explanation

For the die model, calculate the probability of a result above four conditional on an odd result. Check it by enumerating the outcomes before simulating. Then describe a real situation where more recorded rows do not create independent observations. State what population, sample and sampling unit would mean for that investigation.

A useful optional tutor prompt is: “Here is my population and sampling plan. Identify one coverage problem and one possible dependency. Ask for clarification before suggesting a confidence interval.” Your answer should remain grounded in the actual collection design.

Chapter exercisesWorked answers after an attempt