Research Design
A reading guide for lecture p-01 — what a counterfactual is, why the choice of comparison group decides the answer, and how that choice hides inside every null hypothesis.
Section one
The definitions the deck depends on, each stated as a claim.
Key concepts
A counterfactual is a claim about a world that did not happen.
The formal definition the deck opens with, from Lewis (1973):
Every causal claim implies a counterfactual claim.
If you assert “P caused Q in circumstances C”, you have committed yourself to “if P had not occurred in circumstances C, then Q would not have occurred.” Causal judgment and counterfactual judgment are the same act described two ways — which is why research design is mostly the business of building a credible “would have.”
In statistics the counterfactual becomes a difference between two groups.
Start from conditional probability, Pr( A | B ), and let B be a treatment:
Pr( Y=TRUE | Treat=TRUE ) − Pr( Y=TRUE | Treat=FALSE )[ mean(Y) | Treat=TRUE ] − [ mean(Y) | Treat=FALSE ]The first term is the world with the treatment; the second is the world without.
We settle for average treatment effects because we only get one world.
The Paris Climate Accord slide makes the point bluntly: the effect is the 2050 temperature with the accord minus the 2050 temperature without it, and we do not have two planets. So we compare groups that stand in for states of the world, and report the average treatment effect — the easiest thing to measure and the most compact way to state program effectiveness.
"Treatment" is a dosage, not a label.
The part of an average treatment effect that is rarely made explicit: what the typical treatment actually was. “Going to the gym increases muscle mass” — how many visits per week, how long, doing what? An effect size means nothing until the dose is specified.
The null hypothesis *is* the counterfactual, written as a number.
In the regression framing carried over from the first course:
b1 = MEAN(treatment) − MEAN(control), i.e. b1 = T2 − C2b1 = 0 is the null: no program impactb1 contains zeroA comparison group is not automatically a control group.
Both stand in for the untreated world, but only one earns the stronger name:
Section two
The mental map. Why the deck spends most of its slides on examples rather than formulas.
Almost none of these slides are about estimation. The regression machinery — difference
of means, confidence interval, b1 = 0 — is review from the first course and gets about
ten slides. The other forty are examples in which the statistics are held fixed and only
the comparison group changes. That is the argument: the hard part of causal inference
is choosing what to compare against, and no amount of estimation rescues a bad choice.
A school district in suburban California has had multiple student suicides. The superintendent cut counseling services. Parents want to know whether the district’s rate is unusually high. You are hired as the expert evaluator.
The same district rate is then tested against three different nulls:
Every one of these is a defensible counterfactual. The conclusion flips from “higher” to “no different” to “lower” without a single change to the model or the data. As the slide says: the conclusions are driven entirely by the selection of the counterfactual.
The question the deck leaves you with is not which test is correct but which comparison best answers the research question that was actually asked.
The deck poses two cases and asks which finding is more meaningful:
Neither answer is free. Case A is a large effect measured imprecisely, usually because suicide is a rare event in a small district. Case B is a precisely measured effect that may be too small to act on. This is the effect-size-versus-precision tension that lecture p-04 takes up directly.
Hamilton College students scored 472 predictions by 26 media prognosticators on a −10 to +10 accuracy scale. Krugman scored 8.2 (p = 0.001), Friedman 2 (p = 0.2461, not significant), Cal Thomas −8.7 (p = 0.0004).
The null being tested is zero on the prognosticator scale — equivalent to a coin flip. The deck asks whether a coin flip is a good counterfactual for a prediction made on television. It is a low bar chosen for convenience rather than for meaning: nobody believes a paid columnist is guessing at random. Plausible alternatives — the base rate of the event, a naive “no change” forecast, or the average of the other 25 pundits — would each produce a different ranking.
Raw comparison: charter schools outscore public schools. Add controls for the population each school actually serves and the advantage shrinks or reverses, because charters are disproportionately sited in better communities and, per Ravitch, “skim the most motivated students out of the poorest communities.”
Read this as the bridge back to the first course. In the regression sequence this was omitted variable bias. Here it is the same problem stated as a design failure: the comparison group was never equivalent to begin with.
The deck closes by placing the three side by side and citing the within-study comparison literature (Cook, Shadish & Wong 2008; Aiken et al. 1998; West et al. 2000) for the claim that careful quasi-experimental methods can reproduce experimental results. The final slide states the course’s organizing question directly:
Y(treatment) − Y(CONTROL) = Y(treatment) − Y(COMPARISON) ?
The rest of the course is the set of conditions under which that equality holds.
Slides 3–13 are the philosophical and statistical setup. Slides 14–47 are five worked examples — the suicide district, the pundits, the Bingham & Felbinger cognitive-ability study, and the charter schools — and they are the point, not illustrations of it. For each one, stop and answer the deck’s own two questions before advancing.
Section three
The checklist to run against yourself once the deck is done.
The deck states these explicitly, and they are worth memorizing:
Treatment Effect = Y(t) − Y(c), and because we only have one world, we estimate it by
comparing groups rather than timelines.