Contrasts and Effect Size

Section one

Key concepts

The vocabulary for describing a design as a grid of groups and time periods.

Key concepts

A design is a grid: groups down the side, time periods across the top.

The whole deck runs on one picture. Four groups (G1–G4) observed at five time points (t=0 through t=4), with an x marking when each group starts treatment:

G1  t=0   t=1   t=2   t=3   t=4
G2  t=0   t=1   t=2   t=3   t=4
G3  t=0   t=1   t=2   t=3   t=4
G4  t=0   t=1   t=2   t=3   t=4

Once the study is drawn this way, a “contrast” is just a choice of which cells to compare.

A contrast is a chosen comparison, not a fixed property of the study.

The same grid supports many different comparisons, and each one answers a different question. The design does not hand you an answer — it hands you a menu.

Staggered start dates turn a single study into a dose–response study.

Because the groups begin treatment at different times, the grid encodes treatment dosage measured as program duration:

Group 1 reaches 6, 12, 18, and 24 months of treatment across the observation window
Group 2 reaches 6, 12, and 18 months
Comparing G1 against {G2, G3, G4} at the right moment isolates 6 months versus 0 months

Pooling groups buys statistical power and costs interpretability.

Two examples from the deck:

Pool {G2, G3, G4} as untreated to estimate the effect of 6 months of treatment.
Pool G1 (12 months) with G2 (6 months) and the pooled estimate now represents roughly 9 months of treatment — an average dose that no individual group actually received.

More observations shrink the standard error; the price is that the “treatment” being estimated becomes an average of unlike things.

Dosage and age are confounded when programs start at different ages.

Children enter at 3, 3.5, 4, and 4.5 years old. A 6-months-versus-0-months contrast is therefore always conditioned on the age at which the child started the program. You cannot vary duration without also varying age.

Three estimators, three sets of assumptions.

The deck names the estimators the rest of the course develops, each paired with what it must assume to be valid:

Reflexive — the group is its own comparison, before versus after.
Post-test only — treatment versus comparison, measured once, after the fact.
Difference-in-difference — the change in the treated group minus the change in the comparison group.

Section two

Key relationships

The mental map. One study, many contrasts, and what each one costs.

This deck is the applied companion to p-01

Lecture p-01 argued that the choice of counterfactual determines the answer. This deck takes one real study — the Bingham & Felbinger chapter on improving cognitive ability in chronically deprived children — and makes you do the choosing. The grid is the same diagram that appeared at the end of p-01; here it gets eleven slides of variations.

Every question you might ask maps to a different set of cells

Work through the deck by asking, for each slide, which cells are being differenced?

Is there any effect at all? Treated rows against untreated rows at a common time.
Does more treatment help more? Groups at 6, 12, 18, 24 months against untreated.
Do late starters gain as much as early starters? The deck’s “treatment gains for late treatment” slides compare groups that entered at different points.
Is two periods as good as three, but cheaper? A contrast between two dosage levels, which is a cost-effectiveness question rather than an efficacy question.

The high-SES group is doing something different from the control group

One of the discussion questions asks what role the high socioeconomic status group plays. It is not a control group in the strict sense — it is not equivalent to the treated children and was never going to be. It functions as a benchmark: a target level the intervention is trying to move deprived children toward. Confusing a benchmark with a control group is one of the easier mistakes to make when reading a study like this.

This is not an RCT, and the deck wants you to say so

The discussion slide opens by asking directly: Is this an RCT? Do we have an identical control group? The staggered starts, the age confound, and the SES benchmark all point the same way. It is a strong quasi-experimental design, and the course’s whole project is establishing when a design like this can stand in for the randomized one.

Where each estimator shows up later

The three estimators sketched at the end are placeholders for the rest of the sequence. The reflexive and post-test-only estimators are developed in the varieties-of-the- counterfactual lecture; difference-in-difference gets its own lab (lab-05-diff-in-diff), and the gender wage gap lab (lab-07) is the second worked example the deck points to.

Section three

What should be clear in my mind?

The questions the deck asks you to answer about the case study.

The deck’s own discussion questions

Answer these before moving on — they are the assessment for this lecture:

Is this an RCT? Do we have an identical control group?
What role does the high-SES group perform?
Why are there four treatment groups rather than one?
What outcome is measured, and is it valid and reliable?
How would you test whether two treatment periods have the same impact as three, but at lower cost?
Can you identify a weakness in the design, or a threat to validity?

Key takeaways

Draw the design as a groups × time-periods grid before doing anything else. Most design questions become visible once the grid is on paper.
A contrast is a decision. The same study yields different effect sizes depending on which cells you difference.
Staggered entry converts a yes/no treatment into a dose, which is more informative and harder to interpret.
Pooling groups raises power and blurs the definition of the treatment. Say what dose the pooled estimate actually represents.
When entry age varies with dose, duration effects and age effects cannot be separated without further assumptions.
Reflexive, post-test-only, and difference-in-difference are three ways of filling in the missing cell of the grid, and each is only as good as its assumptions.