Regression Foundations
A reading guide for lecture p-02 — how one algebraic trick splits the variance of Y, and why both R2 and the standard error come out of it.
Section one
The formulas and definitions the slides depend on, each stated as a claim.
Key concepts
Inserting the regression line splits every deviation in two.
Take the distance from a point to the mean and add and subtract ŷi:
Squaring and summing gives three sums of squares.
Divide any of them by n − 1 and you have a variance. The slides often work with the raw sums because the divisors cancel.
R2 is the explained share of the total.
Note: these slides use RSS = Regression SS and ESS = Error SS. The deck flags this itself — some textbooks use RSS for residual and ESS for explained.
In Venn terms, the overlap between X and Y is the explained part.
The standard error of the slope is built from the unexplained part.
The same standard error can be written two ways.
Because ∑(xi − )2 = (n − 1) · var(x):
The first form is preferred in the slides because it names its three moving parts out loud:
Compare the standard error of the mean, which has only two moving parts: SE = sx / √n.
Section two
The mental map. How the pieces connect, and how to read the deck.
Add and subtract ŷi. That is it.
The distance from a data point to the mean is the only thing we had before this lecture. By inserting the fitted value in the middle of that distance, it becomes two distances: the part the model accounts for, and the part it does not. Everything downstream — R2, the standard error, the confidence interval, the p-value — is bookkeeping on those two pieces.
Notice what is not being assumed here. This is an identity, not a modeling choice. Any line drawn through the data would split the deviations this way. What makes the regression line special is that it is the line that makes the unexplained piece as small as possible.
Worth knowing if a student asks why TSS = RSS + ESS holds exactly rather than approximately: when you square the sum of the two pieces, the cross-product term sums to zero, because least squares forces the residuals to be uncorrelated with the fitted values. The slides do not derive this — they present the partition as a given — but that orthogonality is what makes it clean.
The deck shows the split three different ways, and the exercise is to be able to move between them:
If you can look at a scatterplot and sketch its bar and its Venn diagram, you have the lecture.
The overlap region is labelled cov(x, y), and that is not decoration. In p-01 the slope was cov(x, y) / var(x). Here the same covariance is the part of Y that X can account for.
So covariance is doing two jobs at once: it sets the size of the slope and it sets the share of variance explained. When the circles barely overlap you get a small b1 and a small R2 together. When they overlap a lot you get both large. That is why “more correlation” and “better fit” feel like the same thing — in simple regression with one X, they are.
This is the relationship worth carrying forward. One partition, two consumers:
| Uses | Reports | |
|---|---|---|
| R2 = RSS/TSS | the explained piece | how well the model fits |
| SEb1 | the unexplained piece | how precise the estimate is |
They are complements, not competitors: R2 goes up exactly when the residual goes down, which is exactly when the standard error shrinks. A model that explains more of Y produces a tighter estimate of the slope. Fit and precision are two readings of the same number.
This is also the honest answer to “why do we care about R2?” On its own it is a description. What matters is that the other half of the same split is what determines whether your finding is statistically significant.
Writing the standard error with (n − 1) · var(x) in the denominator makes visible what the compact ∑(xi − )2 version hides. Three things, and only three, move it:
Contrast the standard error of the mean, sx/√n, which has only the first two. The third lever is new, and it is the one with consequences for research design: how you assign or sample your explanatory variable matters as much as how many cases you collect.
p-01 produced the slope out of covariance and variance. This lecture takes the variance of Y apart and finds the standard error inside the leftover piece. p-03 and p-04 take that standard error and turn it into a confidence interval and a significance test. The road map slide near the end is the same one from p-01, and by now three of its four rungs are built.
| Slides | What they are doing | What to take away |
|---|---|---|
| 1–2 | Variance recap | Squared distances from the mean, divided by n − 1. Nothing new yet. |
| 3–7 | The partition and R2 | TSS = RSS + ESS. This is the core. |
| 8, 14 | One point, close up | The bracket picture. Explained is mean-to-line; unexplained is line-to-point. |
| 9–10 | The two extremes | Perfect fit explains everything; the typical case explains some. |
| 11–12 | The stacked-bar version | R2 as the orange fraction of the column. |
| 13, 15–17 | The Venn version | Overlap = cov(x, y) = explained. Note the RSS/ESS footnote on slide 15. |
| 18–21 | Standard error of the slope | Built from the residual; three levers. |
| 22 | The road map | Same map as p-01. Rung three is now in place. |
| 23 | The checklist | Two items, both about the split. |
The repetition across slides 4, 6, 11, and 12 is not padding — the same diagram is being re-shown with a different annotation layered on each time. Read them as one slide seen four ways.
Section three
The two things this deck exists to settle.