Sums of Squares in Regression

This document explains how the Total Sum of Squares (TSS) can be decomposed into the Explained Sum of Squares (ESS) and the Residual Sum of Squares (RSS) using the trick of inserting the regression line between the observed data \(y_i\) and the mean \(\bar{y}\).

1. Definitions

For observations \(y_i\), their mean \(\bar{y}\), and fitted values from a regression \(\hat{y}_i\):

\[ TSS = \sum_i (y_i - \bar{y})^2 \]

\[ ESS = \sum_i (\hat{y}_i - \bar{y})^2 \]

\[ RSS = \sum_i (y_i - \hat{y}_i)^2 \]

The decomposition is:

\[ TSS = ESS + RSS \]

2. The Trick

We start with:

\[ y_i - \bar{y} = (y_i - \hat{y}_i) + (\hat{y}_i - \bar{y}) \]

Squaring both sides:

\[ (y_i - \bar{y})^2 = (y_i - \hat{y}_i)^2 + (\hat{y}_i - \bar{y})^2 + 2(y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) \]

Summing over \(i\):

\[ TSS = RSS + ESS + 2\sum_i (y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) \]

3. Orthogonality

In OLS regression, the cross-product vanishes:

\[ \sum_i (y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) = 0 \]

Thus, we obtain the decomposition:

\[ TSS = ESS + RSS \]

4. Example in R

Let’s demonstrate this with a small example using simulated data.

set.seed(123)

# Generate data
x <- 1:10
y <- 2 + 0.5 * x + rnorm(10, 0, 1)

# Fit regression
model <- lm(y ~ x)
yhat <- fitted(model)
ybar <- mean(y)

# Create components
df <- data.frame(
  X = x,
  Y = y,
  yhat = yhat,
  ybar = ybar,
  resid = y - yhat,
  fit_dev = yhat - ybar,
  total_dev = y - ybar
)

knitr::kable(df, digits=3)
X Y yhat ybar resid fit_dev total_dev
1 1.940 2.943 4.825 -1.004 -1.881 -2.885
2 2.770 3.362 4.825 -0.592 -1.463 -2.055
3 5.059 3.780 4.825 1.279 -1.045 0.234
4 4.071 4.198 4.825 -0.127 -0.627 -0.754
5 4.629 4.616 4.825 0.014 -0.209 -0.195
6 6.715 5.034 4.825 1.681 0.209 1.890
7 5.961 5.452 4.825 0.509 0.627 1.136
8 4.735 5.870 4.825 -1.135 1.045 -0.090
9 5.813 6.288 4.825 -0.475 1.463 0.989
10 6.554 6.706 4.825 -0.151 1.881 1.730

5. Check Decomposition

We can see that:

\[ TSS = ESS + RSS \]

TSS <- sum((df$total_dev)^2)
ESS <- sum((df$fit_dev)^2)
RSS <- sum((df$resid)^2)

c(TSS = TSS, ESS = ESS, RSS = RSS, ESS_plus_RSS = ESS + RSS) |> pander::pander()
TSS ESS RSS ESS_plus_RSS
22.05 14.42 7.633 22.05