Research Design
A more detailed note on measurement theory — why an index that weights each item by its quality measures a latent construct more reliably than a simple average.
Section one
From simple averages to weighted composites, in plain language.
Key concepts
Start from the measurement model.
Signal, Noise, and Reliable Indices introduced the basic model. Every observed measure of a latent construct is a true score plus error:
and reliability is the share of the observed variance that is signal:
An instrument combines several items so the errors partly cancel. The question for this note is how to combine them.
A simple index treats every item as equally good.
The usual approach adds or averages the items:
Summing and averaging have the same reliability, because dividing by 3 rescales the signal and the noise by the same amount. Both give every item the same weight.
But items are not equally good.
Each item has a part that tracks the true construct and a part that is noise:
The strength of the link between T1, T2, T3 and the true construct T varies from item to item. A simple average gives an item that barely tracks T the same influence as an item that tracks it closely:
A weighted composite gives better items more influence.
The weight on each item should move inversely with its noise. If X1 has a weak relationship with T, then T1 is small, e1 is large, and w1 should be small relative to the weights on the stronger items.
Chosen this way, the weighted composite has less measurement error and higher reliability than the simple average, and it sits closer to the true score T. Factor analysis is the standard way to estimate the weights, because it estimates how strongly each item loads on the construct and how much of each item is unique noise.
Section two
The one-factor model, the optimal weights, and a worked example.
Assume a single latent trait T measured by three items:
λi is the item’s loading, its signal strength. ei is its error, with variance σ2e,i. Collect the loadings in a vector λ, the weights in a vector w, and the error variances and covariances in a matrix Σe.
A linear composite is:
Fix the scale by requiring w′λ = 1. Then the composite is the true score plus a weighted sum of the errors, and its measurement error variance is:
Choose w to minimize the error variance w′Σew, subject to w′λ = 1. The solution is:
When the errors are independent, Σe is diagonal and each weight has a simple form:
High-loading, low-error items get higher weights. Equal weights are optimal only when λi / σ2e,i is the same for every item, for example when all the items have identical loadings and identical error variances.
Three items with these loadings and error variances:
| Item | Loading λ | Error variance σ2e | λ / σ2e | Optimal weight | Equal weight |
|---|---|---|---|---|---|
| X1 | 0.9 | 0.20 | 4.50 | 0.750 | 0.500 |
| X2 | 0.7 | 0.30 | 2.33 | 0.389 | 0.500 |
| X3 | 0.4 | 0.50 | 0.80 | 0.133 | 0.500 |
Both sets of weights are scaled so that w′λ = 1. For equal weights that means w = 1 / (0.9 + 0.7 + 0.4) = 0.5 each.
The weighted composite cuts the error variance by about a third. Nearly all of the gain comes from shrinking the weight on X3, the weakest and noisiest item.
The reliability of a composite is its signal variance over its total variance:
With the constraint w′λ = 1, the numerator is fixed at σ2T, so maximizing reliability is the same as minimizing error variance. The optimal weights maximize reliability, and equal weights reach the same reliability only when the items are equally informative.
In the example, with σ2T = 1:
Cronbach’s alpha is the most common reliability statistic for an index, and it measures internal consistency. It is calculated for the unweighted sum of the items and assumes every item measures the construct equally well. When loadings differ, alpha understates the reliability that a well-weighted composite can achieve. McDonald’s omega relaxes the equal-loadings assumption, and coefficient H gives the reliability of the optimally weighted composite itself.
Factor analysis estimates exactly the ingredients the optimal weights need: the loadings λ and the unique variances σ2e. Bartlett factor scores use the weights Σe−1λ directly, and regression factor scores are proportional to them when there is a single factor.
Principal component analysis also produces a weighted composite, but its weights are chosen to capture total variance. It doesn’t separate signal from noise, so its weights only approximate the reliability-maximizing ones. See the PCA versus Factor Analysis walkthrough.
Section three
The short version, and how it connects to factor analysis.
It depends on how different the items are. In the Factor Analysis walkthrough, six strong socioeconomic items already form a highly reliable simple index (α = 0.94), and the simple average correlates 0.96 with the weighted factor score. When items are uniformly strong, weighting changes little. When one or two items are much noisier than the rest, as in the worked example above, weighting pays off.