Signal, Noise, and Reliable Indices

Section one

Key concepts

Observed scores, true scores, and the signal-to-noise ratio called reliability.

Key concepts

Every measure of a latent construct is a true score plus error.

Measurement theory defines a quantitative measure of a latent construct as:

M=T+e

where M is the observed measurement (the recorded scores on an IQ test), T is the true score (each individual’s actual intelligence quotient relative to the population), and e is random error. M, T, and e are vectors of scores, one per person.

Reliability is a signal-to-noise ratio.

Reliability is the ratio of the variance of the true scores to the variance of the true scores plus measurement error:

α= var(T) var(T + e)

Alpha tells us how reliably our observed measure captures the true level of the latent construct. Cronbach’s alpha is the usual way to estimate this ratio from item-level data.

Instruments combine multiple items to triangulate the construct.

Measures like weight or height only require one measurement. Instruments designed to measure latent constructs usually combine several. All of the verbal reasoning questions on an exam might be combined into a single reading comprehension score, and responses to several survey items might be combined into a single index.

Each item can be decomposed into a component X that accurately captures the latent construct and a component e that is random noise:

item₁=X₁+e₁

An index built from three items combines them:

index=T + e where T = X₁ + X₂ + X₃, e = e₁ + e₂ + e₃

Its reliability is measured in exactly the same way, as signal over signal plus noise: α = var(T) / var(T + e).

Summing and averaging give the same reliability.

If three appraisers value a painting using slightly different methods, adding their three appraisals would not make sense. You would average them to preserve the scale:

index= T + e 3

Dividing by a constant rescales the signal and the noise by the same factor, so the alpha of an averaged index is the same as the alpha of a summed index.

The takeaways, if the math is not crystal clear

Assume that any latent construct in your analysis is measured with error.
Measurement instruments are indices created by averaging multiple items, usually survey responses.
Strong instruments reduce the amount of measurement error in the composite variable.

Section two

Constructing a new instrument

How experts turn a fuzzy idea like neighborhood quality into a validated index.

Experts have developed many ways to define neighborhood quality

The American Human Development Index. Measure of America’s index is modeled on the UN’s Human Development Index for nations. It measures well-being for America’s 435 congressional districts plus Washington, D.C., which shows how uneven socio-economic well-being is not just between but within many of the largest metros. It combines three dimensions:

A long and healthy life, measured by life expectancy at birth.
Access to knowledge, based on school enrollment for ages 3 to 24 (weighted one-third) and degree attainment for adults 25 and older (weighted two-thirds).
Standard of living, based on median earnings for full- and part-time workers 16 and older.

The overall index is graded on a one-to-ten scale, with ten the highest.

Toxic and punishing neighborhoods. Manduca and Sampson (2019) ask how social and physical environments beyond concentrated poverty predict children’s long-term well-being. They focus on neighborhoods that are harsh on child development: high violence, incarceration, and lead exposure. Their measures come from four sources:

a representative Chicago community survey on neighborhood social environments (1995 and 2002, combined N of 12,000)
violent crime rates per 100,000 residents, 1995–2000, from offenses reported to police
incarceration rates, 1995–2000, from prison admission data
childhood lead exposure, 1995–1997, from more than 150,000 blood tests by Chicago’s health department

Simple proxies are often too simple

These are examples of how scholars operationalize neighborhood quality in order to understand how neighborhoods affect the people who live in them. The measurement is not always straightforward. When data are sparse or research is poorly implemented, neighborhood quality is often measured with a single proxy such as the poverty rate. Proxies like that can be overly simplistic, and as a result not very informative.

Think about a neighborhood the way psychologists think about personality

A better approach, from a measurement theory perspective, is to ask whether we can measure the personality of a neighborhood the same way psychologists measure personality types: identify the most salient dimensions, then develop reliable instruments to measure each one precisely.

For reference

Aparasu, R. R. (2011). Measurement theory and practice. In Research Methods for Pharmaceutical Practice and Policy, Ch. 4. Pharmaceutical Press. PDF
Schäffer, U. (2007). Management Accounting & Control Scales Handbook. Springer Science & Business Media. PDF
MacKenzie, S. B., Podsakoff, P. M., & Podsakoff, N. P. (2011). Construct measurement and validation procedures in MIS and behavioral research: Integrating new and existing techniques. MIS Quarterly, 35(2), 293–334. PDF
Manduca, R., & Sampson, R. J. (2019). Punishing and toxic neighborhood environments independently predict the intergenerational social mobility of black and white children. Proceedings of the National Academy of Sciences, 116(16), 7772–7777. PDF

Section three

Practice

Build a reliable index of community well-being in the measurement lab.

Build a reliable index of community well-being

A community is healthy if we expect its residents to achieve a good quality of life and high economic stability while living there. Your task is to create a reliable index that measures it.

Open the measurement lab →

Start with the warm-up exercise for practice.
Then use the Census data on the Index Design tab. Select 5 variables from a list of 30 candidates.
The app calculates Cronbach’s alpha for the variables you select. Your index must reach alpha ≥ 0.70.

How to read alpha

Cronbach’s alpha runs from 0 (all noise) to 1 (an index that measures the construct very precisely). By convention an index needs alpha above 0.6 to be considered reliable, and above 0.8 it is considered highly reliable. Items on a well-designed index produce highly correlated responses because they measure the same construct. The higher the correlations, the higher the alpha.

A tip on direction

You can build an index where a high score means well-being, or one built from items that measure the absence of well-being. The high and low ends of a scale are easy to reverse. Either way, the goal is a reliable index.