Frisch–Waugh–Lovell theorem

The Frisch-Waugh-Lovell theorem is an algebraic result for regressions estimated by least squares, the most commonly used estimator in applied econometrics.^[4] Least squares is a method of estimating coefficients in models which are linear in parameters. That is, the outcome variable is modeled as a linear combination of the input variables plus some error term. The least squares solution is that which sets the input variables' coefficients to minimize the sum of squared errors. Under a certain set of assumptions, the Gauss–Markov theorem, least squares estimation is the best linear unbiased estimator.

Let ${\displaystyle y}$ be any outcome variable and ${\displaystyle X}$ a set of predictor variables, such that ${\displaystyle X=\left[x_{1}~x_{2}~\cdots ~x_{k}\right]}$, and suppose ${\displaystyle n}$ observations of ${\displaystyle (y_{i},X_{i})}$ are sampled. If ${\displaystyle y}$ is modeled as a linear function of ${\displaystyle X}$, it can be written as ${\displaystyle y_{i}=\beta _{0}+\beta _{1}x_{1i}+\beta _{2}x_{2i}+\cdots +\beta _{k}x_{ki}+e_{i}}$. The least squares estimator sets the coefficients ${\displaystyle \beta _{j},j\in \{0,\dots ,k\}}$ to minimize the sum of squared errors ${\textstyle \sum _{i=1}^{n}e_{i}^{2}}$. With ${\displaystyle n}$ observations this involves minimizing across ${\displaystyle n}$ equations, and is typically written in matrix form as ${\displaystyle y=X\beta +e}$, where ${\displaystyle y}$ and ${\displaystyle e}$ are ${\displaystyle n}$-dimensional column vectors and ${\displaystyle X}$ is an ${\displaystyle n}$-by-${\displaystyle k}$-dimensional matrix. Then, the least squares solution is ${\displaystyle \beta =(X^{\prime }X)^{-1}X^{\prime }y}$.^[5]

In regressions estimated by least squares, it is common to refer to a coefficient as the effect of that variable "holding constant" the other input variables.^[6] For example, if wage is modeled as a function of education and work experience, the coefficient for education is interpreted as the difference in the expectation of wage for a unit difference in education, "holding constant" work experience. Econometrician Arthur Goldberger frames the Frisch-Waugh-Lovell theorem as "giving content to th[is] language".^[7]

The Frisch-Waugh-Lovell theorem states that in a least squares-estimated regression of the form

${\displaystyle y_{i}=\beta _{0}+\beta _{1}x_{1i}+\beta _{2}x_{2i}+\cdots +\beta _{k}x_{ki}+e_{i}}$

any coefficient ${\displaystyle \beta _{j},j\in \{1,\dots ,k\}}$ can be estimated by the two-step process of:

1. Regress ${\displaystyle x_{j}}$ on the set of other right-hand-side variables, obtaining residuals ${\displaystyle {\tilde {x}}_{j}}$
2. Regress ${\displaystyle y}$ on ${\displaystyle {\tilde {x}}_{j}}$, obtaining ${\displaystyle \beta _{j}={\frac {{\text{cov}}(y,{\tilde {x}}_{j})}{{\text{var}}({\tilde {x}}_{j})}}}$

This two-step process is referred to as the residual regression or equivalently the regression anatomy theorem.^[8][9][10]

The theorem shows that coefficients in a multiple regression reflect the relationship between the associated variable and the outcome variable after removing the part linearly explained by the other predictor variables.^[11][9] This is the basis for understanding the contribution of each single variable to a multivariate regression (see, for instance, Ch. 13 in ^[12]).

The origin of the theorem is uncertain, but it was well-established in the realm of linear regression before the Frisch and Waugh paper. George Udny Yule's comprehensive analysis of partial regressions, published in 1907, included the theorem in section 9 on page 184.^[14]

Yule emphasized the theorem's importance for understanding multiple and partial regression and correlation coefficients, as mentioned in section 10 of the same paper.^[14]

Yule 1907 also introduced the partial regression notation which is still in use today.

In 1962, Richard Stone generalized the theorem to apply to an arbitrary number of variables which may be chosen for special analysis in the same way that time was distinguished in Frisch's and Waugh's original formulation.^[15]

In 1963, Lovell published a proof considered more straightforward and intuitive.^[2] In recognition, people generally add his name to the theorem name.

Consider the linear regression ${\displaystyle y=X\beta +Z\delta +e}$ and annihilator matrix ${\displaystyle M_{Z}=I-Z(Z^{\prime }Z)^{-1}Z^{\prime }}$. Premultiplying both sides of the regression equation by the annihilator matrix removes from ${\displaystyle y}$ and ${\displaystyle X}$ the component linearly explained by ${\displaystyle Z}$:

${\displaystyle {\begin{aligned}M_{Z}y&=M_{Z}(X\beta +Z\delta +e)\\{\tilde {y}}&=M_{Z}X\beta +M_{Z}Z\delta +M_{Z}e\\{\tilde {y}}&={\tilde {X}}\beta +e\end{aligned}}}$

Then, by the least squares result, ${\displaystyle \beta =({\tilde {X}}^{\prime }{\tilde {X}})^{-1}{\tilde {X}}^{\prime }{\tilde {y}}}$ and ${\displaystyle e={\tilde {y}}-{\tilde {X}}\beta }$. This concludes the proof.^[16][7]

• Helmert–Wolf blocking
• Gram–Schmidt process