Moment matrix

Summarize Timeline Top Qs Fact Check

A multiple linear regression model can be written as

${\displaystyle y=\beta _{0}+\beta _{1}x_{1}+\beta _{2}x_{2}+\dots \beta _{k}x_{k}+u}$

where ${\displaystyle y}$ is the dependent variable, ${\displaystyle x_{1},x_{2}\dots ,x_{k}}$ are the independent variables, ${\displaystyle u}$ is the error, and ${\displaystyle \beta _{0},\beta _{1}\dots ,\beta _{k}}$ are unknown coefficients to be estimated. Given observations ${\displaystyle \left\{y_{i},x_{i1},x_{i2},\dots ,x_{ik}\right\}_{i=1}^{n}}$, we have a system of ${\displaystyle n}$ linear equations that can be expressed in matrix notation.^[3]

${\displaystyle {\begin{bmatrix}y_{1}\\y_{2}\\\vdots \\y_{n}\end{bmatrix}}={\begin{bmatrix}1&x_{11}&x_{12}&\dots &x_{1k}\\1&x_{21}&x_{22}&\dots &x_{2k}\\\vdots &\vdots &\vdots &\ddots &\vdots \\1&x_{n1}&x_{n2}&\dots &x_{nk}\\\end{bmatrix}}{\begin{bmatrix}\beta _{0}\\\beta _{1}\\\vdots \\\beta _{k}\end{bmatrix}}+{\begin{bmatrix}u_{1}\\u_{2}\\\vdots \\u_{n}\end{bmatrix}}}$

or

${\displaystyle \mathbf {y} =\mathbf {X} {\boldsymbol {\beta }}+\mathbf {u} }$

where ${\displaystyle \mathbf {y} }$ and ${\displaystyle \mathbf {u} }$ are each a vector of dimension ${\displaystyle n\times 1}$, ${\displaystyle \mathbf {X} }$ is the design matrix of order ${\displaystyle n\times (k+1)}$, and ${\displaystyle {\boldsymbol {\beta }}}$ is a vector of dimension ${\displaystyle (k+1)\times 1}$. Under the Gauss–Markov assumptions, the best linear unbiased estimator of ${\displaystyle {\boldsymbol {\beta }}}$ is the linear least squares estimator ${\displaystyle \mathbf {b} =\left(\mathbf {X} ^{\mathsf {T}}\mathbf {X} \right)^{-1}\mathbf {X} ^{\mathsf {T}}\mathbf {y} }$, involving the two moment matrices ${\displaystyle \mathbf {X} ^{\mathsf {T}}\mathbf {X} }$ and ${\displaystyle \mathbf {X} ^{\mathsf {T}}\mathbf {y} }$ defined as

${\displaystyle \mathbf {X} ^{\mathsf {T}}\mathbf {X} ={\begin{bmatrix}n&\sum x_{i1}&\sum x_{i2}&\dots &\sum x_{ik}\\\sum x_{i1}&\sum x_{i1}^{2}&\sum x_{i1}x_{i2}&\dots &\sum x_{i1}x_{ik}\\\sum x_{i2}&\sum x_{i1}x_{i2}&\sum x_{i2}^{2}&\dots &\sum x_{i2}x_{ik}\\\vdots &\vdots &\vdots &\ddots &\vdots \\\sum x_{ik}&\sum x_{i1}x_{ik}&\sum x_{i2}x_{ik}&\dots &\sum x_{ik}^{2}\end{bmatrix}}}$

and

${\displaystyle \mathbf {X} ^{\mathsf {T}}\mathbf {y} ={\begin{bmatrix}\sum y_{i}\\\sum x_{i1}y_{i}\\\vdots \\\sum x_{ik}y_{i}\end{bmatrix}}}$

where ${\displaystyle \mathbf {X} ^{\mathsf {T}}\mathbf {X} }$ is a square normal matrix of dimension ${\displaystyle (k+1)\times (k+1)}$, and ${\displaystyle \mathbf {X} ^{\mathsf {T}}\mathbf {y} }$ is a vector of dimension ${\displaystyle (k+1)\times 1}$.

• Design matrix
• Gramian matrix
• Projection matrix