Bayesian model reduction

Consider some model with parameters ${\displaystyle \theta }$ and a prior probability density on those parameters ${\displaystyle p(\theta )}$. The posterior belief about ${\displaystyle \theta }$ after seeing the data ${\displaystyle p(\theta \mid y)}$ is given by Bayes rule:

${\displaystyle {\begin{aligned}p(\theta \mid y)&={\frac {p(y\mid \theta )p(\theta )}{p(y)}}\\p(y)&=\int p(y\mid \theta )p(\theta )\,d\theta \end{aligned}}}$

1

The second line of Equation 1 is the model evidence, which is the probability of observing the data given the model. In practice, the posterior cannot usually be computed analytically due to the difficulty in computing the integral over the parameters. Therefore, the posteriors are estimated using approaches such as MCMC sampling or variational Bayes. A reduced model can then be defined with an alternative set of priors ${\displaystyle {\tilde {p}}(\theta )}$:

${\displaystyle {\begin{aligned}{\tilde {p}}(\theta \mid y)&={\frac {p(y\mid \theta ){\tilde {p}}(\theta )}{{\tilde {p}}(y)}}\\{\tilde {p}}(y)&=\int p(y\mid \theta ){\tilde {p}}(\theta )\,d\theta \end{aligned}}}$

2

The objective of Bayesian model reduction is to compute the posterior ${\displaystyle {\tilde {p}}(\theta \mid y)}$ and evidence ${\displaystyle {\tilde {p}}(y)}$ of the reduced model from the posterior ${\displaystyle p(\theta \mid y)}$ and evidence ${\displaystyle p(y)}$ of the full model. Combining Equation 1 and Equation 2 and re-arranging, the reduced posterior ${\displaystyle {\tilde {p}}(\theta \mid y)}$ can be expressed as the product of the full posterior, the ratio of priors and the ratio of evidences:

${\displaystyle {\begin{aligned}{\frac {{\tilde {p}}(\theta \mid y){\tilde {p}}(y)}{p(\theta \mid y)p(y)}}&={\frac {p(y\mid \theta ){\tilde {p}}(\theta )}{p(y\mid \theta )p(\theta )}}\\\Rightarrow {\tilde {p}}(\theta \mid y)&=p(\theta \mid y){\frac {{\tilde {p}}(\theta )}{p(\theta )}}{\frac {p(y)}{{\tilde {p}}(y)}}\end{aligned}}}$

3

The evidence for the reduced model is obtained by integrating over the parameters of each side of the equation:

${\displaystyle \int {\tilde {p}}(\theta \mid y)\,d\theta =\int p(\theta \mid y){\frac {{\tilde {p}}(\theta )}{p(\theta )}}{\frac {p(y)}{{\tilde {p}}(y)}}\,d\theta =1}$

4

And by re-arrangement:

${\displaystyle {\begin{aligned}1&=\int p(\theta \mid y){\frac {{\tilde {p}}(\theta )}{p(\theta )}}{\frac {p(y)}{{\tilde {p}}(y)}}\,d\theta \\&={\frac {p(y)}{{\tilde {p}}(y)}}\int p(\theta \mid y){\frac {{\tilde {p}}(\theta )}{p(\theta )}}\,d\theta \\\Rightarrow {\tilde {p}}(y)&=p(y)\int p(\theta \mid y){\frac {{\tilde {p}}(\theta )}{p(\theta )}}\,d\theta \end{aligned}}}$

5

Under Gaussian prior and posterior densities, as are used in the context of variational Bayes, Bayesian model reduction has a simple analytical solution.^[1] First define normal densities for the priors and posteriors:

${\displaystyle {\begin{aligned}p(\theta )&=N(\theta$ ;\mu _{0},\Sigma _{0})\\{\tilde {p}}(\theta )&=N(\theta ;{\tilde {\mu }}_{0},{\tilde {\Sigma }}_{0})\\p(\theta \mid y)&=N(\theta ;\mu ,\Sigma )\\{\tilde {p}}(\theta \mid y)&=N(\theta ;{\tilde {\mu }},{\tilde {\Sigma }})\\\end{aligned}}}

6

where the tilde symbol (~) indicates quantities relating to the reduced model and subscript zero – such as ${\displaystyle \mu _{0}}$ – indicates parameters of the priors. For convenience we also define precision matrices, which are the inverse of each covariance matrix:

${\displaystyle {\begin{aligned}\Pi &=\Sigma ^{-1}\\\Pi _{0}&=\Sigma _{0}^{-1}\\{\tilde {\Pi }}&={\tilde {\Sigma }}^{-1}\\{\tilde {\Pi }}_{0}&={\tilde {\Sigma }}_{0}^{-1}\\\end{aligned}}}$

7

The free energy of the full model ${\displaystyle F}$ is an approximation (lower bound) on the log model evidence: ${\displaystyle F\approx \ln {p(y)}}$ that is optimised explicitly in variational Bayes (or can be recovered from sampling approximations). The reduced model's free energy ${\displaystyle {\tilde {F}}}$ and parameters ${\displaystyle ({\tilde {\mu }},{\tilde {\Sigma }})}$ are then given by the expressions:

${\displaystyle {\begin{aligned}{\tilde {F}}&={\frac {1}{2}}\ln |{\tilde {\Pi }}_{0}\cdot \Pi \cdot {\tilde {\Sigma }}\cdot \Sigma _{0}|\\&-{\frac {1}{2}}(\mu ^{T}\Pi \mu +{\tilde {\mu }}_{0}^{T}{\tilde {\Pi }}_{0}{\tilde {\mu }}_{0}-\mu _{0}^{T}\Pi _{0}\mu _{0}-{\tilde {\mu }}^{T}{\tilde {\Pi }}{\tilde {\mu }})+F\\{\tilde {\mu }}&={\tilde {\Sigma }}(\Pi \mu +{\tilde {\Pi }}_{0}{\tilde {\mu }}_{0}-\Pi _{0}\mu _{0})\\{\tilde {\Sigma }}&=(\Pi +{\tilde {\Pi }}_{0}-\Pi _{0})^{-1}\\\end{aligned}}}$

8

Consider a model with a parameter ${\displaystyle \theta }$ and Gaussian prior ${\displaystyle p(\theta )=N(0,0.5^{2})}$, which is the Normal distribution with mean zero and standard deviation 0.5 (illustrated in the Figure, left). This prior says that without any data, the parameter is expected to have value zero, but we are willing to entertain positive or negative values (with a 99% confidence interval [−1.16,1.16]). The model with this prior is fitted to the data, to provide an estimate of the parameter ${\displaystyle q(\theta )}$ and the model evidence ${\displaystyle p(y)}$.

To assess whether the parameter contributed to the model evidence, i.e. whether we learnt anything about this parameter, an alternative 'reduced' model is specified in which the parameter has a prior with a much smaller variance: e.g. ${\displaystyle {\tilde {p}}_{0}=N(0,0.001^{2})}$. This is illustrated in the Figure (right). This prior effectively 'switches off' the parameter, saying that we are almost certain that it has value zero. The parameter ${\displaystyle {\tilde {q}}(\theta )}$ and evidence ${\displaystyle {\tilde {p}}(y)}$ for this reduced model are rapidly computed from the full model using Bayesian model reduction.

The hypothesis that the parameter contributed to the model is then tested by comparing the full and reduced models via the Bayes factor, which is the ratio of model evidences:

${\displaystyle {\text{BF}}={\frac {p(y)}{{\tilde {p}}(y)}}}$

The larger this ratio, the greater the evidence for the full model, which included the parameter as a free parameter. Conversely, the stronger the evidence for the reduced model, the more confident we can be that the parameter did not contribute. Note this method is not specific to comparing 'switched on' or 'switched off' parameters, and any intermediate setting of the priors could also be evaluated.