BDA3 Chapter 14 Exercise 3

Here’s my solution to exercise 3, chapter 14, of Gelman’s Bayesian Data Analysis (BDA), 3rd edition. There are solutions to some of the exercises on the book’s webpage.

\(\DeclareMathOperator{\dbinomial}{Binomial} \DeclareMathOperator{\dbern}{Bernoulli} \DeclareMathOperator{\dpois}{Poisson} \DeclareMathOperator{\dnorm}{Normal} \DeclareMathOperator{\dt}{t} \DeclareMathOperator{\dcauchy}{Cauchy} \DeclareMathOperator{\dexponential}{Exp} \DeclareMathOperator{\duniform}{Uniform} \DeclareMathOperator{\dgamma}{Gamma} \DeclareMathOperator{\dinvgamma}{InvGamma} \DeclareMathOperator{\invlogit}{InvLogit} \DeclareMathOperator{\logit}{Logit} \DeclareMathOperator{\ddirichlet}{Dirichlet} \DeclareMathOperator{\dbeta}{Beta}\)

We need to reexpress \((y - X\beta)^T (y - X\beta)\) as \((\mu - \beta)^T \Sigma^{-1} (\mu - \beta)\), for some \(\mu\), \(\Sigma\). Using the QR-decomposition of \(X = QR\), we see

\[ \begin{align} (y - X\beta)^T(y - X\beta) &= (Q^T(y - X\beta))^TQ^T(y - X\beta) \\ &= (Q^Ty - Q^TX\beta)^T (Q^Ty - Q^TX\beta) \\ &= (Q^Ty - R\beta)^T (Q^Ty - R\beta) , \end{align} \]

where \(Q\) is orthogonal and \(R\) an invertible upper triangular matrix. We can read off the minimum of this quadratic form as

\[ \hat\beta = R^{-1}Q^Ty , \]

which shows that \(\mu = \hat\beta = R^{-1}Q^Ty\). Note that

\[ \begin{align} (X^TX)^{-1}X^T &= (R^TR)^{-1}R^T Q^T \\ &= R^{-1}R^{-T}R^T Q^T \\ &= R^{-1}Q^T \end{align} \]

so that \(\hat\beta = (X^TX)^{-1}X^Ty\).

Expanding the brackets of both quadratic form expressions and comparing the quadratic coefficients, we see that

\[ \Sigma^{-1} = R^T R = X^T X , \]

which shows that \(V_\beta = (X^T X)^{-1}\), in the notation of page 355.