BDA3 Chapter 2 Exercise 14

Here’s my solution to exercise 14, chapter 2, of Gelman’s Bayesian Data Analysis (BDA), 3rd edition. There are solutions to some of the exercises on the book’s webpage.

Suppose we have a normal prior θ∼normal(μ0,1τ0) and a normal sampling distribution y∣θ∼normal(θ,σ), where the variance is known. We will show by induction that the posterior is θ∣y1,…,yn∼normal(μn,1τn) where

1τ2n=1τ20+nσ2andμn=1τ20μ0+nσ2ˉyn1τ20+nσ2

for n=1,…,∞.

Base case

The case n=1 can be reexpressed as

1τ21=σ2+τ20σ2τ20andμ1=σ20μ0+τ20yσ2+τ20.

Now we can combine the fractions in the exponent, expand the brackets, collect the terms as θ-coefficients, rewrite the coefficients in terms of μ1 and τ1, then complete the square in terms of θ:

(y−θ)2σ2+(θ−μ0)2τ20=(y2+θ2−2θy)τ20+(θ2+μ20−2θμ0)σ2σ2τ20=θ2(τ20+σ2)−2θ(σ2μ0+τ20y)+(y2τ20+μ20σ2)σ2τ20=θ2(τ20+σ2)−2θμ1(σ2+τ20)+μ1(σ2+τ20)σ2τ20=θ21τ21−2θμ11τ21+μ11τ21=θ2−2μ1θ+μ21τ21=(θ−μ1)2τ21.

Induction step

The induction hypothesis is that the variance and mean are given by

1τ2n=1τ20+nσ2andμn=1τ20μ0+nσ2ˉy1τ20+nσ2

for some n≥1.

Starting with the variance, the base step can be reexpressed as

1τ2n+1=1τ2n+1σ2.

Now apply the induction hypothesis to get

1τ2n+1=1τ20+nσ2+1σ2=1τ20+n+1σ2.

For the mean we apply the same strategy. The base step can be reexpressed as

μn+1=1τ2nμn+1σ2yn+11τ2n+1σ2.

Applying the induction hypothesis then gives

μn+1=1τ2n(1τ20μ0+nσ2ˉyn1τ20+nσ2)+1σ2yn+11τ20+n+1σ2=1τ20μ0+nσ2ˉyn+1σ2yn+11τ20+n+1σ2=1τ20μ0+n+1σ2ˉyn+11τ20+n+1σ2,

since

(n+1)ˉyn+1:=n+1∑1yi=yn+1+n∑1yi=yn+1+nˉyn.