# BDA3 Chapter 3 Exercise 1

Here’s my solution to exercise 1, chapter 3, of Gelman’s Bayesian Data Analysis (BDA), 3rd edition. There are solutions to some of the exercises on the book’s webpage.

Let $$y \mid \theta \sim \dmultinomial_J(\theta)$$ with prior $$\theta \sim \ddirichlet_J(\alpha)$$. We would like to find the marginal distribution of $$\phi := \frac{\theta_1}{\theta_1 + \theta_2}$$.

## Marginal posterior of Dirichlet-multinomial

As shown in the book, the posterior is $$\theta \mid y \sim \ddirichlet(y + \alpha)$$. The marginal posterior of $$(\theta_1, \theta_2) \mid y$$ can be written as

\begin{align} p(\theta_1, \theta_2 \mid y) &= \int_0^1 p(\theta \mid y) d\theta_3 \dotsm d\theta_{J - 1} \\ &\propto \theta_1^{y_1 + \alpha_1 - 1}\theta_2^{y_2 + \alpha_2 - 1} \int_0^1 \theta_3^{y_3 + \alpha_3 - 1} \dotsm \theta_{J - 1}^{y_{J - 1} + \alpha_{J - 1} - 1} \left(1 - \sum_1^{J - 1} \theta_j \right)^{y_J + \alpha_J - 1} d\theta_3 \dotsm d\theta_{J - 1} . \end{align}

The tricky part is calculating the integral part, which we define

$I := \int_0^1 \theta_3^{y_3 + \alpha_3 - 1} \dotsm \theta_{J - 1}^{y_{J - 1} + \alpha_{J - 1} - 1} \left(1 - \sum_1^{J - 1} \theta_j \right)^{y_J + \alpha_J - 1} d\theta_3 \dotsm d\theta_{J - 1} .$

To calculate $$I$$, first note that

\begin{align} \int_0^1 \theta^s \left( c - \theta \right)^t d\theta &= \int_0^1 \theta^s \left( 1 - \frac{\theta}{c} \right)^t c^t d\theta \\ &= \int_0^1 \left( \frac{\theta}{c} \right)^s \left( 1 - \frac{\theta}{c} \right)^t c^{s + t} d\theta \\ &= \int_0^1 \phi^s \left( 1 - \phi \right)^t c^{s + t + 1} d\phi , \quad \phi := \frac{\theta}{c} \\ &= B(s + 1, t + 1) c^{s + t + 1} , \end{align}

if $$c$$ is not a function of $$\theta$$. With $$c := 1 - \sum_1^{J - 2} \theta_j$$,

\begin{align} I &= \int_0^1 \theta_3^{y_3 + \alpha_3 - 1} \dotsm \theta_{J - 2}^{y_{J - 2} + \alpha_{J - 2} - 1} \theta_{J - 1}^{y_{J - 1} + \alpha_{J - 1} - 1} \left( 1 - \sum_1^{J - 2} \theta_j - \theta_{J - 1} \right)^{y_J + \alpha_J - 1} d\theta_3 \dotsm d\theta_{J - 1} \\ &= \int_0^1 \theta_3^{y_3 + \alpha_3 - 1} \dotsm \theta_{J - 2}^{y_{J - 2} + \alpha_{J - 2} - 1} \left( \int_0^1 \theta_{J - 1}^{y_{J - 1} + \alpha_{J - 1} - 1} \left(c - \theta_{J - 1} \right)^{y_J + \alpha_J - 1} d\theta_{J - 1} \right) d\theta_3 \dotsm d\theta_{J - 2} \\ &\propto \int_0^1 \theta_3^{y_3 + \alpha_3 - 1} \dotsm \theta_{J - 2}^{y_{J - 2} + \alpha_{J - 2} - 1} \left( c \right)^{y_{J-1} + y_J + \alpha_{J-1} + \alpha_J - 1} d\theta_3 \dotsm d\theta_{J - 2} \\ &= \int_0^1 \theta_3^{y_3 + \alpha_3 - 1} \dotsm \theta_{J - 2}^{y_{J - 2} + \alpha_{J - 2} - 1} \left( 1 - \sum_1^{J-2} \theta_j \right)^{y_{J-1} + y_J + \alpha_{J-1} + \alpha_J - 1} d\theta_3 \dotsm d\theta_{J - 2} . \end{align}

Continuing by induction,

$I = \left( 1 - \theta_1 - \theta_2 \right) ^ {\sum_3^J (y_j + \alpha_j) - 1} .$

Now that we have the integral part, the marginal posterior can be written

$p(\theta_1, \theta_2 \mid y) \propto \theta_1^{y_1 + \alpha_1 - 1}\theta_2^{y_2 + \alpha_2 - 1} \left( 1 - \theta_1 - \theta_2 \right) ^ {\sum_3^J (y_j + \alpha_j) - 1} .$

This has the form of a Dirichlet distribution, so the marginal posterior is

$\left( \theta_1, \theta_2, 1 - \theta_1 - \theta_2 \right) \mid y \sim \ddirichlet\left(y_1 + \alpha_1, y_2 + \alpha_2, \sum_3^J (y_j + \alpha_j) \right) .$

## Change of variables

Now define $$(\phi_1, \phi_2) := (\frac{\theta_1}{\theta_1 + \theta_2}, \theta_1 + \theta_2)$$, so that $$(\theta_1, \theta_2) = (\phi_1\phi_2, \phi_2 - \phi_1\phi_2)$$. The Jacobian of this transformation is

$\begin{vmatrix} \frac{\partial\theta_1}{\partial\phi_1} & \frac{\partial\theta_1}{\partial\phi_2} \\ \frac{\partial\theta_2}{\partial\phi_1} & \frac{\partial\theta_2}{\partial\phi_2} \end{vmatrix} = \begin{vmatrix} \phi_2 & \phi_1 \\ -\phi_2 & 1 - \phi_1 \end{vmatrix} = \phi_2 .$

Therefore, the probability distribution of the new variables is

\begin{align} p(\phi_1, \phi_2 \mid y) &= (\phi_1\phi_2)^{y_1 + \alpha_1 - 1} (\phi_2 (1 - \phi_1))^{y_2 + \alpha_2 - 1} (1 - \phi_2)^{\sum_3^J (y_j + \alpha_j) - 1} \frac{1}{\phi_2} \\ &= \phi_1^{y_1 + \alpha_1 - 1} (1 - \phi_1)^{y_2 + \alpha_2 - 1} \phi_2^{y_1 + y_2 + \alpha_1 + \alpha_2 - 3} (1 - \phi_2)^{\sum_3^J (y_j + \alpha_j) - 1} \\ &= p(\phi_1 \mid y) p(\phi_2 \mid y) , \end{align}

where

\begin{align} \phi_1 \mid y &\sim \dbeta(y_1 + \alpha_1, y_2 + \alpha_2 ) \\ \phi_2 \mid y &\sim \dbeta\left(y_1 + y_2 + \alpha_1 + \alpha_2 - 2, \sum_3^J (y_j + \alpha_j)\right) . \end{align}

The marginal posterior for $$\phi_1$$ is equivalent to the posterior obtained from a $$\phi_1 \sim \dbeta(\alpha_1, \alpha_2)$$ prior with a $$y_1 \mid \phi_1 \sim \dbinomial(y_1 + y_2, \phi_1)$$ likelihood.