Let Y \in \mathbb{R}^{N \times D} represent N observations of D dimensions, and let X \in \mathbb{R}^{N \times Q} represent the corresponding latent inputs.
The generative process is defined as follows:
- for n = 1, 2\dots, N
- x_n \sim \mathcal{N}(x_n | 0, I)
- for d = 1, 2, \dots, D
-
f_d \sim \mathcal{N}(f_d | 0, K_X) , f_d \in \mathbb{R}^N , K_X = \left( k(x_i, x_j) \right) \in \mathbb{R}^{N \times N}
-
y_d \sim \mathcal{N}(y_d | 0, \sigma^2 I) , y_d \in \mathbb{R}^N
Here, y_d represents the d-th column vector of Y, and f_d is the corresponding latent vector of y_d.
The joint probability p(Y, X) = p(Y|X) p(X) can be expressed as:
\begin{align*}
p(Y|X) &= \prod_{d=1}^D \int p(y_d | f_d, \sigma^2 I) p(f_d| 0, K_X') df_d \\
&= \prod_{d=1}^D \mathcal{N}(y_d | 0, K_X) ,\quad K_X = K_X' + \sigma^2 I\\
&= \prod_{d=1}^D \frac{1}{(2 \pi)^{\frac{N}{2}} |K_X|^{\frac{1}{2}}} \exp \left( -\frac{1}{2} y_d^\top K_X^{-1} y_d \right) \\
&= \frac{1}{(2 \pi)^{\frac{ND}{2}} |K_X|^{\frac{D}{2}}} \exp \left( -\frac{1}{2} \sum_{d=1}^D y_d^\top K_X^{-1} y_d \right) \\
&= \frac{1}{(2 \pi)^{\frac{ND}{2}} |K_X|^{\frac{D}{2}}} \exp \left( -\frac{1}{2} \mathrm{tr}(K_X^{-1} Y Y^\top) \right) \\
p(X) &= \prod_{n=1}^N \mathcal{N}(x_n | 0, I) ,\quad x_n \in \mathbb{R}^Q \\
&= \prod_{n=1}^N \frac{1}{(2 \pi)^{\frac{Q}{2}}} \exp \left( -\frac{1}{2} x_n^\top x_n\right) ,\quad |I| = 1 \\
&= \frac{1}{(2 \pi)^{\frac{N Q}{2}}} \exp \left( -\frac{1}{2} \sum_{n=1}^N x_n^\top x_n\right) \\
&= \frac{1}{(2 \pi)^{\frac{N Q}{2}}} \exp \left( -\frac{1}{2} \mathrm{tr}(X X^\top) \right) \\
\end{align*}
The log probability is given by:
\begin{align*}
\ln p(Y, X) &= \ln p(Y|X) + \ln p(X) \\
&= - \frac{ND}{2} \ln (2 \pi) - \frac{D}{2} \ln |K_X| - \frac{1}{2} \mathrm{tr}(K_X^{-1} Y Y^\top)
-\frac{NQ}{2} \ln (2 \pi) - \frac{1}{2} \mathrm{tr}(X X^\top).
\end{align*}
To estimate X, we maximize the log likelihood \mathcal{L}(X) = \ln p(Y, X) with respect to X.
Discussion