Formulas and Proofs for Conditional Expectation and Variance

Formulas to be Derived

Here are the formulas to be derived.

$$\begin{aligned} \mathbb{E}_{X}[X] & = \mathbb{E}_{Y}\left[\mathbb{E}_{X \mid Y}[X]\right] \\ \mathbb{V}_{X}[X] & = \mathbb{E}_{Y}\left[V_{X \mid Y}[X]\right] + V_{Y}\left[\mathbb{E}_{X \mid Y}[X]\right] \end{aligned}$$

Deriving the Formula for Conditional Expectation

For simplicity, we assume here that the order of integration can be swapped.

$$\begin{aligned}\mathbb{E}_{X}[g(x)] & = \int g(x) f_{X}(x) d x \\ & = \iint g(x) f_{X, Y}(x, y) d y d x \\ & =\iint g(x) \frac{f_{X, Y}(x, y)}{f_{Y}(y)} f_{Y}(y) d y d x \\ & = \iint g(x) \frac{f_{X, Y}(x, y)}{f_{Y}(y)} f_{Y}(y) d x d y \\ & = \int\left\{\int g(x) \frac{f_{X, Y}(x, y)}{f_{Y}(y)} d x\right\} f_{Y}(y) d y \\ & = \int \mathbb{E}_{X|Y}[g(x)] f_{Y}(y) d y \\ & = \mathbb{E}_{Y}\left[\mathbb{E}_{X|Y}[g(x)]\right] \end{aligned}$$

The following transformation appears in the middle, utilizing the fact that the right-hand side is the definition of the left-hand side.

$$f_{X|Y}(x) = \frac{f_{X, Y}(x, y)}{f_{Y}(y)}$$

Using the validity of the above equation, you can see that the following equality holds.

$$\int g(x) \frac{f_{X, Y}(x, y)}{f_{Y}(y)} d x = \mathbb{E}_{X|Y}[g(x)]$$

Deriving the Formula for Conditional Variance

The second formula can be derived using the first formula for conditional expectation.

$$\begin{aligned} \mathbb{V}_{X}[X] & = \mathbb{E}\left[X^{2}\right]-\mathbb{E}[X]^{2} \\ & = \mathbb{E}_{Y}\left[\mathbb{E}_{X \mid Y}\left[X^{2}\right]\right] - \mathbb{E}_{Y}\left[\mathbb{E}_{X \mid Y}[X]\right]^{2} \\ & = \mathbb{E}_{Y}\left[V_{X \mid Y}[X] + \mathbb{E}_{X \mid Y}[X]^{2}\right] - \mathbb{E}_{Y}\left[\mathbb{E}_{X \mid Y}[X]\right]^{2} \\ & = \mathbb{E}_{Y}\left[V_{X \mid Y}[X]\right] + \mathbb{E}_{Y}\left[\mathbb{E}_{X \mid Y}[X]^{2}\right] - \mathbb{E}_{Y}\left[\mathbb{E}_{X \mid Y}[X]\right]^{2} \\ & = \mathbb{E}_{Y}\left[V_{X \mid Y}[X]\right] + V_{Y}\left[\mathbb{E}_{X \mid Y}[X]\right] \end{aligned}$$

The last transformation is slightly difficult to follow.

$$+ \mathbb{E}_{Y}\left[\mathbb{E}_{X \mid Y}[X]^{2}\right] - \mathbb{E}_{Y}\left[\mathbb{E}_{X \mid Y}[X]\right]^{2}$$

It may be easier to understand by viewing the above part as follows.

$$+ \mathbb{E}_{Y}\left[ h(Y)^{2} \right] - \mathbb{E}_{Y}\left[ h(Y) \right]^{2}$$

Note that the conditional expectation $\mathbb{E}_{X \mid Y}[X]$ becomes an expression of $Y$ because it is integrated over $X$.