An Introduction to Statistical Signal Processing

and show that a is a k × k identity matrix, d is an m × m identity matrix, and that c and d contain all zeros so that the right hand matrix is indeed an identity matrix.

The conditional pdf for Y given X follows directly from the definitions

as

fY |X(y|x)

Again using some brute force linear algebra, it can be shown that the quadratic terms in the exponential can be expressed in the form

=(y − mY − KY XKX−1(x − mX))tKY−|1X(y − mY − KY XKX−1(x − mX)).

which shows that conditioned on X = x, Y has a Gaussian density. This means that we can immediately recognize the conditional expectation of Y given X as

so that the conditional expectation is an a ne function of the vector x. We can also infer from the form that KY |X is the (conditional) covariance

which unlike the conditional mean does not depend on the vector x! Furthermore, since we know how the normalization must relate to the covariance matrix, we have that

These relations completely describe the conditional densities of one subvector of a Gaussian vector given another subvector. We shall see, however, that the importance of these results goes beyond the above evaluation and provides some fundamental results regarding optimal nonlinear estimation for Gaussian vectors and optimal linear estimation in general.

4.8Expectation as Estimation

Suppose that one is asked to guess the value that a random variable Y will take on, knowing the distribution of the random variable. What is the

ˆ

best guess or estimate, say Y ? Obviously there are many ways to define a best estimate, but one of the most popular ways to define a cost or

called mean squared error or MSE. Many arguments have been advanced in support of this approach, perhaps the simplest being that if one views the error as a voltage, then the average squared error is the average energy in the error. The smaller the energy, the weaker the signal in some sense. Perhaps a more honest reason for the popularity of the measure is its tractability in a wide variety of problems, it often leads to nice solutions that indeed work well in practice. As an example, we show that the optimal estimate of the value of an unknown random variable is in fact the mean of the random variable, a result that is highly intuitive. Rather than use calculus to prove this result — a tedious approach requiring setting derivatives to zero and then looking at second derivatives to verify that indeed the stationary point is a minimum — we directly prove the global optimality of the result.

ˆ

Suppose that that our estimate is Y = a, some constant. We will show that this estimate can never have mean squared error smaller than that resulting from using the expected value of Y as an estimate. This is accomplished by a simple sequence of equalities and inequalities. Begin by adding and subtracting the mean, expanding the square, and using the second and third properties of expectation as

E[(Y − a)2] = E[(Y − EY + EY − a)2]

= E[(Y − EY )2] + 2E[(Y − EY )(EY − a)] + (EY − a)2.

The cross product is evaluated using the linearity of expectation and the fact that EY is a constant as

E[(Y − EY )(EY − a)] = (EY )2 − aEY − (EY )2 + aEY = 0

and hence from Property 1 of expectation,

E[(Y − a)2] = E[(Y − EY )2] + (EY − a)2 ≥ E[(Y − EY )2], (4.45)

which is the mean squared error resulting from using the mean of Y as an estimate. Thus the mean of a random variable is the minimum mean squared error estimate (MMSE) of the value of a random variable in the absence of any a priori information.

What if one is given a priori information? For example, suppose that now you are told that X = x. What then is the best estimate of Y , say

ˆ

Y (X)? This problem is easily solved by modifying the previous derivation to use conditional expectation, that is, by using the conditional distribution for Y given X instead of the a priori distribution for Y . Once again we try to minimize the mean squared error:

x

Each of the terms in the sum, however, is just a mean squared error between a random variable and an estimate of that variable with respect to a distribution, here the conditional distribution pY |X(·|x). By the same argument as was used in the unconditional case, the best estimate is the mean, but now the mean with respect to the conditional distribution, i.e.,

| ˆ

E(Y x). In other words, for each x the best Y (x) in the sense of minimizing the mean squared error is E(Y |x). Plugging in the random variable X in place of the dummy variable x we have the following interpretation

The conditional expectation E(Y |X) of a random variable Y given a random variable X is the minimum mean squared estimate of Y given X.

A direct proof of this result without invoking the conditional version of the result for unconditional expectation follows from general iterated expectation. Suppose that g(X) is an estimate of Y given X. Then the resulting mean squared error is

E[(Y − g(X))2] = E[(Y − E(Y |X) + E(Y |X) − g(X))2]

=E[(Y − E(Y |X))2]

−2E[(Y − E(Y |X))(E(Y |X) − g(X))] +E[(E(Y |X) − g(X))2].

stationary random process, say X = (Xn−1, Xn−2, . . . , Xn−k) and Y is the next, or “future”, sample Y = Xn. In this case the goal is to find the best one-step predictor given the finite past. In the second example, Y is a rectangular subblock of pixels in a sampled image intensity raster and X consists of similar subgroups above and to the left of Y . Here the goal is to use portions of an image already coded or processed to predict a new portion of the same image. This vector prediction problem is depicted in Figure 4.1 where subblocks A, B, and C would be used to predict subblock D.

A B

CD

Figure 4.1: Vector Prediction of Image Subblocks

The following theorem shows that the best nonlinear predictor of Y given X is simply the conditional expectation of Y given X. Intuitively, our best guess of an unknown vector is its expectation or mean given whatever observations that we have. This extends the interpretation of a conditional expectation as an optimal estimator to the vector case.

Theorem 4.5 Given two random vectors Y and X, the minimum mean squared error estimate of Y given X is

Proof: As in the scalar case, the proof does not require calculus or

ˆ

Lagrange minimizations. Suppose that Y is the claimed optimal estimate

˜ ˜

and that Y is some other estimate. We will show that Y must yield a mean

ˆ

squared error no smaller than does Y . To see this consider

− ˆ |

E[(Y Y ) X] = 0.

as claimed, which proves the theorem.

As in the scalar case, the conditional expectation is in general a di cult function to evaluate with the notable exception of jointly Gaussian vectors. Recall that (4.41)–(4.44) the conditional pdf for jointly Gaussian vectors Y

and X with K(X,Y ) = E[((Xt, Y t)−(mtX −mtY ))t((Xt, Y t)−(mtX −mtY ))], KY = E[(Y − mY )(Y − mY )t], KX = E[(X − mX)(X − mX)t], KXY =

E[(X − mX)(Y − mY )t], KY X = E[(Y − mY )(Y − mY )t] is

which is an a ne (linear plus constant) function of X! The resulting mean squared error is (using iterated expectation)

Expanding the MSE and using some linear algebra results in MMSE(A, b)

=Tr &(Y − AX − b)(Y − AX − b)t'

=Tr ((Y − mY + A(X − mX) − b + mY + AmX)'

× (Y − mY + A(X − mX) − b + mY + AmX)t

=Tr &KY − AKXY − KY XAt + AKXAt' +(b − mY − AmX)t(b − mY − AmX)

where all the remaining cross terms are zero. Regardless of A the final term is nonnegative and hence it is bound below by 0, a minimum achieved by the choice

(just expand this expression to verify it is the same as the previous ex-

and the minimimum is achieved by A = KY XKX−1 and b = E(Y )+AE(X).