An Introduction to Statistical Signal Processing

196 CHAPTER 4. EXPECTATION AND AVERAGES

The qualification “measurable” is needed in the theorem to guarantee the existence of the expectation. Measurability is satisfied by almost any function that you can think of and, for all practical purposes, can be neglected.

As a simple example of the use of this formula, consider a random variable X with a uniform pdf on [−1/2, 1/2]. Define the random variable Y = X2, that is g(r) = r2. We can use the derived distribution formula

(3.40) to write

fY (y) = y−1/2fX(y1/2) ; y ≥ 0 ,

and hence

fY (y) = y−1/2 ; y (0, 1/4] ,

where we have used the fact that fX(y1/2) is 1 only if the nonnegative argument is less than 1/2 or y ≤ 1/4. We can then find EY as

Note that the result is the same for each method. However, the second calculation is much simpler, especially if one considers the work which has already been done in chapter 3 in deriving the density formula for the square of a random variable.

[4.5] A second example generalizes an observation of chapter 2 and shows that expectations can be used to express probabilities (and hence that probabilities can be considered as special cases of expectation). Recall that the indicator function of an event F is defined by

1 if x F

1F (x) = 0 otherwise .

The probability of the event F can be written in the following form which is convenient in certain computations:

E1F (X) = 1F (x) dFX(x) = dFX(x) = PX(F ) , (4.10)

F

where we have used the universal Stieltjes integral representation of (3.32) to save writing out both sums of pmf’s and integrals of pdf’s

Repeated di erentiation can be used to show more generally that the kth moment can be found as

If one needs several moments of a given random variable, it is usually easier to do one integration to find the characteristic function and then several di erentiations than it is to do the several integrations necessary to find the moments directly. Note that if we make the substitution w = ju and di erentiate with respect to w, instead of u,

dkk MX(w)|w=0 = E(Xk) . dw

Because of this property, characteristics function with ju = w are called moment-generating functions. From the defining sum or integral for characteristic functions in example [4.6], the moment-generating function may not exist for all w = v + ju, even when it exists for all w = ju with u real. This is a variation on the idea that a Laplace transform might not exist for all complex frequencies s = σ + jω even when the it exists for all s = jω with ω real, that is, the Fourier transform exists.

Example [4.6] illustrates an obvious extension of the fundamental theorem of expectation. In [4.6] the complex function is actually a vector function of length 2. Thus it is seen that the theorem is valid for vector functions, g(x), as well as for scalar functions, g(x). The expectation of a vector is simply the vector of expected values of the components.

As a simple example, recall from (3.112) that the characteristic function of a binary random variable X with parameter p = pX(1) = 1 − pX(0) is

The relationship between the characteristic function of a distribution and the moments of a distribution becomes particularly striking when the

characteristic function is su ciently nice near the origin to possess a Taylor series expansion. The Taylor series of a function f(u) about the point u = 0 has the form

where the derivatives

f(k)(0) = dkk f(u)|u=0 ; du

are assumed to exist, that is, the function is assumed to be analytic at the origin. Combining the Taylor series expansion with the moment-generating property (4.12) yields

This result has an interesting implication: knowing all of the moments of the random variable is equivalent to knowing the behavior of the characteristic function near the origin. If the characteristic function is su ciently well behaved for the Taylor series to be valid over the entire range of u rather than just in the area around 0, then knowing all of the moments of a random variable is su cient to know the transform. Since the transform in turn implies the distribution, this guarantees that knowing all of the moments of a random variable completely describes the distribution. This is true, however, only when the distribution is su ciently “nice,” that is, when the technical conditions ensuring the existence of all of the required derivatives and of the convergence of the Taylor series hold.

The approximation of (4.15) plays an important role in the central limit theorem, so it is worth pointing out that it holds under even more general conditions than having an analytic function. In particular, if X has a second moment so that E[X2] < ∞, then

where o(u2) contains higher order terms that go to zero as u → 0 faster than u2. See, for example, Breiman’s treatment of characteristic functions [6].

The most important application of the characteristic function is its use in deriving properties of sums of independent random variables, as was be seen in (3.111).

4.3Functions of Several Random Variables

Thus far expectations have been considered for functions of a single random variable, but it will often be necessary to treat functions of multiple random variables such as sums, products, maxima, and minima. For example, given random variables U and V defined on a common probability space we might wish to find the expectation of Y = g(U, V ). The fundamental theorem of expectation has a natural extension (which is proved in the same way).

Theorem 4.2 Fundamental Theorem of Expectation for Functions of Several Random Variables

Given random variables X0, X1, . . . , Xk−1 described by a cdf FX0,X1,... ,Xk−1 and given a measurable function g : k → ,

E[g(X0, . . . , Xk−1)]

=g(x0, . . . , xk−1) dFX0,... ,Xk−1 (x0, . . . , xk−1)

As examples of expectation of several random variables we will consider correlation, covariance, multidimensional characteristic functions, and differential entropy. First, however, we develop some simple and important properties of expectation that will be needed.

4.4Properties of Expectation

Expectation possesses several basic properties that will prove useful. We now present these properties and prove them for the discrete case. The continuous results follow by using integrals in place of sums.

Property 1. If X is a random variable such that Pr(X ≥ 0) = 1, then

EX ≥ 0.

Proof. Pr(X ≥ 0) = 1 implies that the pmf pX(x) = 0 for x < 0. If pX(x) is nonzero only for nonnegative x, then the sum defining the expectation contains only terms xpX(x) ≥ 0, and hence them sum and EX are nonnegative. Note that property 1 parallels Axiom 2.1 of probability. That is, the nonnegativity of probability measures implies property 1.

Property 2. If X is a random variable such that for some fixed number r, Pr(X = r) = 1, then EX = r. Thus the expectation of a constant equals the constant.

Proof. Pr(X = r) = 1 implies that pX(r) = 1. Thus the result follows from the definition of expectation. Observe that property 2 parallels Axiom 2.2 of probability. That is, the normalization of the total probability to 1 leaves the constant unscaled in the result. If total probability were di erent from 1, the expectation of a constant as defined would be a di erent, scaled value of the constant.

Property 3. Expectation is linear; that is, given two random variables X and Y and two real constants a and b,

E(aX + bY ) = aEX + bEY .

Proof. For simplicity we focus on the discrete case, the proof for pdf’s is the obvious analog. Let g(x, y) = ax + by, where a and b are constants. In this case the fundamental theorem of expectation for functions of several (here two) random variables implies that

Using the consistency of marginal and joint pmf’s of (3.13)–(3.14) this becomes

Keep in mind that this result has nothing to do with whether or not the random variables are independent.

The linearity of expectation follows from the additivity of probability. That is, the summing out of joint pmf’s to get marginal pmf’s in the proof was a direct consequence of Axiom 2.4 . The alert reader will likely have noticed the method behind the presentation of the properties of expectation: each follows directly from the corresponding axiom of probability. Furthermore, using (4.10), the converse is true: That is, instead of starting with the axioms of probability, suppose we start by using the properties of expectation as the axioms of expectation. Then the axioms of probability become the derived properties of probability. Thus the first three axioms of probability and the first three properties of expectation are dual; one can start with either and get the other. One might suspect that to get a useful theory based on expectation, one would require a property analogous to Axiom 2.4 of probability, that is, a limiting form of expectation property 3. This is, in fact, the case, and the fourth basic property of expectation is the countably infinite version of property 3. When dealing with expectations, however, the fourth property is more often stated as a continuity property, that is, in a form analogous to Axiom 2.4 of probability given in equation (2.28). For reference we state the property below without proof:

Property 4. Given an increasing sequence of nonnegative random variables Xn; n = 0, 1, 2, . . . , that is, Xn ≥ Xn−1 for all n (i.e., Xn(ω) ≥ Xn−1(ω) for all ω Ω), which converge to a limiting random variable X = limn→∞ Xn, then

E lim Xn = lim EXn .

n→∞ n→∞

Thus as with probabilities, one can in certain cases exchange the order of limits and expectation. The cases include but are not limited to those of property 4. Property 4 is called the monotone convergence theorem and is one of the basic properties of integration as well as expectation. This theorem is discussed in appendix B along with another important limiting result, the dominated convergence theorem.

In fact, the four properties of expectation can be taken as a definition of an integral (viz., the Stieltjes integral) and used to develop the general Lebesgue theory of integration. That is, the theory of expectation is really just a specialization of the theory of integration. The duality between probability and expectation is just a special case of the duality between measure theory and the theory of integration.

4.5. EXAMPLES: FUNCTIONS OF SEVERAL RANDOM VARIABLES203

4.5Examples: Functions of Several Random Variables

4.5.1Correlation

We next introduce the idea of correlation or expection of products of random variables that will lead to the development of a property of expectation that is special to independent random variables. A weak form of this property will be seen to provide a weak form of independence that will later be useful in characterizing certain random processes. Correlations will later be seen to play a fundamental role in many signal processing applications. Suppose we have two independent random variables X and Y and we have two functions or measurements on these random variables, say g(X) and h(Y ), where g : → , h : → , and E[g(X)] and E[h(Y )] exist and are finite. Consider the expected value of the product of these two functions, called the correlation between g(X) and h(Y ). Applying the two-dimensional vector case of the fundamental theorem of expectation to discrete random variables results in

x,y

=g(x)h(y)pX(x)pY (y)

xy

=(E(g(X)))(E(h(Y ))) .

A similar manipulation with integrals shows the same to be true for random variables possessing pdf’s. Thus we have proved the following result, which we state formally as a lemma.

Lemma 4.1 For any two independent random variables X and Y ,

E(g(X)h(Y )) = (Eg(X))(Eh(Y )) (4.18)

for all functions g and h with finite expectation.

By stating that the functions have finite expectation we implicitly assume them to be measurable, i.e., to have a distribution with respect to which we can evaluate an expectation. Measurability is satisfied by almost all functions so that the qualification can be ignored for all practical purposes.

To cite the most important example, if g and h are identity functions (h(r) = g(r) = r), then we have that independence of X and Y implies that

that is, the correlation of X and Y is the product of the means, in which case the two random variables are said to be uncorrelated. (The term linear independence is sometimes used as a synonym for uncorrelated.)

We have shown that if two discrete random variables are independent, then they are also uncorrelated. Note that independence implies not only that two random variables are uncorrelated but also that all functions of the random variables are uncorrelated — a much stronger property. In particular, two uncorrelated random variables need not be independent. For example, consider two random variables X and Y with the joint pmf

A simple calculation shows that

E(XY ) = 1/4(1 − 1) + 1/2(0) = 0

and

(EX)(EY ) = (0)(1/2) = 0 ,

and hence the random variables are uncorrelated. They are not, however, independent. For example, Pr(X = 0|Y = 0) = 1 while Pr(X = 0) = 1/2. As another example, consider the case where pX(x) = 1/3 for x = −1, 0, 1 and Y = X2. X and Y are correlated but not independent.

Thus uncorrelation does not imply independence. If, however, all possible functions of the two random variables are uncorrelated — that is, if (4.18) holds — then they must be independent. To see this in the discrete case, just consider all possible functions of the form 1a(x), that is, indicator functions of all of the points. (1a(x) is 1 if x = a and zero otherwise.) Let g = 1a and h = 1b for a in the range space of X and b in the range space of Y . It follows from (4.18) and (4.10) that

pX,Y (a, b) = pX(a)pY (b) .

Obviously the result holds for all a and b. Thus the two random variables are independent. It can now be seen that (4.18) provides a necessary and su cient condition for independence, a fact we formally state as a theorem.