An Introduction to Statistical Signal Processing

136 CHAPTER 3. RANDOM OBJECTS

of minimizing the probability of error is the most probable value of X given the observation. This is called the maximum a posteriori or MAP decision rule. In our binary example it reduces to choosing xˆ = y if * < 1/2 and xˆ = 1 − y if * > 1/2. If * = 1/2 you can give up and flip a coin or make an arbitrary decision. (Why?) Thus the minimum (optimal) error probability over all possible rules is min(*, 1 − *).

The astute reader will notice that having introduced conditional pmf’s pY |X, the example considered the alternative pmf pX|Y . The two are easily related by Bayes’ rule (3.49).

A generalization of the simple binary detection problem provides the typical form of a statistical classification system. Suppose that Nature selects a “class” H, a random variable described by a pmf pH(h), which is no longer assumed to be binary. Once the class is selected, Nature then generates a random “observation” X according to a pmf pX|H. For example, the class might be a medical condition and the observations the results of blood pressure, patients age, medical history, and other information regarding the patients health. Alternatively, the class might be an “input signal” put into a noisy channel which has the observation X as an “output signal.” The

ˆ

question is: Given the observation X = x, what is the best guess H(x) of the unseen class? If by “best” we adopt the criterion that the best guess is

ˆ

the one that minimizes the error probability Pe = Pr(H(X) = H), then the optimal classifer is again the MAP rule argmaxupH|X(u|x). More generally we might assign a cost Cy,h resulting if the true class is h and we guess y. Typically it is assumed that Ch,h = 0, that is, the cost is zero if our guess is correct. (In fact it can be shown that this assumption involves no real

x,h

which reduces to the probability of error if the cost function is given by

The optimal classifier in the sense of minimizing the Bayes risk is then found by observing that the inequality

the minimum average Bayes risk classifier. This reduces to the MAP detection rule when Cy,h = 1 − δy,h.

3.9Additive Noise

The next examples of the use of conditional distributions treats the distributions arising when one random variable (thought of as a “noise” term) is added to another, independent random variable (thought of as a “signal” term). This is an important example of a derived distribution problem that yields an interesting conditional probability. The problem also suggests a valuable new tool which will provide a simpler way of solving many similar derived distributions — the characteristic function of random variables.

Discrete Additive Noise

Consider two independent random variables X and W and form a new random variable Y = X + W . For example, this could be a description of how errors are actually caused in a noisy communication channel connecting a binary information source to a user. In order to apply the detection and classification signal processing methods, we must first compute the appropriate conditional probabilities of the outpout Y given the input X. To do this we begin by computing the joint pmf of X and Y using the inverse image formula:

pX,Y (x, y) = Pr(X = x, Y = y)

=Pr(X = x, X + W = y)

=pX,W (α, β)

Note that this formula only makes sense if y − x is one of the values in the range space of W . Thus from the definition of conditional pmf’s:

an answer that should be intuitive: given the input is x, the output will equal a certain value y if and only if the noise exactly makes up the di erence, i.e., W = y − x. Note that the marginal pmf for the output Y can be found by summing the joint probability:

x

a formula that is known as a discrete convolution or convolution sum. Anyone familiar with convolutions know that they can be unpleasant to

evaluate, so we postpone further consideration to the next section and turn to the continuous analog.

The above development assumed ordinary arithmetic, but it is worth pointing out that for discrete random variables sometimes other types of arithmetic are appropriate, e.g., modulo 2 arithmetic for binary random variables. The binary example of section 3.8 can be considered as an addi-

tive noise example if we define a random variable W which is independent of X and has a pmf pW (w) = *w(1−*)1−w; w = 0, 1 and where Y = X +W is interpreted as modulo 2 arithmetic, that is, as Y = X W . This additive noise definition is easily seen to yield the conditional pmf of (3.64) and the output pmf via a convolution. To be precise,

x

a modulo 2 convolution.

Continuous Additive Noise

An entirely analogous formula arises in the continous case. Again suppose that X is a random variable, a signal, with pdf fX, and that W is a random variable, the noise, with pdf fW . The random variables X and W are assumed to be independent. Form a new random variable Y , an observed signal plus noise. The problem is to find the conditional pdf’s fY |X(y|x) and fX|Y (x|y). The operation of producing an output Y from an input signal X is called an additive noise channel in communications systems. The channel is completely described by fY |X. The second pdf, fX|Y will prove useful later when we try to estimate X given an observed value of Y .

Independence of X and W implies that the joint pdf is fX,W (x, w) = fX(x)fW (w). To find the needed joint pdf fX,Y , first evaluate the joint cdf and then take the appropriate derivative. The cdf is a straightforward derived distribution problem:

FX,Y (x, y) = Pr(X ≤ x, Y ≤ y)

=Pr(X ≤ x, X + W ≤ y)

=dαfX(α)FW (y − α).

−∞

Taking the derivatives yields

fX,Y (x, y) = fX(x)fW (y − x)

and hence

The marginal pdf for the sum Y = X + W is then found as

a convolution integral of the pdf’s fX and fW , analogous to the convolution sum found when adding independent independent discrete random variables. Thus the evaluation of the pdf of the sum of two independent continuous random variables is the same as the evaluation of the output of a linear system with an input signal fX and an impulse response fW .

We will later see an easy way to accomplish this using transforms The pdf fX|Y follows from Bayes’ rule:

It is instructive to work through the details of the previous example for the special case of Gaussian random variables. For simplicity the means are assumed to be zero and hence it is assumed that fX is N(0, σX), that fW is N(0, σY2 ), and that as in the Example X and W are independent and Y = X + W . From (3.78)

from which the conditional pdf can be immediately recognized as being Gaussian with mean x and variance σW2 , that is, as N(x, σW2 ).

To evlauate the pdf fX|Y using Bayes’ rule, we begin with the denominator fY of (3.54) and write

∞

This convolution of two Gaussian “signals” can be accomplished using an old trick called “completing the square.” Call the integral in the square brackets at the end of the above equation I and note that integrand resem-

bles

e−12 ( ασ−2m )2

which we know from (B.15) in appendix B integrates to

since a Gaussian pdf integrates to 1. The trick is to modify I to resemble this integral with an additional factor. Compare the two exponents:

The exponent from I will equal the left two terms of the expanded exponent in the known integral if we choose

where the addition of the leftmost term is called “completing the square.” With this identification and again using (3.85) – (3.86) we have that

two zero mean independent Gaussian random variables is another zero mean Gaussian random variable with variance equal to the sum of the variances of the two random variables being added.

Finally we turn to the a posteriori probability fX|Y . From Bayes’ rule

The mean of a conditional distribution is called a conditional mean and the variance of a conditional distribution is called a conditional variance.

Continuous Additive Noise with Discrete Input

Additive noise provides a situation in which mixed distributions having both discrete and continuous parts naturally arise. Suppose that the signal X is binary, say with pmf pX(x) = px(1 − p)1−x. The noise term W is assumed to be a continuous random variable described by pdf fW (w), independent of X, with variance σW2 . The observation is defined by Y = X + W . In this case the joint distribution is not defined by a joint pmf or a joint pdf, but by a combination of the two. Some thought may lead to the reasonable guess that the continuous observation given the discrete signal should be describable by a conditional pdf fY |X(y|x) = fW (y − x), where now the conditional pdf is of the elementary variety, the given event

distribution in the opposite direction, which since X is discrete should be a conditional pmf:

Observe that unlike previously treated conditional pmf’s, this one is not an elementary conditional probability since the conditioning event does not have nonzero probability. Thus it cannot be defined in the original manner, but must be justified in the same way as conditional pdf’s, that is, by the fact that we can rewrite the joint distribution (3.92) as

so that pX|Y (x|y) indeed plays the role of a mass of conditional probability, that is,

F

Applying these results to the specific case of the binary input and Gaussian noise, the conditional pmf of the binary input given the noisy observation is

This formula now permits the analysis of a classical problem in communications, the detection of a binary signal in Gaussian noise.

3.10Binary Detection in Gaussian Noise

The derivation of the MAP detector or classifier extends immediately to the the situation of a binary input random variable and independent Gaussian

ˆ

noise just treated. As in the purely discrete case, the MAP detector X(y) of X given Y = y is given by