An Introduction to Statistical Signal Processing

converge in probability to 0?

47.A discrete time martingalemartingale {Yn n = 0.1.2. . . . } is a process with the property that

E[Yn|Y0, Y1, . . . , Yn−1] = Yn.

In words, the conditional expectation of the expectation of the current value is the previous value. Suppose that {Xn} is iid. Is

n−1

Yn = Xn

n=0

amartingale?

48.Let {Yn} be the one-dimensional random walk of chapter 3.

(a)Find the pmf pYn for n = 0, 1, 2.

(b)Find the mean E[Yn] and variance σY2n .

(c)Does Yn/n converge as n gets large?

(d)Find the conditional pmf’s pYn|Y0,Y1,... ,Yn−1 (yn|y0, y1, . . . , yn−1) and pYn|Yn−1 (yn|yn−1). Is this process Markov?

(e)What is the minimum MSE estimate of Yn given Yn−1? What is the probability that Yn which actually equal its minimum MSE estimate?

49.Let {Xn} be a binary iid process with pX(±1) = 0.5. Define a new process {Wn; n = 0, 1, . . . } by

Wn = Xn + Xn−1.

This is an example of a moving average process, so-called because it computes a short term average of the input process. Find the mean, variance, and covariance function of {Wn}. Prove a weak law of large numbers for Wn.

50.How does one generate a random process? It is often of interest to do so in order to simulate a physical system in order to test an algorithm before it is applied to genuine data. Using genuine physical data may be too expensive, dangerous, or politically risky. One might connect a sensor to a resistor and heat it up to produce thermal noise, or flip a coin a few million times. One solution requires uncommon hardware and the other physical e ort. The usual solution is to use a computer to generate a sequence that is not actually random, but pseudo random in that it can produce a long sequence of numbers that appear to be random and which will satisfy several tests for randomness, provided that the tests are not too stringent. An example is the rand command used in MatlabTM. It uses the linear congruential method which starts with a “seed” X0 and then recursively defines the sequence

This produces a sequence of integers in the range from 0 to 231 − 1. Dividing by 231 (which is just a question of shifting in binary arithmetic) produces a number in the range [0, 1). Find a computer with Matlab or program this algorithm yourself and try it out with di erent starting sequences. Find the sample average Sn of a sequence of 100, 1000, and 10000 samples and compare them to the expected value of the uniform pdf random variable considered in this chapter. How might you determine whether or not the sequence being viewed was indeed random or not if you did not know how it was generated?

51.Suppose that U is a random variable with pdf fU (u) = 1 for u [0, 1). Describe a function q : [0, 1) → A, where A = {0, a1, . . . , K − 1, so that the random variable X = q(U) is discrete with pmf

1

pX(k) = K ; k = 0, 1, . . . , K − 1.

You have produced a uniform discrete random variable from a uniform continuous random variable.

(a) What is the minimum mean squared error estimator of U given

ˆ

X = k? Call this estimator U(k). Write an expression for the resulting MSE

final output (assuming that q is fixed). You have just demonstrated one of the key properties of a Lloyd-Max quantizer.

and σ ˆ . How do the mean and variance of the U compare with

U

those of U? (I.e., equal, bigger, smaller?)

52.Modify the development in the text for the minimum mean squared error estimator to work for discrete random variables. What is the minimum MSE estimator for Yn given Yn−1 for the binary Markov process developed in the chapter? Which do you think makes more sense for guessing the next outcome for a binary Markov process, the minimum probability of error classifier or the minimum MSE estimator? Explain.

53.Let {Yn; n = 0, 1, . . . } be the binary Markov process developed in the chapter. Find a new process {Wn; n = 1, 2, . . . } defined by Wn = Yn Yn−1. Describe the process Wn.

54.(Problem courtesy of the ECE Department of the Technion.) Let X be a Gaussian random variable with zero mean and variance σ2.

(a)Find E[cos(nX)], n = 1, 2, . . . .

(b)Find E[Xn], n = 1, 2, . . . .

(c)Let N be a Poisson random variable with parameter λ and assume that X and N are independent. Find E[Xn].

Hint: Use characteristic functions and iterated expecttation.

55.(Problem courtesy of the ECE Department of the Technion.) Let X be a random variable with uniform pdf on [−1, 1]. Define a new

random variable Y by

Y = X X ≤ 0 1 X > 0

(a)Find the cdf FY (y) and plot it.

(b)Find the pdf fY (y).

(c)Find E(Y ) and σY2 .

(d)Find E(X|Y ).

(e)Find E[(X − E(X|Y ))2].

56.(Problem courtesy of the ECE Department of the Technion.) Let X1, X2, . . . , Xn be zero mean statistically independent random variables. Define

n

Yn = Xi.

i=1

Find E(Y7|Y1, Y2, Y3).

57.(Problem courtesy of the ECE Department of the Technion.) Let U

denote a binary random variable with pmf pU (u) = .5 for u = ±1. Let Y = U +X, where X is N(0, σ2) and where U and X are independent. Find E(U|Y ).

58.(Problem courtesy of the ECE Department of the Technion.) Let {Xn; n = 1, 2, . . . } be an iid sequence with mean 0 and unit variance. Let K be a discrete random variable, independent of the Xn, which has a uniform pmf on {1, 2, . . . , 16}. Define

n

Yn = Xi.

i=1

(a)Find E(Y ) and σY2 .

(b)Find the optimal linear estimator in the MSE sense of X1 given Y and calculate the resulting MSE.

(c)Find the optimal linear estimator in the MSE sense of K given Y and calculate the resulting MSE.

59.(Problem courtesy of the ECE Department of the Technion.) Let Y, N1, N2 be zero mean, unit variance, mutually independent random variables. Define

(a)Find the linear MMSE estimator of Y given X1 and X2.

(b)Find the resulting MSE.

(c)For what value of α [0, ∞) does the mean squared error become zero? Provide an intuitive explanation.

60.(Problem courtesy of the ECE Department of the Technion.) Let {Xn; n = 1, 2, . . . } be an iid sequence of N(m, σ2) random variables. Define for any positive integer N

N

SN = Xn.

n=1

(a) For K < N find the pdf fSN .SK (α, β).

(b) Find the MMSE estimator of SK given SN , E(SK|SN ). Define

VK = K X2. Find the MMSE of VK given VN .

n=1 n

61.(Problem courtesy of the ECE Department of the Technion.) Let Xi = S + Wi, i = 1, 2, . . . , N, where S and the Wi are mutually inde-

pendent with zero mean. The variance of S is σS and the variances of all the Wi are all σW2 .

(a)Find the linear MMSE of S given the observations Xi, i = 1, 2, . . . , N.

(b)Find the resulting MSE.

Chapter 5

Second-Order Moments

In chapter 4 we have seen that the second-order moments of a random process — the mean and covariance or, equivalently, the autocorrelation

— play a fundamental role in describing the relation of limiting sample averages and expectations. We have also seen, e.g., in Section 4.5.1 and problem 4.23, and we shall see again that these moments also play a key role in signal processing applications of random processes, linear least squares estimation in particular. Because of the fundamental importance of these particular moments, this chapter considers their properties in greater depth and their evaluation for several important examples. A primary focus is on a second-order moment analog of a derived distribution problem: suppose that we are given the second-order moments of one random process and that this process is then used as an input to a linear system; what are the resulting second-order moments of the output random process? These results are collectively known as second-order moment input/output or I/O relations for linear systems.

Linear systems may seem to be a very special case. As we will see, their most obvious attribute is that they are easier to handle analytically, which leads to more complete, useful, and stronger results than can be obtained for the class of all systems. This special case, however, plays a central role and is by far the most important class of systems. The design of engineering systems frequently involves the determination of an optimum system — perhaps the optimum signal detector for a signal in noise, the filter that provides the highest signal-to-noise ratio, the optimum receiver, etc. Surprisingly enough, the optimum is frequently a linear system. Even when the optimum is not linear, often a linear system is a good enough approximation to the optimal system so that a linear system is used for the sake of economical design. For these reasons it is of interest to study

281

the properties of the output random process from a linear system that is driven by a specified input random process. In this chapter we consider only second-order moments; in the next chapter we consider examples in which one can develop a more complete probabilistic description of the output process. As one might suspect the less complete second-order descriptions are possible under far more general conditions.

With the knowledge of the second-order properties of the output process when a linear system is driven by a given random process, one will have the fundamental tools for the analysis and optimization of such linear systems. As an example of such analysis, the chapter closes with an application of second-order moment theory to the design of systems for linear least squares estimation.

Because the primary engineering application of these systems is to noise discrimination, we will group them together under the name “linear filters.” This designation denotes the suppression or “filtering out” of noise from the combination of signal and noise. The methods of analysis are not limited to this application, of course.

As usual, we emphasize discrete time in the development, with the obvious extensions to continuous time provided by integrals. Furthermore, we restrict attention in the basic development to linear time-invariant filters. The extension to time-varying systems is obvious but cluttered with obfuscating notation. Time-varying systems will be encountered briefly when considering recursive estimation.

5.1Linear Filtering of Random Processes

Suppose that a random process {X(t); t T }, (or {Xt; t T }) is used as an input to a linear time-invariant system described by a δ response h. Hence the output process, say {Y (t)} or {Yt} is described by the convolution integral of (A.22) in the continuous time case of the convolution sum of (A.29) in the discrete time case. To be precise, we have to be careful about how the integral or sum is defined; that is, integrals and infinite sums of random processes are really limits of random variables, and those limits can converge in a variety of ways, such as quadratic mean or with probability one. For the moment we will assume that the convergence is pointwise (that is, with probability one), i.e., that each realization or sample function of the output is related to the corresponding realization of the input via (A.22) or (A.29). That is, we take

Y (t) = X(t − s)h(s) ds (5.1)

s: t−sT

k:n−k T

to mean actual equality for all elementary events ω on the underlying probability space Ω. More precisely,

s: t−s T

Yn(ω) = Xn−k(ω)hk ,

k:n−k T

respectively. Rigorous consideration of conditions under which the various limits exist is straightforward for the discrete time case. It is obvious that the limits exist for the so-called finite impulse response (FIR) discrete time filters where only a finite number of the hk are nonzero and hence the sum has only a finite number of terms. It is also possible to show mean square convergence for the general discrete time convolution if the input process has finite mean and variance and if the filter is stable in the sense of (A.30). In particular, for a two-sided input process, (5.2) converges in quadratic mean; i.e.,

n−1

l.i.m. Xn−khk

N→∞

k=0

exists for all n. Convergence with probability 1 can be established using more advanced methods provided su cient technical conditions are satisfied. The theory is far more complicated in the continuous time case. As usual, we will by and large ignore these problems and just assume that the convolutions are well defined.

Unfortunately, (A.24) and (A.30) are not satisfied in general for sample functions of interesting random processes and hence in general one cannot take Fourier transforms of both sides of (5.1) and (5.2) and obtain a useful spectral relation. Even if one could, the Fourier transform of a random process would be a random variable for each value of frequency! Because of this, the frequency domain theory for random processes is quite di erent from that for deterministic processes. Relations such as (A.26) may on occasion be useful for intuition, but they must be used with extreme care.

With the foregoing notation and preliminary considerations, we now turn to the analysis of discrete time linear filters with random process inputs.

5.2Second-Order Linear Systems I/O Relations

Discrete Time Systems

Ideally one would like to have a complete specification of the output of a linear system as a function of the specification of the input random process. Usually this is a di cult proposition because of the complexity of the computations required. However, it is a relatively easy task to determine the mean and covariance function at the output. As we will show, the output mean and covariance function depend only on the input mean and covariance function and on no other properties of the input random process. Furthermore, in many, if not most, applications, the mean and covariance functions of the output are all that are needed to solve the problem at hand. As an important example: if the random process is Gaussian, then the mean and covariance functions provide a complete description of the process.

Linear filter input/output (I/O) relations are most easily developed using the convolution representation of a linear system. Let {Xn} be a discrete time random process with mean function mn = EXn and covariance function KX(n, k) = E[(Xn − mn)(Xk − mk)]. Let {hk} be the Kronecker δ response of a discrete time linear filter. For notational convenience we assume that the δ response is causal. The non-causal case simply involves a change of the limits of summation. Next we will find the mean and covariance functions for the output process {Yn} that is given in the convolution equation of (5.2).

From (5.2) the mean of the output process is found using the linearity of expectation as

assuming, of course, that the sum exists. The sum does exist if the filter is stable and the input mean is bounded. That is, if there is a constant m < ∞, such that |mn| ≤ |m| for all n and if the filter is stable in the sense of equation (A.30), then

≤ |m| |hk| < ∞ .

k

If the input process {Xn} is weakly stationary, then the input mean function