An Introduction to Statistical Signal Processing

1 and the variance of Yi is ri for a fixed parameter |r| < 1. The observed data are gathered at a central processing unit to form an estimate of the unknown random variable X as

(a) Find the mean, variance, and probability density function of the

ˆ

estimate XN .

(b) Find the probability density function f N (α) of the error

− ˆ

*N = X XN .

ˆ

(c) Does XN converge in probability to the true value X?

26. Suppose that {Nt; t ≥ 0} is a process with independent and stationary increments and that

(a)What is the characteristic function for Nt?

(b)What is the characteristic function for the increment Nt − Ns for t > s?

(c)Suppose that Y is a discrete random variable, independent of Nt, with probability mass function

pY (k) = (1 − p)pk, k = 0, 1, · · · .

Find the probability P (Y = Nt).

(d)Suppose that we form the discrete time process {Xn n = 1, 2, · · · } by

Xn = N2n − N2(n−1).

What is the covariance of Xn?

(e) Find the conditional probability mass function

pXn|N2(n−1) (k|m).

(f) Find the expectation

1

E(Nt + 1).

27.Does the weak law of large numbers hold for a random process consisting of Nature selecting a bias uniformly on [0, 1] and then a coin with that bias is flipped forever? In any case, is it true that Sn converges? If so, to what?

28.Suppose that {X(t)} is a continuous time weakly stationary Gaus-

sian random process with zero mean and autocorrelation function RX(τ) = e−2α|τ|, where α > 0. The signal is passed through an RC filter with transfer function

β

H(f) = β + j2πf ,

where β = 1/RC, to form an output process {Y )t)}.

(a)Find the power spectral densities SX(f) and SY (f)?

(b)Evaluate the average powers E[X2(t)] and E[Y 2(t)].

(c)What is the marginal pdf fY (t)?

(d)Now form a discrete time random process {Wn} by Wn = X(nT ), for all integer n. This is called sampling with a sampling period of T . Find the mean, autocorrelation function, and, if it exists, the power spectral density of {Wn}.

(e)Is {Y (t)} a Gaussian random process? Is {Wn} a Gaussian random process? Are they stationary in the strict sense?

(f)Let {Nt} be a Poisson counting process. Let i(t) be the deter-

ministic waveform defined by

i(t) =

1 if t [0, δ]

0 otherwise

— that is, a flat pulse of duration δ. For k = 1, 2, . . . , let tk denote the time of the kth jump in the counting process (that is, tk is the smallest value of t for which Nt = k). Define the random process {Y (t)} by

Nt

Y (t) = i(t − tk) .

k=1

This process is a special case of a class of processes known as filtered Poisson processes. This particular example is a model for shot noise in vacuum tubes. Draw some sample waveforms of this process. Find MY (t)(ju) and pY (t)(n).

Hint: You need not consider any properties of the random variables {tk} to solve this problem.

29.In the physically motivated development of the Poisson counting process, we fixed time values and looked at the random variables giving the counts and the increments of counts at the fixed times. In this problem we explore the reverse description: What if we fix the counts and look at the times at which the process achieves these counts? For

example, for each strictly positive integer k, let rk denote the time that the kth count occurs; that is, rk = α if and only if

Nα = k ; N < k ; all t < α .

Define r0 = 0. For each strictly positive integer k, define the interarrival times τk by

τk = rk − rk−1 ,

and hence

k

rk = τi .

i=1

(a)Find the pdf for rk for k = 1, 2, . . . .

Hint: First find the cdf by showing that

Frk (α) = Pr(kth count occurs before or at time α)

= Pr(Nα ≥ k) ,

and then using the Poisson pmf to write an expression for this sum, di erentiate to find the pdf. You may have to do some algebra to reduce the answer to a simple form not involving any sums. This is most easily done by writing a di erence of two sums in which all terms but one cancel. The final answer is called the Erlang family of pdf’s. You should find that the pdf or r1 is an exponential density.

(b)Use the basic properties of the Poisson counting process to prove that the, interarrival times are iid

Hint: Prove that

Fτn|τ1,... ,τn−1 (α|β1, . . . , βn−1) =

Fτn (α) = 1 − e−λα ; n = 1, 2, . . . ; α ≥ 0 .

Appendix A

Preliminaries: Set Theory,

Mappings, Linear

Algebra, and Linear

Systems

The theory of random processes is constructed on a large number of abstractions. These abstractions are necessary to achieve generality with precision while keeping the notation used manageably brief. Students will probably find learning facilitated if, with each abstraction, they keep in mind (or on paper) a concrete picture or example of a special case of the abstraction. From this the general situation should rapidly become clear. Concrete examples and exercises are introduced throughout the book to help with this process.

A.1 Set Theory

In this section the basic set theoretic ideas that are used throughout the book are introduced. The starting point is an abstract space, or simply a space, consisting of elements or points, the smallest quantities with which we shall deal. This space, often denoted by Ω, is sometimes referred to as the universal set. To describe a space we may use braces notation with either a list or a description contained within the braces { }. Examples are:

[A.0] The abstract space consisting of no points at all, that is, an empty

389

(or trivial) space. This possibility is usually excluded by assuming explicitly or implicitly that the abstract space is nonempty, that is, to contain at least one point.

[A.1] The abstract space with only the two elements zero and one to denote the possible receptions of a radio receiver of binary data at one particular signaling time instant. Equivalently, we could give di erent names to the elements and have a space {0, 1}, the binary numbers, or a space with the elements heads and tails. Clearly the structure of all of these spaces is the same; only the names have been changed. They are di erent, however, in that one is numeric, and hence we can perform arithmetic operations on the outcomes, while the other is not. Spaces which do not have numeric points (or points labeled by numeric vectors, sequences, or waveforms) are sometimes referred to as categorical. Notationally we describe these spaces as {zero, one}, {0, 1}, and {heads, tails}, respectively.

[A.2] Given a fixed positive integer k, the abstract space consisting of all possible binary k−tuples, that is, all 2k k−dimensional binary vectors. This space could model the possible sequences of k flips of the same coin or a single flip of k coins. Note the example [A.1] is a special case of example [A.2].

[A.3] The abstract space with elements consisting of all infinite sequences of ones and zeros or 1 s and 0 s denoting the sequence of possible receptions of a radio receiver of binary data over all signaling times. The sequences could be one-sided in the sense of beginning at time zero and continuing forever, or they could be two-sided in the sense of beginning in the infinitely remote past (time −∞) and continuing into the infinitely remote future.

[A.4] The abstract space consisting of all ASCII (American Standard Code for Information Interchange) codes for characters (letters, numerals, and control characters such as line feed, rub out, etc.). These might be in decimal, hexadecimal, or binary form. In general, we can consider this space as just a space {ai, i = 1, . . . , N} containing a finite number of elements (which here might well be called symbols, letters, or characters).

[A.5] Given a fixed positive integer k, the space of all k−dimensional vectors with coordinates in the space of example [A.4]. This could model all possible contents of an ASCII bu er used to drive a serial printer.

[A.6] The abstract space of all infinite (single-sided or double-sided) sequences of ASCII character codes.

[A.7] The abstract space with elements consisting of all possible voltages measured at the output of a radio receiver at one instant of time. Since all physical equipment has limits to the values of voltage (called “dynamic range”) that it can support, one model for this space is a subset of the real line such as the closed interval [−V, V ] = {r : −V ≤ r ≤ V }, i.e., the set of all real numbers r such that −V ≤ r ≤ +V. If, however, the dynamic range is not precisely known or if we wish to use a single space as a model for several measurements with di erent dynamic ranges, then we might wish to use the entire real line = (−∞, ∞) = {r : −∞ < r < ∞}. The fact that the space includes “impossible” as well as “possible” values is acceptable in a model.

[A.8] Given a positive integer k, the abstract space of all k−dimensional vectors with coordinates in the space of example [A.7]. If the real line is chosen as the coordinate space, then this is k−dimensional Euclidean space.

[A.9] The abstract space with elements being all infinite sequences of members of the space of example [A.7], e.g., all single-sided real-valued sequences of the form {xn, n = 0, 1, 2, . . . }, where xn for all n = 1, 2, . . .

[A.10] Instead of constructing a new space as sequences of elements from another space, we might wish to consider a new space consisting of all waveforms whose instantaneous values are elements in another space, e.g., the space of all waveforms {x(t); t (−∞, ∞)}, where x(t) , all t. This would model, for instance, the space of all possible voltage-time waveforms at the output of a radio receiver. Examples of members of this space are x(t) = cos ωt, z(t) = est, x(t) = 1, x(t) = t, and so on. As with sequences, the waveforms may begin in the remote past or they might be defined for t running from 0 to ∞.

The preceding examples focus on three related themes that will be considered throughout the book: Examples [A.1], [A.4], and [A.7] present models for the possible values of a single measurement. The mathematical model for such a measurement with an unknown outcome is called a random variable. Such simple spaces describe the possible values that a random variable can assume. Examples [A.2], [A.5], and [A.8] treat vectors (finite collections or finite sequences) of individual measurements. The mathematical model for such a vector-valued measurement is called a random vector. Since a vector is made up of a finite collection of scalars, we can also view this random object as a collection (or family) of random variables. These two viewpoints — a single random vector-valued measurement and a collection

of random scalar-valued measurements — will both prove useful. Examples [A.3], [A.6], and [A.9] consider infinite sequences of values drawn from a common alphabet and hence the possible values of an infinite sequence of individual measurements. The mathematical model for this is called a random process (or a random sequence or a random time series). Example [A.10] considers a waveform taking values in a given coordinate space. The mathematical model for this is also called a random process. When it is desired to distinguish between random sequences and random waveforms, the first is called a discrete time random process and the second is called a continuous time random process.

In chapter 3 we shall define precisely what is meant by a random variable, a random vector, and a random process. For now, random variables, random vectors, and random processes can be viewed simply as abstract spaces such as in the preceding examples for scalars, vectors, and sequences or waveforms together with a probabilistic description of the possible outcomes, that is, a means of quantifying how likely certain outcomes are. It is a crucial observation at this point that the three notions are intimately connected: random vectors and processes can be viewed as collections or families of random variables. Conversely, we can obtain the scalar random variables by observing the coordinates of a random vector or random process. That is, if we “sample” a random process once, we get a random variable. Thus we shall often be interested in several di erent, but related, abstract spaces. For example, the individual scalar outputs may be drawn from one space, say A, which could be any of the spaces in examples [A.1], [A.4], or [A.7]. We then may also wish to look at all possible k−dimensional vectors with coordinates in A, a space that is often denoted by Ak, or at spaces of infinite sequences of waveforms of A. These latter spaces are called product spaces and will play an important role in modeling random phenomena.

Usually one will have the option of choosing any of a number of spaces as a model for the outputs of a given random variable. For example, in flipping a coin one could use the binary space {head, tail}, the binary space {0, 1} (obtained by assigning 0 to head and 1 to tail), or the entire real line. Obviously the last space is much larger than needed, but it still captures all of the possible outcomes (along with many “impossible” ones). Which view and which abstract space is the “best” will depend on the problem at hand, and the choice will usually be made for reasons of convenience.

Given an abstract space, we shall consider groupings or collections of the elements that may be (but are not necessarily) smaller than the whole space and larger than single points. Such groupings are called sets. If every point in one set is also a point in a second set, then the first set is said to be a subset of the second. Examples (corresponding respectively to the

previous abstract space examples) are:

[A.11] The empty set consisting of no points at all. Thus we could rewrite example [A.0] as Ω = . By convention, the empty set is considered to be a subset of all other sets.

[A.12] The set consisting of the single element one. This is an example of a one-point set or singleton set.

[A.13] The set of all k−dimensional binary vectors with exactly one zero coordinate.

[A.14] The set of all infinite sequences of ones and zeros with exactly 50% of the symbols being one (as defined by an appropriate mathematical limit).

[A.15] The set of all ASCII characters for capital letters.

[A.16] The set of all four-letter English words.

[A.17] The set of all infinite sequences of ASCII characters excluding those representing control characters.

[A.18] Intervals such as the set of all voltages lying between 1 volt and 20 volts are useful subsets of the real line. These come in several forms, depending on whether or not the end points are included. Given b > a, define the “open” interval (a, b) = {r : a < r < b}, and given b ≥ a, define the “closed” interval [a, b] = {r : a ≤ r ≤ b}. That is, we use a bracket if the end point is included and a parenthesis if it is not. We will also consider “half-open” or “half-closed” intervals of the form (a, b] = {r : a < r ≤ b} and [a, b) = {r : a ≤ r < b}. (We use quotation marks around terms like open and closed because we are not rigorously defining them, we are implicitly defining them by their most important examples, intervals of the real line).

[A.19] The set of all vectors of k voltages such that the largest value is less than 1 volt.

[A.20] The set of all sequences of voltages which are all nonnegative.

[A.21] The set of all voltage-time waveforms that lie between 1 and 20 volts for all time.

Given a set F of points in an abstract space Ω, we shall write ω F for “the point ω is contained in the set F ” and ω F for “the point ω is not contained in the set F.” The symbol is referred to as the element

394 APPENDIX A. PRELIMINARIES

inclusion symbol. We shall often describe a set using this notation in the form F = {ω : ω has some property. Thus F = {ω : ω F }. For example, a set in the abstract space Ω = {ω : −∞ < ω < ∞} (the real line ) is {ω : −2 ≤ ω < 4.6}. The abstract space itself is a grouping of elements and hence is also called a set. Thus Ω = {ω : ω Ω}.

If a set F is a subset of another set G; that is, if ω F implies that also ω G, then we write F G. The symbol is called the set inclusion symbol. Since a set is included within itself, every set is a subset of itself.

An individual element or point ω0 in F can be considered both as a point or element in the space and as a one-point set or singleton set {ω0} = {ω : ω = ω0}. Note, however, that the braces notation is more precise when we are considering the one-point set and that ω0 Ω while {ω0} Ω.

The three basic operations on sets are complementation, intersection, and union. The definitions are given next. Refer also to Figure A.1 as an aid in visualizing the definitions. In Figure A.1 Ω is pictured as the outside box and the sets F and G are pictured as arbitrary blobs within the box. Such diagrams are called Venn diagrams.

Given a set F, the complement of F is denoted by F c, which is defined

by

F c = {ω : ω F } ,

that is, the complement of F contains all of the points of Ω that are not in

F.

Given two sets F and G, the intersection of F and G is denoted by F ∩ G, which is defined by

F ∩ G = {ω : ω F and ω G} ,

that is, the intersection of two sets F and G contains the points which are in both sets.

If F and G have no points in common, then F ∩ G = , the null set, and F and G are said to be disjoint or mutually exclusive.

Given two sets F and G, the union of F and G is denoted by F G, which is defined by

F G = {ω : ω F or ω G} ,

that is, the union of two sets F and G contains the points that are either in one set of the other, or both.

Observe that the intersection of two sets is always a subset of each of them, e.g., F ∩ G F. The union of two sets, however, is not a subset of either of them (unless one set is a subset of the other). Both of the original sets are subsets of their union, e.g., F F G.