An Introduction to Statistical Signal Processing
.pdfAn Introduction to Statistical Signal Processing
f−1(F )
f
F
Pr(f F ) = P ({ω : ω F }) = P (f−1(F ))
May 5, 2000
ii
An Introduction to Statistical Signal Processing
Robert M. Gray
and
Lee D. Davisson
Information Systems Laboratory
Department of Electrical Engineering
Stanford University
and
Department of Electrical Engineering and Computer Science
University of Maryland
iv
c 1999 by the authors.
v
to our Families
vi
Contents
Preface |
|
|
xi |
|
Glossary |
|
xv |
||
1 |
Introduction |
1 |
||
2 |
Probability |
11 |
||
|
2.1 |
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . |
11 |
|
|
2.2 |
Spinning Pointers and Flipping Coins . . . . . . . . . . . . |
15 |
|
|
2.3 |
Probability Spaces . . . . . . . . . . . . . . . . . . . . . . . |
23 |
|
|
|
2.3.1 |
Sample Spaces . . . . . . . . . . . . . . . . . . . . . |
28 |
|
|
2.3.2 |
Event Spaces . . . . . . . . . . . . . . . . . . . . . . |
31 |
|
|
2.3.3 |
Probability Measures . . . . . . . . . . . . . . . . . . |
42 |
|
2.4 |
Discrete Probability Spaces . . . . . . . . . . . . . . . . . . |
45 |
|
|
2.5 |
Continuous Probability Spaces . . . . . . . . . . . . . . . . |
56 |
|
|
2.6 |
Independence . . . . . . . . . . . . . . . . . . . . . . . . . . |
70 |
|
|
2.7 |
Elementary Conditional Probability . . . . . . . . . . . . . |
71 |
|
|
2.8 |
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
75 |
|
3 |
Random Objects |
85 |
||
|
3.1 |
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . |
85 |
|
|
|
3.1.1 |
Random Variables . . . . . . . . . . . . . . . . . . . |
85 |
|
|
3.1.2 |
Random Vectors . . . . . . . . . . . . . . . . . . . . |
89 |
|
|
3.1.3 |
Random Processes . . . . . . . . . . . . . . . . . . . |
93 |
|
3.2 |
Random Variables . . . . . . . . . . . . . . . . . . . . . . . |
95 |
|
|
3.3 |
Distributions of Random Variables . . . . . . . . . . . . . . |
104 |
|
|
|
3.3.1 |
Distributions . . . . . . . . . . . . . . . . . . . . . . |
104 |
|
|
3.3.2 |
Mixture Distributions . . . . . . . . . . . . . . . . . |
108 |
|
|
3.3.3 |
Derived Distributions . . . . . . . . . . . . . . . . . |
111 |
|
3.4 |
Random Vectors and Random Processes . . . . . . . . . . . |
115 |
|
|
3.5 |
Distributions of Random Vectors . . . . . . . . . . . . . . . |
117 |
vii
viii |
CONTENTS |
3.5.1 Multidimensional Events . . . . . . . . . . . . . . . 118
3.5.2 Multidimensional Probability Functions . . . . . . . 119
3.5.3Consistency of Joint and Marginal Distributions . . 120
3.6 |
Independent Random Variables . . . . . . . . . . . . . . . . |
127 |
|
|
3.6.1 |
IID Random Vectors . . . . . . . . . . . . . . . . . . |
128 |
3.7 |
Conditional Distributions . . . . . . . . . . . . . . . . . . . |
129 |
|
|
3.7.1 |
Discrete Conditional Distributions . . . . . . . . . . |
130 |
|
3.7.2 |
Continuous Conditional Distributions . . . . . . . . |
131 |
3.8 |
Statistical Detection and Classification . . . . . . . . . . . . |
134 |
|
3.9 |
Additive Noise . . . . . . . . . . . . . . . . . . . . . . . . . |
137 |
|
3.10 |
Binary Detection in Gaussian Noise . . . . . . . . . . . . . |
144 |
|
3.11 |
Statistical Estimation . . . . . . . . . . . . . . . . . . . . . |
146 |
|
3.12 |
Characteristic Functions . . . . . . . . . . . . . . . . . . . . |
147 |
|
3.13 |
Gaussian Random Vectors . . . . . . . . . . . . . . . . . . . |
152 |
|
3.14 |
Examples: Simple Random Processes . . . . . . . . . . . . . |
154 |
|
3.15 |
Directly Given Random Processes . . . . . . . . . . . . . . |
157 |
|
|
3.15.1 |
The Kolmogorov Extension Theorem . . . . . . . . . |
157 |
|
3.15.2 |
IID Random Processes . . . . . . . . . . . . . . . . . |
158 |
|
3.15.3 |
Gaussian Random Processes . . . . . . . . . . . . . . |
158 |
3.16 |
Discrete Time Markov Processes . . . . . . . . . . . . . . . |
159 |
|
|
3.16.1 |
A Binary Markov Process . . . . . . . . . . . . . . . |
159 |
|
3.16.2 |
The Binomial Counting Process . . . . . . . . . . . . |
162 |
|
3.16.3 |
Discrete Random Walk . . . . . . . . . . . . . . . . |
165 |
|
3.16.4 |
The Discrete Time Wiener Process . . . . . . . . . . |
166 |
|
3.16.5 |
Hidden Markov Models . . . . . . . . . . . . . . . . |
167 |
3.17 |
Nonelementary Conditional Probability . . . . . . . . . . . |
168 |
|
3.18 |
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
170 |
|
4 Expectation and Averages |
187 |
||
4.1 |
Averages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
187 |
|
4.2 |
Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . |
190 |
|
|
4.2.1 |
Examples: Expectation . . . . . . . . . . . . . . . . |
192 |
4.3 |
Functions of Several Random Variables . . . . . . . . . . . . |
200 |
|
4.4 |
Properties of Expectation . . . . . . . . . . . . . . . . . . . |
200 |
4.5Examples: Functions of Several Random Variables . . . . . 203
|
4.5.1 |
Correlation . . . . . . . . . . . . . . . . . . . . . . . |
203 |
|
4.5.2 |
Covariance . . . . . . . . . . . . . . . . . . . . . . . |
205 |
|
4.5.3 |
Covariance Matrices . . . . . . . . . . . . . . . . . . |
206 |
|
4.5.4 |
Multivariable Characteristic Functions . . . . . . . . |
207 |
|
4.5.5 |
Example: Di erential Entropy of a Gaussian Vector |
209 |
4.6 |
Conditional Expectation . . . . . . . . . . . . . . . . . . . . |
210 |
|
4.7 |
Jointly Gaussian Vectors . . . . . . . . . . . . . . . . . . . |
213 |
CONTENTS |
ix |
|
4.8 |
Expectation as Estimation . . . . . . . . . . . . . . . . . . . |
216 |
4.9 |
Implications for Linear Estimation . . . . . . . . . . . . . |
222 |
4.10 |
Correlation and Linear Estimation . . . . . . . . . . . . . . |
224 |
4.11 |
Correlation and Covariance Functions . . . . . . . . . . . . |
231 |
4.12 |
The Central Limit Theorem . . . . . . . . . . . . . . . . . |
235 |
4.13 |
Sample Averages . . . . . . . . . . . . . . . . . . . . . . . . |
237 |
4.14 |
Convergence of Random Variables . . . . . . . . . . . . . . |
239 |
4.15 |
Weak Law of Large Numbers . . . . . . . . . . . . . . . . . |
244 |
4.16 |
Strong Law of Large Numbers . . . . . . . . . . . . . . . . |
246 |
4.17 |
Stationarity . . . . . . . . . . . . . . . . . . . . . . . . . . . |
251 |
4.18 |
Asymptotically Uncorrelated Processes . . . . . . . . . . . . |
256 |
4.19 |
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
259 |
5 Second-Order Moments |
281 |
|
5.1 |
Linear Filtering of Random Processes . . . . . . . . . . . . |
282 |
5.2 |
Second-Order Linear Systems I/O Relations . . . . . . . . . |
284 |
5.3 |
Power Spectral Densities . . . . . . . . . . . . . . . . . . . . |
289 |
5.4 |
Linearly Filtered Uncorrelated Processes . . . . . . . . . . . |
292 |
5.5 |
Linear Modulation . . . . . . . . . . . . . . . . . . . . . . . |
298 |
5.6 |
White Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . |
301 |
5.7 |
Time-Averages . . . . . . . . . . . . . . . . . . . . . . . . . |
305 |
5.8 |
Di erentiating Random Processes . . . . . . . . . . . . . . |
309 |
5.9 |
Linear Estimation and Filtering . . . . . . . . . . . . . . . |
312 |
5.10 |
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
326 |
6 A Menagerie of Processes |
343 |
|
6.1 |
Discrete Time Linear Models . . . . . . . . . . . . . . . . . |
344 |
6.2 |
Sums of IID Random Variables . . . . . . . . . . . . . . . . |
348 |
6.3 |
Independent Stationary Increments . . . . . . . . . . . . . . |
350 |
6.4 |
Second-Order Moments of ISI Processes . . . . . . . . . . |
353 |
6.5 |
Specification of Continuous Time ISI Processes . . . . . . . |
355 |
6.6 |
Moving-Average and Autoregressive Processes . . . . . . . . |
358 |
6.7 |
The Discrete Time Gauss-Markov Process . . . . . . . . . . |
360 |
6.8 |
Gaussian Random Processes . . . . . . . . . . . . . . . . . . |
361 |
6.9 |
The Poisson Counting Process . . . . . . . . . . . . . . . . |
361 |
6.10 |
Compound Processes . . . . . . . . . . . . . . . . . . . . . . |
364 |
6.11 |
Exponential Modulation . . . . . . . . . . . . . . . . . . . |
366 |
6.12 |
Thermal Noise . . . . . . . . . . . . . . . . . . . . . . . . . |
371 |
6.13 |
Ergodicity and Strong Laws of Large Numbers . . . . . . . |
373 |
6.14 |
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
377 |
x |
|
|
CONTENTS |
A |
Preliminaries |
389 |
|
|
A.1 |
Set Theory . . . . . . . . . . . . . . . . . . . . . . |
. . . . . 389 |
|
A.2 |
Examples of Proofs . . . . . . . . . . . . . . . . . . . |
. . . . 397 |
|
A.3 |
Mappings and Functions . . . . . . . . . . . . . . . . |
. . . . 401 |
|
A.4 |
Linear Algebra . . . . . . . . . . . . . . . . . . . . . |
. . . . 402 |
|
A.5 |
Linear System Fundamentals . . . . . . . . . . . . . |
. . . . 405 |
|
A.6 |
Problems . . . . . . . . . . . . . . . . . . . . . . . . |
. . . . 410 |
B |
Sums and Integrals |
417 |
|
|
B.1 |
Summation . . . . . . . . . . . . . . . . . . . . . . . |
. . . . 417 |
|
B.2 |
Double Sums . . . . . . . . . . . . . . . . . . . . . . |
. . . . 420 |
|
B.3 |
Integration . . . . . . . . . . . . . . . . . . . . . . . |
. . . . 421 |
|
B.4 |
The Lebesgue Integral . . . . . . . . . . . . . . . . |
. . . . 423 |
C |
Common Univariate Distributions |
427 |
|
D |
Supplementary Reading |
429 |
|
Bibliography |
434 |
||
Index |
|
438 |