An Introduction to Statistical Signal Processing

In addition to the three basic set operations, there are two others that will come in handy. Both can be defined in terms of the three basic operations. Refer to Figure A.2 as a visual aid in understanding the definitions.

Figure A.2: Set Di erence Operations

Given two sets F and G, the set di erence of F and G is denoted by F − G, which is defined as

F − G = {ω : ω F and ω G} = F ∩ Gc;

that is, the di erence of F and G contains all of the points in F that are not also in G. Note that this operation is not completely analogous to the “minus” of ordinary arithmetic because there is no such thing as a “negative set.”

Given two sets F and G, their symmetric di erence is denoted by F ∆G, which is defined as

F ∆G = {ω : ω F or ω G but not both}

=(F − G) (G − F ) = (F ∩ Gc) (F c ∩ G)

=(F G) − (F ∩ G);

that is, the symmetric di erence between two sets is the set of points that are in one of the two sets but are not common to both sets. If both sets are the same, the symmetric di erence consists of no points, that is, it is the empty set. If F G, then obviously F ∆G = G − F.

Observe that two sets F and G will be equal if and only if F G and G F. This observation is often useful as a means of proving that two

sets are identical: first prove that each point in one set is in the other and hence the first set is subset of the second. Then prove the opposite inclusion. Surprisingly, this technique is frequently much easier than a direct proof that two sets are identical by a pointwise argument of commonality.

We will often wish to combine sets in a series of operations and to reduce the expression for the resulting set to its simplest and most compact form. Although the most compact form frequently can be seen quickly with the aid of a Venn diagram, as in Figures A.1 and A.2, to be completely rigorous, the use of set theory or set algebra to manipulate the basic operations is required. Table A.1 collects the most important such identities. The first seven relations can be taken as axioms in an algebra of sets and used to derive all other relations, including the remaining relations in the table. Some examples of such derivations follow the table. Readers who are familiar with Boolean algebra will find a one-to-one analogy between the algebra of sets and Boolean algebra.

DeMorgan’s “laws” (A.6) and (A.10) are useful when complementing unions of intersections. Relation (A.16) is useful for writing the union of overlapping sets as a union of disjoint sets. A set and its complement are always disjoint by relation (A.5).

A.2 Examples of Proofs

Relation (A.8). From the definition of intersection and Figure A.1 we verify the truth of (A.8). Algebraically, we show the same thing from the basic seven axioms: From (A.4) and (A.6) we have that

A ∩ B = ((A ∩ B)c)c = (Ac Bc)c ,

and using (A.1), (A.4), and (A.6), this becomes

(Bc Ac)c = (Bc)c ∩ (Ac)c

as desired.

Relation (A.18). Set F = Ω in (A.5) to obtain Ω ∩ Ωc = , which with (A.7) and (A.8) yields (A.19).

Relation (A.11). Complement (A.5), (F c ∩ F )c = c, and hence, using (A.6), (F c F ) = c, and finally, using (A.4) and (A.18), F c F = Ω.

Relation (A.12). Using F c in (A.7): F c ∩ Ω = F c. Complementing the result: (F c ∩ Ω)c = (F c)c = F (by (A.4)). Using (A.6): (F c ∩ Ω)c =

Table A.1: Set Algebra

F Ωc = F. From (A.18) Ωc = , yielding (A.12).

and (A.7) to the left-hand side of this relation yields F ∩ Ω = F = F ∩ F.

Relation (A.19). Complement (A.20) using (A.6) and replace F c by F .

The proofs for the examples were algebraic in nature, manipulating the operations based on the axioms. Proofs can also be constructed based on the definitions of the basic operations. For example, DeMorgan’s law can be proved directly by considering individual points. To prove that (F ∩ G)c = F c Gc it su ces to show separately that (F ∩ G)c F c Gc and F c Gc (F ∩ G)c. Suppose that ω (F ∩ G)c, then ω F ∩ G from the definition of complement and hence ω F or ω G from the definition of intersection (if ω were in both, it would be in the intersection). Thus either ω F c or ω Gc and hence ω F c Gc. Conversely, if ω F c Gc, then ω F c or ω Gc, and hence either ω = F or ω = G, which implies that ω = F ∩ G, which in turn implies that ω (F ∩ G)c, completing the proof.

We will have occasion to deal with more general unions and intersections, that is, unions or intersections of more than two or three sets. As long as the number of unions and intersections is finite, the generalizations are obvious. The various set theoretic relations extend to unions and intersections of finite collections of sets. For example, DeMorgan’s law for finite collections of sets is

For example, consider the finite set version of DeMorgan’s law

This result can be proved using the axioms or by induction. Point arguments are often more direct. Define the set on the left hand side of the equation as G and that on the right hand side as H and to prove G = H by considering individual points. This is done by separately showing that

400 APPENDIX A. PRELIMINARIES

n

and hence ω Fi for some i. This implies that ω i=1 Fi and hence that ω G, completing the proof.

The operations can also be defined for quite general infinite collections of sets as well. Say that we have an indexed collection of sets {Ai; i I}, sometimes denoted {Ai}i I, for some index set I. In other words, this collection is a set whose elements are sets — one set Ai for each possible value of an index i drawn from I. We call such a collection a family or class of sets. (To avoid confusion we never say a “set of sets.”) The index set I can be thought of as numbering the sets. Typical index sets are the set Z+ of all nonnegative integers, Z = {. . . , −1, 0, 1, . . . }, or the real line . The index set may be finite in that it has only a finite number of entries, say I = Zk = {0, 1, . . . , k−1}. The index set is said to be countably infinite if its elements can be counted, that is, can be put into a one-to-one correspondence with a subset of the nonnegative integers Z+; e.g., Z+ or Z itself. If an index set has an infinity of elements, but the elements cannot be counted, then it is said to be uncountably infinite, for example or the unit interval [0, 1] (see problem 11).

The family of sets is said to be finite, countable, or uncountable if the respective index set is finite, countable, or uncountable. As an example, the family of sets {[0, 1/r); r I} is countable if I = Z and uncountable if I = . Another way of describing countably infinite sets is that they can be put into one-to-one correspondence with the integers. For example, the set of rational numbers is countable because it can be enumerated, the set of irrational numbers is not.

The obvious extensions of the pairwise definitions of union and intersection will now be given. Given an indexed family of sets {Ai; i I}, define the union by

Ai = {ω : ω Ai for at least one i I}

i I

and define the intersection by

Ai = {ω : ω Ai for all i I} .

i I

In certain special cases we shall make the notation more specific for particular index sets. For example, if I = {0, . . . , n −1}, then we write the union and intersection as

respectively.

A collection of sets {Fi; i I} is said to be disjoint or pairwise disjoint or mutually exclusive if

Fi ∩ Fj = ; all i, j I, i = j ,

that is, if no sets in the collection contain points contained by other sets in the collection.

The class of sets is said to be collectively exhaustive or to exhaust the

that is, together the Fi contain all the points of the space.

A collection of sets {Fj; i I} is called a partition of the space Ω if the collection is both disjoint and collectively exhaustive. A collection of sets {Fi; i I} is said to partition a set G if the collection is disjoint and the union of all of its members is identical to G.

A.3 Mappings and Functions

We shall make much use of mappings of functions from one space to another. This is of importance in a number of applications. For example, the waveforms and sequences that we considered as members of an abstract space describing the outputs of a random process are just functions of time, e.g., for each value of time t in some continuous discrete collection of possible times we assigned some output value to the function. As a more complicated example, consider a binary digit that is transmitted to a receiver at some destination by sending either plus or minus V volts through a noisy environment called a “channel.” At the receiver a decision is made whether +V or −V was sent. The receiver puts out a 1 or a 0, depending on the decision. In this example three mappings are involved: The transmitter maps a binary symbol in {0, 1} into either +V or −V. During transmission, the channel has an input either +V or −V and produces a real number, not usually equal to 0, 1, +V, or −V. At the receiver, a real number is viewed and a binary number produced.

We will encounter a variety of functions or mappings, from simple arithmetic operations to general filtering operations. We now introduce some common terminology and notation for handling such functions. Given two abstract spaces Ω and A, an A-valued function or mapping F or, in more detail, f : ω → A, is an assignment of a unique point in A to each point in Ω; that is, given any point ω Ω, f(ω) is some value in A. Ω is called the domain or domain of definition of the function f, and A is called the range

of f. Given any sets F Ω and G A, define the image of F (under f) as the set

f(F ) = {a : a = f(ω) for some ω F }

and the inverse image (also called the preimage) of G (under f) as the set

f−1(G) = {ω : f(ω) G} .

Thus f(F ) is the set of all points in A obtained by mapping points in F, and f−1(G) is the set of all points in Ω that map into G.

For example, let Ω = [−1, 1] and A = [−10, 10]. Given the function

f(ω) = ω2 with domain Ω and range A, define the sets F = (−1/2, 1/2) Ω and G = (−1/4, 1) A. Then f(F ) = [0, 1/4) and f−1(G) = [−1, 1]. As

you can see from this example, not all points in G have to correspond to points in F . In fact, the inverse image can be empty; e.g., continuing the same example, f−1((−1/4, 0)) = .

The image of the entire space Ω is called the range space of f, and it need not equal the range; e.g., the function f could map the whole input space into a single point in A. For example, f : → defined by f(r) = 1, all r, has a range space of a single point. If the range space equals the range, the mapping is said to be onto. (Is the mapping f of the preceding example onto? What is the range space? Is the range unique?)

A mapping f is called one-to-one if x = y implies that f(x) = f(y).

A.4 Linear Algebra

We collect a few definitions and results for vectors, matrices, and determinants.

There is a variety of notational variation for vectors. Historically a common form was to use boldface, as in x = (x0, x0, . . . , xk−1), denote a k-dimensional vector with k components xi, i = 0, 1, . . . , k − 1. When dealing with linear algebra, however, it is most commonly the convention to assume that vectors are column vectors, e.g.,

or, as an in-line equation, x = (x0, x0, . . . , xk−1)t, where t denots “transpose.” We will often be lazy and write vectors inline without explicitly denoting the transpose unless it is needed, e.g., in vector/matrix equations.

Although boldface makes it clear which symbols are vectors and which are scalars, in modern practice it is more common to drop the distinction and not use boldface, i.e., to write a vector as simply x = (x0, x1, . . . , xk−1) or, if it is desired to make clear it is a column vector, as x = (x0, x1, . . . , xk−1)t. Both boldface and non-boldface notations are used in this book. Generally, early on the boldface notation is used to clarify when vectors or scalars are being used while later in the book boldface is often dropped.

The inner product (or dot product) of two real-valued n-dimensional vectors y and n id defined by the scalar value

If the vectors are more generally complex valued, then the transpose is replaced by a conjugate transpose

with m rows and n columns. Boldface notation is also used for matrices. If m = n the matrix is said to be square. A matrix is symmetric if At = A,

A is said to be Hermitian.

The product of an m × n matrix and an n-dimensional vector y = Ax is an m dimensional vector with components

n−1

yi = ai,kxi, k=0

that is, the inner product of x and the ith row of A.

The outer product of two n-dimensional vectors y and n id defined as thte n by n matrix

Given a square matrix A, a scalar λ is called an eigenvalue and a vector u is called an eigenvector if

A n by n matrix has n eigenvalues and eigenvectors, but they need not be distinct. Eigenvalues provide interesting formulas for two attributes of matrices, the trace defined by

n−1

Tr(A) = ai,i

i=0

and the determinant of the matrix det(A):

The arithmetic mean/geometric mean inequality says that the arithmetic mean is bound below by the geometric mean:

with equality if and only if the λ are all the same. Application of the inequality to the eigenvalue representation of the determinant and trace provides the inequality