An Introduction to Statistical Signal Processing

form of the pdf or pmf, only its starting point. This has the discouraging implication that if, for example, the time for the next arrival of a bus is described by an exponential pdf, then knowing you have already waited for an hour does not change your pdf to the next arrival from what it was when you arrived.

2.8Problems

1.Suppose that you have a set function P defined for all subsets F Ω of a sample space Ω and suppose that you know that this set function satisfies (2.7-2.9). Show that for arbitrary (not necessarily disjoint) events,

P (F G) = P (F ) + P (G) − P (F ∩ G) .

2.Describe the sigma-field of subsets of generated by the points or singleton sets. Does this sigma-field contain intervals of the form (a, b) for b > a?

3.Given a finite subset A of the real line , prove that the power set of A and B(A) are the same. Repeat for a countably infinite subset of

.

4.Given that the discrete sample space Ω has n elements, show that the power set of Ω consists of 2n elements.

5.*Let Ω = , the real line, and consider the collection F of subsets ofdefined as all sets of the form

for all possible choices of nonnegative integers k and m and all possible choices of real numbers ai < bi, ci < di. If k or m is 0, then the respective unions are defined to be empty so that the empty set itself has the form given. In other words, F contains all possible finite unions of half-open intervals of this form and complements of such half-open intervals. Every set of this form is in F and every set in F has this form. Prove that F is a field of subsets of Ω. Does F contain the points? For example, is the singleton set {0} in F? Is F

a sigma-field?

6.Let Ω = [0, ∞) be a sample space and let F be the sigma-field of subsets of Ω generated by all sets of the form (n, n+1) for n = 1, 2, . . .

(a)Are the following subsets of Ω in F? (i) [0, ∞), (ii) Z+ = {0, 1, 2, . . . },

(iii)[0, k] [k + 1, ∞) for any positive integer k, (iv) {k} for any positive integer k, (v) [0, k] for any positive integer k, (vi) (1/3, 2).

(b)Define the following set function on subsets of Ω :

(If there is no i for which i + 1/2 F , then the sum is taken as zero.) Is P a probability measure on (Ω, F) for an appropriate choice of c? If so, what is c?

(c)Repeat part (b) with B, the Borel field, replacing F as the event space.

(d)Repeat part (b) with the power set of [0, ∞) replacing F as the event space.

(e)Find P (F ) for the sets F considered in part (a).

7.Show that an equivalent axiom to 2.3 of probability is the following: If F and G are disjoint, then P (F G) = P (F ) + P (G) ,

that is, we really need only specify finite additivity for the special case of n = 2.

8. Consider the measurable space ((0, 1], B([0, 1])). Define a set function P on this space as follows:

Is P a probability measure?

9. Let S be a sphere in 3 : S = {(x, y, z) : x2 + y2 + z2 ≤ r2}, where r is a fixed radius. In the sphere are fixed N molecules of gas, each molecule being considered as an infinitesimal volume (that is, it occupies only a point in space). Define for any subset F of S the function

#(F ) = {the number of molecules in F } .

Show that P (F ) = #(F )/N is a probability measure on the measurable space consisting of S and its power set.

10.Suppose that you are given a probability space (Ω, F, P ) and that a collection FP of subsets of Ω is defined by

FP = {F N; all F F, all N G for which G F and P (G) = 0}.

(2.103)

In words: FP contains every event in F along with every subset N which is a subset of zero probability event G F, whether or not N is itself an event (a member of F). Thus FP is formed by adding any sets not already in FP which happen to be subsets of zero probability events. We can define a set function P for the measurable space (Ω, FP ) by

P (F N) = P (F ) if F F and N G F, where P (G) = 0. (2.104)

Show that (Ω, FP , P ) is a probability space, i.e., you must show that

FP is an event space and that P is a probability measure. A probability space with the property that all subsets of zero probability events are also events is said to be complete and the probability space (Ω, FP , P ) is called the completion of the probability space (Ω, F, P ).

In problems 2.11 to 2.17 let (Ω, F, P ) be a probability space and assume that all given sets are events.

11.If G F , prove that P (F −G) = P (F )−P (G). Use this fact to prove that if G F , then P (G) ≤ P (F ).

12.Let {Fi} be a countable partition of a set G. Prove that for any event

P (H ∩ Fi) = P (H ∩ G) .

i

13.If {Fi, i = 1, 2, . . . } forms a partition of Ω and {Gi; i = 1, 2, . . . } forms a partition of Ω, prove that for any H,

=1 j=1

14.Prove that |P (F ) − P (G)| ≤ P (F ∆G).

15.Prove that P (F G) ≤ P (F ) + P (G). Prove more generally that for any sequence (i.e., countable collection) of events Fi,

This inequality is called the union bound or the Bonferoni inequality. (Hint: Use problem A.2 or 2.1.)

16. Prove that for any events F, G, and H,

P (F ∆G) ≤ P (F ∆H) + P (H∆G) .

In words: If the probability of the symmetric di erence of two events is small, then the two events must have approximately the same probability. The astute observer may recognize this as a form of the triangle inequality; one can consider P (F ∆G) as a distance or metric on events.

17.Prove that if P (F ) ≥ 1 − δ and P (G) ≥ 1 − δ, then also P (F ∩ G) ≥ 1 −2δ. In other words, if two events have probability nearly one, then their intersection has probability nearly one.

18.*The Cantor set Consider the probability space (Ω, B(Ω), P ) where

P is described by a uniform pdf on Ω = [0, 1). Let F1 = (1/3, 2/3), the middle third of the sample space. Form the set G1 = Ω − F1 by removing the middle third of the unit interval. Next define F2

as union of the middle thirds of all of the intervals in G1, i.e., F2 =

(1/9, 2/9) (7/9, 8/9). Define G2 as what remains when remove F2 from G1, that is,

G2 = G1 − F2 = [0, 1] − (F1 F2).

Continue in this manner. At stage n Fn is the union of the middle

n−1

thirds of all of the intervals in Gn−1 = [0, 1] − k=1 Fn. The Cantor set is defined as the limit of the Gn, that is,

(a)Prove that C B(Ω), i.e., that it is an event.

(b)Prove that

(c)Prove that P (C) = 0, i.e., that the Cantor set has zero probability.

One thing that makes this problem interesting is that unlike most simple examples of nonempty events with zero probability, the Cantor set has an uncountable infinity of points and not a discrete set. This can be shown be first showing that a point x C if and only if the

point can be expressed as a ternary number x = ∞ an3−n where

n=1

all the an are either 0 or 2. Thus the number of points in the Cantor set is the same as the number of real numbers that can be expressed in this fashion, which is the same as the number of real numbers that can be expressed in a binary expansion (since each an can have only two values), which is the same as the number of points in the unit interval, which is uncountably infinite.

19.Six people sit at a circular table and pass around and roll a single fair die (equally probable to have any face 1 through 6 showing) beginning with person # 1. The game continues until the first 6 is rolled, the person who rolled it wins the game. What is the probability that player # 2 wins?

20.Show that given (2.22) through (2.24), (2.28) or (2.29) implies (2.25). Thus (2.25), (2.28), and (2.29). provide equivalent candidates for the fourth axiom of probability.

21.Suppose that P is a probability measure on the real line and define the sets Fn = (0, 1/n) for all positive integer n. Evaluate limn→∞ P (Fn).

22.Answer true or false for each of the following statements. Answers must be justified.

(a)The following is a valid probability measure on the sample space Ω = {1, 2, 3, 4, 5, 6} with event space F = all subsets of Ω:

P (F ) = 211 i; all F F.

i F

(b) The following is a valid probability measure on the sample space Ω = {1, 2, 3, 4, 5, 6} with event space F = all subsets of Ω:

P (F ) = 1 if 2 F or 6 F 0 otherwise

(c)If P (G F ) = P (F ) + P (G), then F and G are independent.

(d)P (F |G) ≥ P (G) for all events F and G.

(e)Mutually exclusive (disjoint) events with nonzero probability cannot be independent.

(f) For any finite collection of events Fi; i = 1, 2, · · · , N

N

P ( Ni=1Fi) ≤ P (Fi).

i=1

23.Prove or provide a counterexample for the relation P (F |G)+P (F |Gc) = P (F ).

24.Find the mean, second moment, and variance of a uniform pdf on an interval [a, b).

25.Given a sample space Ω = {0, 1, 2, · · · } define

γ

p(k) = 2k ; k = 0, 1, 2, · · ·

(a)What must γ be in order for p(k) to be a pmf?

(b)Find the probabilities P ({0, 2, 4, 6, · · · }), P ({1, 3, 5, 7, · · · }), and

P ({0, 1, 2, 3, 4, . . . , 20}).

(c)Suppose that K is a fixed integer. Find P ({0, K, 2K, 3K, . . . }).

(d)Find the mean, second moment, and variance of this pmf.

26.Suppose that p(k) is a geometric pmf. Define q(k) = (p(k)+p(−k))/2. Show that this is a pmf and find its mean and variance. Find the probability of the sets {k : |k| ≥ K} and {k : k is a multiple of 3}. Find the probability of the sets {k : k is odd }

27.Define a pmf p(k) = Cλ|k|/|k|! for k Z. Evaluate the constant C and find the mean and variance of this pmf.

28.A probability space consists of a sample space Ω = all pairs of positive integers (that is, Ω = {1, 2, 3, . . . }2) and a probability measure P described by the pmf p defined by

p(k, m) = p2(1 − p)k+m−2 .

(a)Find P ({(k, m) : k ≥ m}).

(b)Find the probability P ({(k, m) : k + m = r}) as a function of r for r = 2, 3, . . . Show that the result is a pmf.

(c)Find the probability P ({(k, m) : k is an odd number}).

(d)Define the event F = {(k, m) : k ≥ m}. Find the conditional pmf pF (k, m) = P ({k, m}|F ). Is this a product pmf?

29. Define the uniform probability density function on [0, 1) in the usual

points and this should be easy. What is F c f(r) dr?

(b) Now define the set F as the collection of all rational numbers in [0, 1), that is, all numbers that can be expressed as k/n for some integers 0 ≤ k < n. What is the integral F f(r) dr? Is

30.Given the uniform pdf on [0, 1], f(x) = 1; x [0, 1], find an expression for P ((a, b)) for all real b > a. Define the cumulative distribution function or cdf F as the probability of the event {x : x ≤ r} as a function of r :

r

Find the cdf for the uniform pdf. Find the probability of the event

31. Let Ω be a unit square {(x, y) : (x, y) 2, −1/2 ≤ x ≤ 1/2, −1/2 ≤ y ≤ 1/2} and let F be the corresponding product Borel field. Is the circle {(x, y) : (x2 + y2)1/2 ≤ 1/2} in F? (Give a plausibility argument.) If so, find the probability of this event if one assumes a uniform density function on the unit square.

32.Given a pdf f, find the cumulative distribution function or cdf F defined as in (2.107) for the exponential, Laplacian, and Gaussian

pdf’s. In the Gaussian case, express the cdf in terms of the Φ function. Prove that if a ≥ b, then F (a) ≥ F (b). What is dFdr(r) ?

P ({(x, y) : x ≤ α}) for all real α. Is f a product pdf?

34.Prove that the product k−dimensional pdf integrates to 1 over

35.Given the one-dimensional exponential pdf, find P ({x : x > r}) and the cumulative distribution function P ({x : x ≤ r}) for r .

36.Given the k−dimensional product doubly exponential pdf, find the probabilities of the following events in k: {x : x0 ≥ 0}, {x : xi >

0, all i = 0, 1, . . . , k − 1}, {x : x0 > x1}.

37.Let (Ω, F) = ( , B( )). Let P1 be the probability measure on this space induced by a geometric pmf with parameter p and let P2 be the probability measure induced on this space by an exponential pdf

with parameter λ. Form the mixture measure P = P1/2+P2/2. Find P ({ω : ω > r}) for all r [0, ∞).

Let Ω = 2 and suppose we have a pdf f(x, y) such that f(x, y) = Cg(x)g(y − x) .

Find the constant C. Find an expression for the probability P ({(x, y) : y ≤ α}) as a function of the parameter α. If f a product pdf?

40. Let Ω = 2 and suppose we have a pdf such that

C|x| −1 ≤ x ≤ 1; −1 ≤ y ≤ x

f(x, y) =

0otherwise .

Find the constant C. Is f a product pdf?

41.Suppose that a probability space has as sample space Rn, n-dimensional Euclidean space. (This is a product space.) Suppose that a multidimensional pdf f is defined on this space by

that is, f(x) = C when −1/2 ≤ xi ≤ 1/2 for i = 0, 1, ··· , n − 1 and is 0 otherwise.

(a)What is C?

(b)Is f a product pdf?

(c)What is P ({x : mini xi ≥ 0}), that is, the probability that the smallest coordinate value is nonnegative.

Suppose next that we have a pdf g defined by

g(x) = K; ||x|| ≤ 1

is the Euclidean norm of the vector x. Thus g is K inside an n-dimensional sphere of radius 1 centered at the origin.

(d)What is the constant K? (You may need to go to a book of integral tables to find this.)

(e)Is this density a product pdf?

42. Let (Ω, F, P ) be a probability space and consider events F, G, and H for which P (F ) > P (G) > P (H) > 0. Events F and G form a partition of Ω, and events F and H are independent. Can events G and H be disjoint?

43.Given a probability space (Ω, F, P ), and let F, G, and H be events such that P (F ∩ G|H) = 1. Which of the following statements are true? Why or why not?

(a)P (F ∩ G) = 1

(b)P (F ∩ G ∩ H) = P (H)

(c)P (F c|H) = 0

(d)H = Ω

44.(Courtesy of Prof. T. Cover) Suppose that the evidence of an event F increases the likelihood of a criminals guilt; that is, if G is the event that the criminal is guilty, then P (G|F ) ≥ P (G). The prosecutor discovers that the event F did not occur. What do you now know about the criminal’s guilt? Prove your answer.

45.Suppose that X is a binary random variable with outputs {a, b} with a

Suppose that p = 0.5, but you are free to choose a and b subject only to the constraint that (a2 + b2)/2 = Eb. Which is a better choice, a = −b or a nonzero with b = 0? What can you say about the minimum achievable Pe?