Econometrics2011

Thus its density is

r!

r!r !

01

where cr = 1 when r 1 and cr = mr 1 when r 1:

Jensen’ Inequality (probabilistic form). If g( ) : Rm ! R is convex, then for any random

Conditional Jensen’ Inequality. If g( ) : Rm ! R is convex, then for any random vectors (y; x) for which Ekyk < 1 and Ekg (y)k < 1;

Hölder’s Inequality. If p > 1 and q > 1 and p1 + 1q = 1; then for any random m n matrices X and Y;

Matrix Cauchy-Schwarz Inequality. Tripathi (1999). For any random x 2 Rm and y 2 R`,

Markov’s Inequality (standard form). For any random vector x and non-negative function

Markov’s Inequality (strong form). For any random vector x and non-negative function

Proof of Minkowski’ Inequality. Note that by rewriting, using the triangle inequality (A.9), and then Hölder’s Inequality to the two expectations

where the second equality picks q to satisfy 1=p+1=q = 1; and the …nal equality uses this fact to make the substitution q = p=(p 1) and then collects terms. Dividing both sides by E(kX + Y kp)(p 1)=p ; we obtain (B.22).

Proof of Markov’s Inequality. Let F denote the distribution function of x: Then

Z

the inequality using the region of integration fg(u) > g: This establishes the strong form (B.25). Since 1 (g(x) > ) 1; the …nal expression is less than 1E(g(x)) ; establishing the standard form (B.24).

Proof of Chebyshev’ Inequality. De…ne y = (x Ex)2 and note that Ey = var (x) : The events fjx Exj > g and y > 2 are equal, so by an application Markov’s inequality we …nd

Pr(jx Exj > ) = Pr(y > 2) 2E(y) = 2 var (x)

as stated.

B.11 Maximum Likelihood

In this section we provide a brief review of the asymptotic theory of maximum likelihood estimation.

When the density of yi is f(y j ) where F is a known distribution function and 2 is an unknown m 1 vector, we say that the distribution is parametric and that is the parameter of the distribution F: The space is the set of permissible value for : In this setting the method of maximum likelihood is an appropriate technique for estimation and inference on : We let denote a generic value of the parameter and let 0 denote its true value.

The joint density of a random sample (y1; :::; yn) is

Yn

fn (y1; :::; yn j ) = f (yi j ) :

i=1

The likelihood of the sample is this joint density evaluated at the observed sample values, viewed as a function of . The log-likelihood function is its natural logarithm

Xn

log L( ) = log f (yi j ) :

i=1

The likelihood score is the derivative of the log-likelihood, evaluated at the true parameter value.

The matrix I is called the information, and the equality (B.32) is called the information matrix equality.

^

The maximum likelihood estimator (MLE) is the parameter value which maximizes the likelihood (equivalently, which maximizes the log-likelihood). We can write this as

2

^

In some simple cases, we can …nd an explicit expression for as a function of the data, but these

^

cases are rare. More typically, the MLE must be found by numerical methods.

^

To understand why the MLE is a natural estimator for the parameter observe that the standardized log-likelihood is a sample average and an estimator of Elog f (yi j ) :

^

As the MLE maximizes the left-hand-side, we can see that it is an estimator of the maximizer of the right-hand-side. The …rst-order condition for the latter problem is

@

0 = @ Elog f (yi j )

^

which holds at = 0 by (B.30). This suggests that is an estimator of 0: In. fact, under

We omit the regularity conditions for Theorem B.11.2, but the result holds quite broadly for models which are smooth functions of the parameters. Theorem B.11.2 gives the general form for the asymptotic distribution of the MLE. A famous result shows that the asymptotic variance is the smallest possible.

Theorem B.11.3 Cramer-Rao Lower Bound. If is an unbiased reg-

The Cramer-Rao Theorem shows that the …nite sample variance of an unbiased estimator is

bounded below by (nI) 1 : This means that the asymptotic variance of the standardized estimator p e 1

n 0 is bounded below by I : In other words, the best possible asymptotic variance among

all (regular) estimators is I 1: An estimator is called asymptotically e¢ cient if its asymptotic variance equals this lower bound. Theorem B.11.2 shows that the MLE has this asymptotic variance, and is thus asymptotically e¢ cient.

Theorem B.11.4 The MLE is asymptotically e¢ cient in the sense that its asymptotic variance equals the Cramer-Rao Lower Bound.

Theorem B.11.4 gives a strong endorsement for the MLE in parametric models.

b b

Finally, consider functions of parameters. If = g( ) then the MLE of is = g( ): This is because maximization (e.g. (B.33)) is una¤ected by parameterization and transformation. Applying the Delta Method to Theorem B.11.2 we conclude that

bb

,and the MLE = g( ) is asymptotically e¢ cient.

Z

ee

= E = (Y ) f (Y ; ) dY :