Chaitin. Algorithmic information theory

By applying Theorem I0(a,b) to (6.1) and (6.2), we see that Theorem I4 holds with c = sim(D) + 2.

How does the computer D work? First of all, it checks whether the free data that it has been given is or t . These two cases can be distinguished, for by Theorem I0(c) it is impossible for t to be equal to .

(a)If D has been given the free data , it enumerates T without repetitions and simulates the computer determined by the set of all requirements of the form

Thus (s; n) is taken as a requirement i n − lg PC (s)+1. Hence the number of programs p of length n such that D(p; ) = s is 1 if n − lg PC (s) + 1 and is 0 otherwise, which immediately yields (6.1).

However, we must check that the requirements (6.3) on D satisfy the Kraft inequality and are consistent.

X2−jpj = 2lg PC (s) < PC (s):

D(p; )=s

Hence

X2−jpj < X PC (s) 1

by Theorem I0(j). Thus the hypothesis of Theorem I2 is satis-ed, the requirements (6.3) indeed determine a computer, and the proof of (6.1) and Theorem I4(a) is complete.

(b)If D has been given the free data t , it enumerates Tt without repetitions and simulates the computer determined by the set of all requirements of the form

Thus (s; n) is taken as a requirement i n − lg PC (s=t) + 1. Hence the number of programs p of length n such that D(p; t ) = s is 1 if n − lg PC (s=t) + 1 and is 0 otherwise, which immediately yields (6.2).

However, we must check that the requirements (6.4) on D satisfy the Kraft inequality and are consistent.

X2−jpj = 2lg PC (s=t) < PC (s=t):

D(p;t )=s

Hence

X2−jpj < X PC (s=t) 1

by Theorem I0(k). Thus the hypothesis of Theorem I2 is satis ed, the requirements (6.4) indeed determine a computer, and the proof of (6.2) and Theorem I4(b) is complete. Q.E.D.

Theorem I5

For each computer C there is a constant c such that

(

(b)H(s) = − log2 P (s) + O(1);

H(s=t) = − log2 P (s=t) + O(1):

Proof

Theorem I5(a) follows immediately from Theorem I4 using the fact

that

P (s) 2−H(s)

and

P (s=t) 2−H(s=t)

(Theorem I0(l,m)). Theorem I5(b) is obtained by taking C = U in Theorem I4 and also using these two inequalities. Q.E.D.

Remark

Theorem I4(a) extends Theorem I0(a,b) to probabilities. Note that Theorem I5(a) is not an immediate consequence of our weak de nition of a universal computer.

In words, \a minimal program tells us its size as well as its output." This is just one pair of a large family of identities, as we now proceed to show.

Keeping Theorem I9(a) in mind, consider modifying the computer C used in the proof of Theorem I1(f) so that it also measures the lengths H(s) and H(t=s) of its subroutines s and p, and halts indicating (s, t, H(s), H(t=s)) to be the result of the computation instead of (s; t). It follows that

H(s; t) = H(s; t; H(s); H(t=s)) + O(1)

and

H(H(s); H(t=s)=s; t) = O(1):

In fact, it is easy to see that

H(H(s); H(t); H(t=s); H(s=t); H(s; t)=s; t) = O(1);

which implies

H(H(s : t)=s; t) = O(1):

And of course these identities generalize to tuples of three or more strings.

6.4Random Strings

In this section we begin studying the notion of randomness or algorithmic incompressibility that is associated with the program-size complexity measure H.

Theorem I10

(Bounds on the complexity of positive integers).

(a) Pn 2−H(n) 1.

Consider a computable total function f that carries positive integers into positive integers.

(b)Pn 2−f (n) = 1 ) H(n) > f (n) in nitely often.

(c)Pn 2−f (n) < 1 ) H(n) f (n) + O(1).

Proof

(a) By Theorem I0(l,j),

diverges, and

H(n) f (n)

held for all but nitely many values of n, then

X 2−H(n) n

would also diverge. But this would contradict Theorem I10(a), and thus

H(n) > f (n)

in nitely often.

(c) If

X 2−f (n) n

converges, there is an n0 such that

X 2−f (n) 1:

n n0

Thus the Kraft inequality that Theorem I2 tells us is a necessary and su cient condition for the existence a computer C determined by the requirements

f(n; f (n)) : n n0g

is satis ed. It follows that

H(n) f (n) + sim(C)

for all n n0. Q.E.D.

Remark

H(n) can in fact be characterized as a minimal function computable in the limit from above that lies just on the borderline between the convergence and the divergence of

X 2−H(n):

Theorem I11

(Maximal complexity nite bit strings).

(a)maxjsj=n H(s) = n + H(n) + O(1).

(b)# (fs : jsj = n & H(s) n + H(n) − kg) 2n−k+O(1).

Proof

Consider a string s of length n. By Theorem I9(a),

H(s) = H(n; s) + O(1) = H(n) + H(s=n) + O(1):

We now obtain Theorem I11(a,b) from this estimate for H(s). There is a computer C such that

C(p; jpj ) = p

for all p. Thus

H(s=n) n + sim(C);

and

H(s) n + H(n) + O(1):

On the other hand, by Theorem I0(q), fewer than 2n−k of the s satisfy

H(s=n) < n − k:

Hence fewer than 2n−k of the s satisfy

H(s) < n − k + H(n) + O(1):

This concludes the proof of Theorem I11. Q.E.D.

De nition of Randomness (Finite Case)

In the case of nite strings, randomness is a matter of degree. To the question \How random is s?" one must reply indicating how close

H(s) is to the maximum possible for strings of its size. A string s is most random if H(s) is approximately equal to jsj+H(jsj). As we shall see in the next chapter, a good cut-o to choose between randomness and non-randomness is H(s) jsj.

The natural next step is to de ne an in nite string to be random if all its initial segments are nite random strings. There are several other possibilities for de ning random in nite strings and real numbers, and we study them at length in Chapter 7. To anticipate, the undecidability of the halting problem is a fundamental theorem of recursive function theory. In algorithmic information theory the corresponding theorem is as follows: The base-two representation of the probability Ω that U halts is a random (i.e., maximally complex) in nite string.