Chaitin. Algorithmic information theory

which diverges. Q.E.D.

8.4Incompleteness Theorems for Random Reals: H(Axioms)

Theorem C is a remarkable extension of Theorem R6:

(1)We have seen that the information content of knowing the rst n bits of Ω is n − c.

(2)Now we show that the information content of knowing any n bits of Ω (their positions and 0/1 values) is n − c.

Lemma C

X #fs : H(s) < ng2−n 1:

n

Proof

1 Ω X 2−H(s)

s

= X #fs : H(s) = ng2−n = X #fs : H(s) = ng2−n X 2−k

Q.E.D.

Theorem C

If a theory has H(axiom) < n, then it can yield at most n + c (scattered) bits of Ω.

Proof

Consider a particular k and n. If there is an axiom with H(axiom) < n which yields n + k scattered bits of Ω, then even without knowing which axiom it is, we can cover Ω with an r.e. set of intervals of measure

= #fs : H(s) < ng2−n−k:

But by the preceding lemma, we see that

Thus if even one theory with H < n yields n+k bits of Ω, for any n, we get a cover for Ω of measure 2−k. This can only be true for nitely many values of k, or Ω would not be Martin-L¨of random. Q.E.D.

Corollary C

No n-bit theory ever yields more than n + H(n) + c bits of Ω.

Proof

This follows immediately from Theorem C and the fact that

H(axiom) jaxiomj + H(jaxiomj) + c;

which is an immediate consequence of Theorem I11(a). Q.E.D.

Lemma C2

If g(n) is computable and unbounded, then H(n) < g(n) for in-nitely many values of n.

Proof

De ne the inverse of g as follows:

g−1(n) = min k:

g(k) n

Then it is easy to see that for all su ciently large values of n:

H(g−1(n)) H(n) + O(1) O(log n) < n g(g−1(n)):

I.e., H(k) < g(k) for all k = g−1(n) and n su ciently large. Q.E.D.

Corollary C2

Let g(n) be computable and unbounded. For in nitely many n, no n-bit theory yields more than n + g(n) + c bits of Ω.

Proof

This is an immediate consequence of Corollary C and Lemma C2. Q.E.D.

Note

In appraising Corollaries C and C2, the trivial formal systems in which there is always an n-bit axiom that yields the rst n bits of Ω should be kept in mind. Also, compare Corollaries C and A, and Corollaries C2 and A2.

In summary,

Theorem D

There is an exponential diophantine equation

which has only nitely many solutions x1; : : : ; xm if the nth bit of Ω is a 0, and which has in nitely many solutions x1; : : : ; xm if the nth bit of Ω is a 1. Let us say that a formal theory \settles k cases" if it enables one to prove that the number of solutions of (8.4) is nite or that it is in nite for k speci c values (possibly scattered) of the parameter n. Let f (n) and g(n) be computable functions.

(a)P 2−f (n) < 1 ) all n-bit theories settle n + f (n) + O(1) cases.

(b)P 2−f (n) = 1 & f (n) f (n + 1) ) for in nitely many n, there

is an n-bit theory that settles n + f (n) cases.

(c)H(theory) < n ) it settles n + O(1) cases.

(d)n-bit theory ) it settles n + H(n) + O(1) cases.

(e)g unbounded ) for in nitely many n, all n-bit theories settlen + g(n) + O(1) cases.

Proof

The theorem combines Theorem R8, Corollaries A and B, Theorem C, and Corollaries C and C2. Q.E.D.

are so complicated, and in what sense we can understand them.1 I.e., how do biological \laws" compare with the laws of physics?2

We have seen that Ω is about as random, patternless, unpredictable and incomprehensible as possible; the pattern of its bit sequence de-es understanding. However with computations in the limit, which is equivalent to having an oracle for the halting problem,3 Ω seems quite understandable: it becomes a computable sequence. Biological evolution is the nearest thing to an in nite computation in the limit that we will ever see: it is a computation with molecular components that has proceeded for 109 years in parallel over the entire surface of the earth. That amount of computing could easily produce a good approximation to Ω, except that that is not the goal of biological evolution. The goal of evolution is survival, for example, keeping viruses such as those that cause AIDS from subverting one's molecular mechanisms for their own purposes.

This suggests to me a very crude evolutionary model based on the game of matching pennies, in which players use computable strategies for predicting their opponent's next play from the previous ones.4 I don't think it would be too di cult to formulate this more precisely and to show that prediction strategies will tend to increase in programsize complexity with time.

Perhaps biological structures are simple and easy to understand only if one has an oracle for the halting problem.

1Compare my previous thoughts on theoretical biology, Chaitin (1970b) and Chaitin (1979). There I suggest that mutual information H (s : t) can be used to pick out the highly correlated regions of space that contain organisms. This view is static; here we are concerned with the dynamics of the situation. Incidentally, it is possible to also regard these papers as an extremely abstract discussion of musical structure and metrics between compositional styles.

2In Chaitin (1985) I examine the complexity of physical laws by actually programming them, and the programs turn out to be amazingly small. I use APL instead of LISP.

3See Chaitin (1977a,1976c).

4See the discussion of matching pennies in Chaitin (1969a).

G.J. Chaitin (1974b), \Information-theoretic limitations of formal systems," Journal of the ACM 21, pp. 403{424.

G.J. Chaitin (1975a), \Randomness and mathematical proof," Scienti c American 232 (5), pp. 47{52. Also published in the French, Japanese, and Italian editions of Scienti c American.

G.J. Chaitin (1975b), \A theory of program size formally identical to information theory," Journal of the ACM 22, pp. 329{340.

G.J. Chaitin (1976a), \A toy version of the LISP language," Report RC 5924, Yorktown Heights: IBM Watson Research Center.

G.J. Chaitin (1976b), \Information-theoretic characterizations of recursive in nite strings," Theoretical Computer Science 2, pp. 45{48.

G.J. Chaitin (1976c), \Algorithmic entropy of sets," Computers & Mathematics with Applications 2, pp. 233{245.

G.J. Chaitin (1977a), \Program size, oracles, and the jump operation," Osaka Journal of Mathematics 14, pp. 139{149.

G.J. Chaitin (1977b), \Algorithmic information theory," IBM Journal of Research and Development 21, pp. 350{359, 496.

G.J. Chaitin and J.T. Schwartz (1978), \A note on Monte Carlo primality tests and algorithmic information theory," Communications on Pure and Applied Mathematics 31, pp. 521{527.

G.J. Chaitin (1979), \Toward a mathematical de nition of `life'," in R.D. Levine and M. Tribus (1979), The Maximum Entropy Formalism, Cambridge, Mass.: MIT Press, pp. 477{498.

G.J. Chaitin (1982a), \Algorithmic information theory," in Encyclopedia of Statistical Sciences I, New York: Wiley, pp. 38{41.

G.J. Chaitin (1982b), \G¨odel's theorem and information,"

International Journal of Theoretical Physics 22, pp. 941{954. Reprinted in Tymoczko (1986).

217

G.J. Chaitin (1985), \An APL2 gallery of mathematical physics|a course outline," Proceedings Japan 85 APL Symposium, Publication N:GE18-9948-0, IBM Japan, pp. 1{56.

G.J. Chaitin (1986), \Randomness and G¨odel's theorem," Mondes en Developpement 14 (54{55), pp. 125{128, 356.

G.J. Chaitin (1987a), \Computing the busy beaver function," in T.M. Cover and B. Gopinath (1987), Open Problems in Communication and Computation, New York: Springer, pp. 108{ 112.

G.J. Chaitin (1987b), \Incompleteness theorems for random reals," Advances in Applied Mathematics 8, pp. 119{146.

G.J. Chaitin (1987c), Information, Randomness & Incompleteness | Papers on Algorithmic Information Theory, Singapore: World Scienti c.

J.H. Conway (1986), private communication.

R. Courant and H. Robbins (1941), What is Mathematics?,

Oxford: Oxford University Press.

M.F. Cowlishaw (1985), The REXX Language, Englewood Cli s, NJ: Prentice-Hall.

M. Davis, H. Putnam and J. Robinson (1961), \The decision problem for exponential diophantine equations," Annals of Mathematics 74, pp. 425{436.

M. Davis (1965), The Undecidable|Basic Papers on Undecidable Propositions, Unsolvable Problems and Computable Functions, Hewlett: Raven Press.

M. Davis, Y.V. Matijasevic and J. Robinson (1976), \Hilbert's tenth problem. Diophantine equations: positive aspects of a negative solution," in Mathematical Developments Arising from Hilbert Problems, Providence: American Mathematical Society, pp. 323{378.

M. Davis (1978), \What is a computation?," in L.A. Steen (1978), Mathematics Today, New York: Springer, pp. 241{267.

S. Feferman et al. (1986), Kurt G¨odel: Collected Works I: Publications 1929-1936, New York: Oxford University Press.

W. Feller (1970), An Introduction to Probability Theory and Its Applications I, New York: Wiley.

J. Ford (1983), \How random is a coin toss?," Physics Today 36 (4), pp. 40{47.

K. Godel¨ (1931), \On formally undecidable propositions of

Principia mathematica and related systems I," Monatshefte f¨ur Mathematik und Physik 38, pp. 173{198. Reprinted in Feferman (1986).

G.H. Hardy (1952), A Course of Pure Mathematics, Cambridge: Cambridge University Press.

R.V. Jensen (1987), \Classical chaos," American Scientist 75, pp. 168{181.

J.P. Jones et al. (1976), \Diophantine representation of the set of prime numbers," American Mathematical Monthly 83, pp. 449{ 464.

J.P. Jones and Y.V. Matijasevic (1984), \Register machine proof of the theorem on exponential diophantine representation of enumerable sets," Journal of Symbolic Logic 49, pp. 818{829.

M. Levin (1974), \Mathematical Logic for Computer Scientists," Report TR-131, Cambridge, Mass.: MIT Project MAC.

B.B. Mandelbrot (1982), The Fractal Geometry of Nature,

San Francisco: Freeman.

P. Martin-Lof¨ (1966), \The de nition of random sequences,"

Information and Control 9, pp. 602{619.