Prime Numbers

3.3Recognizing smooth numbers

A very important subroutine in many number-theoretic algorithms involves identifying the smooth numbers in a given list of numbers. We have many methods for recognizing these smooth numbers, since any factorization algorithm will do. However, some factorization algorithms, such as trial division, find the smaller prime factors of the number being factored before finding the larger prime factors. Such a method could presumably reject a number for not being y-smooth before completely factoring it. Factorization methods with this property include trial division, the Pollard rho method, and the Lenstra elliptic curve method, the latter two methods being discussed later in the book. Used as smoothness tests, these three factorization methods have the following rough complexities: Trial division takes y1+o(1) operations per number examined, the rho method takes y1/2+o(1) operations, and the elliptic curve method takes about exp((2 ln y ln ln y)1/2) = yo(1) operations. Here an “operation” is an arithmetic step with numbers the size of the specific number being examined. (It should be pointed out that the complexity estimates for both the rho method and the elliptic curve method are heuristic.)

Sometimes we can use a sieve to recognize smooth numbers, and when we can, it is very fast. For example, if we have a string of consecutive integers or more generally a string of consecutive values of a polynomial with integer coe cients (and with low degree), and if this string has length L ≥ y, with maximal member M , then the time to examine every single one of the L numbers for y-smoothness is about L ln ln M ln ln y, or about ln ln M ln ln y bit operations per number. (The factor ln ln M arises from using approximate logarithms, as discussed in Section 3.2.5.) In fact, sieving is so fast that the run time is dominated more by retrieving numbers from memory than by doing actual computations.

In this section we shall discuss an important new method of D. Bernstein (see [Bernstein 2004d]), which can recognize the smooth numbers in any set of at least y numbers, and whose amortized time per number examined is almost as fast as sieving: It is (ln2 y ln M )1+o(1) bit operations per number, if the numbers are at most M . To achieve this complexity, though, one must use sophisticated subroutines for large-integer arithmetic, such as the fast Fourier transform or equivalent convolution techniqes (see our Chapter 8.8 and [Bernstein 2004e]).

We shall illustrate the Bernstein method with the smoothness bound y set at 20, and with the set of numbers being examined being 1001, 1002, . . . , 1008. (It is not important that the numbers be consecutive, it is just easier to keep track of them for the illustration.) A moment’s inspection shows the 20smooth numbers in the list to be the first and last, namely 1001 and 1008. The algorithm not only tells us this, it gives the largest 20-smooth divisor for each number in the list.

The plan of the Bernstein algorithm, as applied to this example, is first to find the product of all of the primes up to 20, namely 9699690, and then reduce this product modulo each of the eight numbers on our list. Say x is on

our list and 9699690 mod x = r. Then r = ab, where a = gcd(9699690, x) and gcd(b, x) = 1. If the highest exponent on any prime in the prime factorization of x is bounded above by 2e, then gcd(r2e mod x, x) is the 20-smooth part of x. So in our case, we can take e = 4, since 224 > 1008. Let us see what happens for the number x = 1008. First, we have r = 714. Next we take 7142i mod 1008 for i = 1, 2, 3, 4, getting 756, 0, 0, 0. Of course, we ought to be smart enough to stop when we get the first 0, since this already implies that 1008 is 20-smooth. If we apply this idea to x = 1004, we get r = 46, and the requisite powers are 108, 620, 872, 356. We take gcd(356, 1004) and find it to be 4. Surely this must be the long way around! But as we shall see, the method scales beautifully. Further, we shall see that it is not interesting to focus on any one number, but on all numbers together.

We form the product 9699690 of the primes up to 20 via a “product tree;” see [Bernstein 2004e]. This is just the binary tree as so:

Product tree for P = {2, 3, 5, 7, 11, 13, 17, 19}

We start at the leaves, multiplying ourselves through the binary tree to the root, whose label is the product P = 9699690 of all of the leaves.

We wish to find each residue P mod x as x varies over the numbers we are examining for smoothness. If we do this separately for each x, since P is so large, the process will take too long. Instead, we first multiply all the numbers x together! We do this as with the primes, with a product tree. However, we never need to form a product that is larger than P ; say we simply indicate such large products with an asterisk. Let us consider the product tree T formed from the numbers 1001, 1002, . . . , 1008:

Product tree T for X = {1001, 1002, . . . , 1008}

Pmod T for a given integer P and a given product tree T is the corresponding tree in which each label in T being replaced by its remainder when it is divided into P . This relabeling is achieved by replacing the label R at the root of T with P mod R, and then working toward the leaves, each entry is replaced with the remainder after dividing this entry into the new label of its parent. We illustrate with the product tree T formed from 1001, . . . , 1008 and the number

P= 9699690 found in our first product tree. We may evidently convert each asterisk in T to P .

Remainder tree P mod T

For each x that we are examining for smoothness, the corresponding leaf value in the remainder tree is P mod x. Take this residue, sequentially square modulo x the requisite number of times, and at last take the gcd of the final result with x. A value of 0 signifies that x is smooth over the primes in P , and a nonzero value is itself the largest divisor of x that is smooth over the primes in P . Here is pseudocode for this beautiful algorithm.

Algorithm 3.3.1 (Batch smoothness test (Bernstein)). We are given a fi- nite set X of positive integers and a finite set P of primes. For each x X , this algorithm returns the largest divisor of x composed of primes from P.

1. [Compute product trees]

Compute the product tree for P;

Set P as the product of the members of P;

// We find P at the root of the product tree for P. Compute the product tree T for X , but only for products at most P ;

3. [Find smooth parts]

Set e as the least positive integer with max X ≤ 22e ; for(x X ){

Find P mod x in the remainder tree P mod T ;

// No additional mod calculation is necessary.

r = P mod x;

s = r2e mod x; // Compute s by sequential squaring and reducing.

g = gcd(s, x);

return “the largest divisor of x composed of primes from P is g”;

}

The Bernstein Algorithm 3.3.1 is an important addition to the repertoire of computational number theory. It can profitably be used to speed up various other algorithms where smoothness is desired. One example arises in the step [Factor orders] of the Atkin–Morain primality test (Algorithm 7.6.3). Algorithm 3.3.1 can even be useful in situations in which sieving is completely appropriate, such as in the quadratic sieve and number field sieve factoring algorithms (see Chapter 6). Indeed, in these algorithms, the yield rate of smooth numbers can be so small, it is advantageous to sieve only partially (forget about small primes in the factor base, which involve the most memory retrievals), tune the sieve to report candidates with a large smooth divisor, and then run Algorithm 3.3.1 on the much smaller, but still large, reported set. This idea of removing small primes from a sieve can be found already in [Pomerance 1985], but with Algorithm 3.3.1 it can be used more aggressively.

3.4 Pseudoprimes

Suppose we have a theorem, “If n is prime, then S is true about n,” where “S ” is some easily checkable arithmetic statement. If we are presented with a large number n, and we wish to decide whether n is prime or composite, we may very well try out the arithmetic statement S and see whether it actually holds for n. If the statement fails, we have proved the theorem that n is composite. If the statement holds, however, it may be that n is prime, and it also may be that n is composite. So we have the notion of S-pseudoprime, which is a composite integer for which S holds.

One example might be the theorem, If n is prime, then n is 2 or n is odd. Certainly this arithmetic property is easily checked for any given input n. However, as one can readily see, this test is not very strong evidence of primality, since there are many more pseudoprimes around for this test than there are genuine primes. Thus, for the concept of “pseudoprime” to be useful, it will have to be the case that there are, in some appropriate sense, few of them.

3.4.1Fermat pseudoprimes

The fact that the residue ab (mod n) may be rapidly computed (see Algorithm 2.1.5) is fundamental to many algorithms in number theory. Not least of these is the exploitation of Fermat’s little theorem as a means to distinguish between primes and composites.

132 Chapter 3 RECOGNIZING PRIMES AND COMPOSITES

Proofs of Fermat’s little theorem may be found in any elementary number theory text. One particularly easy proof uses induction on a and the binomial theorem to expand (a + 1)n.

When a is coprime to n we may divide both sides of (3.2) by a to obtain

Thus, (3.3) holds whenever n is prime and n does not divide a.

We say that a composite number n is a (Fermat) pseudoprime if (3.2) holds. For example, n = 91 is a pseudoprime base 3, since 91 is composite and 391 ≡ 3 (mod 91). Similarly, 341 is a pseudoprime base 2. The base a = 1 is uninteresting, since every composite number is a pseudoprime base 1. We suppose now that a ≥ 2.

Theorem 3.4.2. For each fixed integer a ≥ 2, the number of Fermat pseudoprimes base a that are less than or equal to x is o(π(x)) as x → ∞. That is, Fermat pseudoprimes are rare compared with primes.

For pseudoprimes defined via the congruence (3.3), this theorem was first proved in [Erd˝os 1950]. For the possibly larger class of pseudoprimes defined via (3.2), the theorem was first proved in [Li 1997].

Theorem 3.4.2 tells us that using the Fermat congruence to distinguish between primes and composites is potentially very useful. However, this was known as a practical matter long before the Erd˝os proof.

Note that odd numbers n satisfy (3.3) for a = n−1, so that the congruence does not say very much about n in this case. If (3.3) holds for a pair n, a, where 1 < a < n − 1, we say that n is a probable prime base a. Thus, if n is a prime, then it is a probable prime base a for every integer a with 1 < a < n − 1. Theorem 3.4.2 asserts that for a fixed choice of a, most probable primes base a are actually primes. We thus have a simple test to distinguish between members of a set that contains a sparse set of composite numbers and all of the primes exceeding a+1, and members of the set of the remaining composite numbers exceeding a + 1.

Algorithm 3.4.3 (Probable prime test). We are given an integer n > 3 and an integer a with 2 ≤ a ≤ n − 2. This algorithm returns either “n is a probable prime base a” or “n is composite.”

2. [Return decision]

if(b == 1) return “n is a probable prime base a”; return “n is composite”;

We have seen that with respect to a fixed base a, pseudoprimes (that is, probable primes that are composite) are sparsely distributed. However, paucity notwithstanding, there are infinitely many.

Theorem 3.4.4. For each integer a ≥ 2 there are infinitely many Fermat pseudoprimes base a.

so that n is composite. Using (3.2) for the prime p we get upon squaring both

we see that n − 1 is the sum of an even number of terms of the same parity, so n − 1 must be even. So far, we have learned that both 2 and p are divisors of n − 1, so that 2p must likewise be a divisor. Then a2p − 1 is a divisor of an−1 − 1. But a2p − 1 is a multiple of n, so that (3.3) holds, as does (3.2).

3.4.2Carmichael numbers

In search of a simple and quick method of distinguishing prime numbers from composite numbers, we might consider combining Fermat tests for various bases a. For example, though 341 is a pseudoprime base 2, it is not a pseudoprime base 3. And 91 is a base-3, but not a base-2 pseudoprime. Perhaps there are no composites that are simultaneously pseudoprimes base 2 and 3, or if such composites exist, perhaps there is some finite set of bases such that there are no pseudoprimes to all the bases in the set. It would be nice if this were true, since then it would be a simple computational matter to test for primes.

However, the number 561 = 3 · 11 · 17 is not only a Fermat pseudoprime to both bases 2 and 3, it is a pseudoprime to every base a. It may be a shock that such numbers exist, but indeed they do. They were first discovered by R. Carmichael in 1910, and it is after him that we name them.

Definition 3.4.5. A composite integer n for which an ≡ a (mod n) for every integer a is a Carmichael number.

It is easy to recognize a Carmichael number from its prime factorization.

Theorem 3.4.6 (Korselt criterion). An integer n is a Carmichael number if and only if n is positive, composite, squarefree, and for each prime p dividing n we have p − 1 dividing n − 1.

Remark. A. Korselt stated this criterion for Carmichael numbers in 1899, eleven years before Carmichael came up with the first example. Perhaps

An identical formula is conjectured for P2(x), the number of base-2 pseudoprimes up to x. It has been proved, see [Pomerance 1981], that both

C(x) < x1−ln ln ln x/ ln ln x,

P2(x) < x1−ln ln ln x/(2 ln ln x),

for all su ciently large values of x.

3.5 Probable primes and witnesses

The concept of Fermat pseudoprime, developed in the previous section, is a good one, since it is easy to check and for each base a > 1 there are few pseudoprimes compared with primes (Theorem 3.4.2). However, there are composites, the Carmichael numbers, for which (3.2) is useless as a means of recognizing them as composite. As we have seen, there are infinitely many Carmichael numbers. There are also infinitely many Carmichael numbers that have no small prime factor (see [Alford et al. 1994b]), so that for these numbers, even the slightly stronger test (3.3) is computationally poor.

We would ideally like an easy test for which there are no pseudoprimes. Failing this, we would like a family of tests, such that each composite is not a pseudoprime for a fixed, positive fraction of the tests in the family. The Fermat family does not meet this goal, since there are infinitely many Carmichael numbers. However, a slightly di erent version of Fermat’s little theorem (Theorem 3.4.1) does meet this goal.

The proof of Theorem 3.5.1 uses only Fermat’s little theorem in the form (3.3) and the fact that for n an odd prime, the only solutions to x2 ≡ 1 (mod n) in Zn are x ≡ ±1 (mod n). We leave the details to the reader.

In analogy to probable primes, we can now define a strong probable prime base a. This is an odd integer n > 3 for which (3.4) holds for a, where 1 < a < n − 1. Since every strong probable prime base a is automatically a probable prime base a, and since every prime greater than a + 1 is a strong probable prime base a, the only di erence between the two concepts is that possibly fewer composites pass the strong probable prime test.

Algorithm 3.5.2 (Strong probable prime test). We are given an odd number n > 3, represented as n = 1 + 2st, with t odd. We are also given an integer a with 1 < a < n − 1. This algorithm returns either “n is a strong probable prime

if(b == 1 or b == n − 1) return “n is a strong probable prime base a”;

J. Selfridge rediscovered the test and popularized it.

We now consider the possibility of showing that an odd number n is composite by showing that (3.4) fails for a particular number a. For example, we saw in the previous section that 341 is pseudoprime base 2. But (3.4) does not hold for n = 341 and a = 2. Indeed, we have 340 = 22 · 85, 285 ≡ 32 (mod 341), and 2170 ≡ 1 (mod 341). In fact, we see that 32 is a nontrivial square root of 1 (mod 341).

Now consider the pair n = 91 and a = 10. We have 90 = 21 · 45 and 1045 ≡ −1 (mod 91). So (3.4) holds.

Definition 3.5.3. We say that n is a strong pseudoprime base a if n is an odd composite, n − 1 = 2st, with t odd, and (3.4) holds.

Thus, 341 is not a strong pseudoprime base 2, while 91 is a strong pseudoprime base 10. J. Selfridge proposed using Theorem 3.5.1 as a pseudoprime test in the early 1970s, and it was he who coined the term “strong pseudoprime.” It is clear that if n is a strong pseudoprime base a, then n is a pseudoprime base a. The example with n = 341 and a = 2 shows that the converse is false.

For an odd composite integer n we shall let

and let S(n) = #S(n). The following theorem was proved independently in [Monier 1980] and [Rabin 1980].

Recall that ϕ(n) is Euler’s function evaluated at n. It is the number of integers in [1, n] coprime to n; that is, the order of the group Zn. If we know the prime factorization of n, it is easy to compute ϕ(n): We have

ϕ(n) = n p|n(1 − 1/p), where p runs over the prime factors of n.

Before we prove Theorem 3.5.4, we first indicate why it is a significant result. If we have an odd number n and we wish to determine whether it is prime or composite, we might try verifying (3.4) for some number a with 1 < a < n − 1. If (3.4) fails, then we have proved that n is composite. Such a number a might be said to be a witness for the compositeness of n. In fact, we make a formal definition.

Definition 3.5.5. If n is an odd composite number and a is an integer in [1, n − 1] for which (3.4) fails, we say that a is a witness for n. Thus, for

an odd composite number n, a witness is a base for which n is not a strong pseudoprime.

A witness for n is thus the key to a short proof that n is composite. Theorem 3.5.4 implies that at least 3/4 of all integers in [1, n − 1] are

witnesses for n, when n is an odd composite number. Since one can perform a strong pseudoprime test very rapidly, it is easy to decide whether a particular number a is a witness for n. All said, it would seem that it is quite an easy task to produce witnesses for odd composite numbers. Indeed, it is, if one uses a probabilistic algorithm. The following is often referred to as “the Miller–Rabin test,”, though as one can readily see, it is Algorithm 3.5.2 done with a random choice of the base a. (The original test in [Miller 1976] was somewhat more complicated and was a deterministic, ERH-based test. It was M. Rabin, see [Rabin 1976, 1980], who suggested a probabilistic algorithm as below.)

Algorithm 3.5.6 (Random compositeness test). We are given an odd number n > 3. This probabilistic algorithm attempts to find a witness for n and thus prove that n is composite. If a is a witness, (a, YES) is returned; otherwise, (a, NO) is returned.

1. [Choose possible witness]

Choose random integer a [2, n − 2];

Via Algorithm 3.5.2 decide whether n is a strong probable prime base a;

2. [Declaration]

if(n is a strong probable prime base a) return (a, NO); return (a, YES);

One can see from Theorem 3.5.4 that if n > 9 is an odd composite, then the probability that Algorithm 3.5.6 fails to produce a witness for n is < 1/4. No one is stopping us from using Algorithm 3.5.6 repeatedly. The probability that we fail to find a witness for an odd composite number n with k (independent) iterations of Algorithm 3.5.6 is < 1/4k. So clearly we can make this probability vanishingly small by choosing k large.

Algorithm 3.5.6 is a very e ective method for recognizing composite numbers. But what does it do if we try it on an odd prime? Of course it will fail to produce a witness, since Theorem 3.5.1 asserts that primes have no witnesses.

Suppose n is a large odd number and we don’t know whether n is prime or composite. Say we try 20 iterations of Algorithm 3.5.6 and fail each time to produce a witness. What should be concluded? Actually, nothing at all can be concluded concerning whether n is prime or composite. Of course, it is reasonable to strongly conjecture that n is prime. The probability that 20 iterations of Algorithm 3.5.6 fail to produce a witness for a given odd composite is less than 4−20, which is less than one chance in a trillion. So yes, n is most likely prime. But it has not been proved prime and in fact might not be.