Prime Numbers

482 Chapter 9 FAST ALGORITHMS FOR LARGE-INTEGER ARITHMETIC

}

3. [Make ping-pong parity decision]

if(d even) return (complex data at X); return (complex data at Y );

The useful loop aspect of this algorithm is the fact that the loop variable j runs contiguously (from J down), and so on a vector machine one may process chunks of data all at once, by picking up, then putting back, data as vectors.

Incidentally, to perform an inverse FFT is extremely simple, once the forward FFT is implemented. One approach is simply to look at Definition 9.5.3 and observe that the root g can be replaced by g−1, with a final overall normalization 1/D applied to an inverse FFT. But when complex fields are used, so that g−1 = g , the procedure for F F T −1 can be, if one desires, just a sequence:

Though these Cooley–Tukey and Gentleman–Sande FFTs are most often invoked over the complex numbers, so that the root is g = e2πi/D, say, they are useful also as number-theoretical transforms, with operations carried out in a finite ring or field. In either the complex or finite-field cases, it is common that a signal to be transformed has all real elements, in which case we call the signal “pure-real.” This would be so for complex signal x CD but such that xj = aj + 0i for each j [0, D − 1]. It is important to observe that the analogous signal class can occur in certain fields, for example Fp2 when p ≡ 3 (mod 4). For in such fields, every element can be represented as xj = aj + bj i, and we can say that a signal x FDp2 is pure-real if and only if every bj is zero. The point of the pure-real signal class designation is that in general, an FFT for pure-real signals has about 1/2 the usual complexity. This makes sense from an information-theoretical standpoint: Indeed, there is “half as much” data in a pure-real signal. A basic way to cut down thus the FFT complexity for pure-real signals is to embed half of a pure-real signal in the imaginary parts of a signal of half the length, i.e., to form a complex signal

yj = xj + ixj+D/2

for j [0, D/2 − 1]. Note that signal y now has length D/2. One then performs a full, half-length, complex FFT and then uses some reconstruction

formulae to recover the correct DFT of the original signal x. An example of this pure-real signal approach for number-theoretical transforms as applied to cyclic convolution is embodied in Algorithm 9.5.22, with split-radix symbolic pseudocode given in [Crandall 1997b] (see Exercise 9.51 for discussion of the negacyclic scenario).

Incidentally, there are yet lower-complexity FFTs, called split-radix FFTs, which employ an identity more complicated than the Danielson–Lanczos formula. And there is even a split-radix, pure-real-signal FFT due to Sorenson that is quite e cient and in wide use [Crandall 1994b]. The vast “FFT forest” is replete with specially optimized FFTs, and whole texts have been written in regard to the structure of FFT algorithms; see, for example, [Van Loan 1992].

Even at the close of the 20th century there continue to be, every year, a great many published papers on new FFT optimizations. Because our present theme is the implementation of FFTs for large-integer arithmetic, we close this section with one more algorithm: a “parallel,” or “piecemeal,” FFT algorithm that is quite useful in at least two practical settings. First, when signal data are particularly numerous, the FFT must be performed in limited memory. In practical terms, a signal might reside on disk memory, and exceed a machine’s physical random-access memory. The idea is to “spool” pieces of the signal o the larger memory, process them, and combine in just the right way to deposit a final FFT in storage. Because computations occur in large part on separate pieces of the transform, the algorithm can also be used in a parallel setting, with each separate processor handling a respective piece of the FFT. The algorithm following has been studied by various investigators [Agarwal and Cooley 1986], [Swarztrauber 1987], [Ashworth and Lyne 1988], [Bailey 1990], especially with respect to practical memory usage. It is curious that the essential ideas seem to have originated with [Gentleman and Sande 1966]. Perhaps, in view of the extreme density and proliferation of FFT research, one might forgive investigators for overlooking these origins for two decades.

The parallel-FFT algorithm stems from the observation that a length- (D = W H) DFT can be performed by tracking over rows and columns of an H × W (height times width) matrix. Everything follows from the following algebraic reduction of the DFT X of x:

=xJ+M W g−(J+M W )(K+N H)

484 Chapter 9 FAST ALGORITHMS FOR LARGE-INTEGER ARITHMETIC

where g, gH , gW are roots of unity of order W H, H, W , respectively, and the indices (K + N H) have K [0, H − 1], N [0, W − 1]. The last double sum here can be seen to involve the FFTs of rows and columns of a certain matrix, as evidenced in the explicit algorithm following:

Algorithm 9.5.7 (Parallel, “four-step” FFT). Let x be a signal of length

D = W H. For algorithmic e ciency we consider the input signal x to be a matrix T arranged in “columnwise order”; i.e., for j [0, W − 1] the j-th column of T contains the (originally contiguous) H elements (xjH+M )HM−=01 . Then, conveniently, each FFT operation of the overall algorithm occurs on some row of some matrix (the k-th row vector of a matrix U will be denoted by U (k)). The final DFT resides likewise in columnwise order.

1. [H in-place, length-W FFTs, each performed on a row of T ] for(0 ≤ M < H) T (M ) = DF T T (M ) ;

2.[Transpose and twist the matrix]

(TJK ) = (TKJ g−JK );

3.[W in-place, length-H FFTs, each performed on a row of the new T ]

Note that whatever scheme is used for the transpose (see Exercise 9.53) can also be used to convert lexicographically arranged input data x into the requisite columnwise format, and likewise for converting back at algorithm’s end to lexicographic DFT format. In other words, if the input data is assumed to be stored initially lexicographically, then two more transpositions can be placed, one before and one after the algorithm, to render Algorithm 9.5.7 a standard length-W H FFT. A small worked example is useful here. Algorithm 9.5.7 wants, for a length-N = 4 = 2 · 2 FFT, and so primitive fourth root of unity g = e2πi/4 = i, the input data in columnwise order like so:

The first algorithm step is to do (H = 2) row FFTs each of length (W = 2), to get

Then we transpose, and twist via dyadic multiply by the phase matrix

whence a final set of row FFTs gives

Incidentally, if one wonders how this di ers from a two-dimensional FFT such as an FFT in the field of image processing, the answer to that is simple: This four-step (or six-step, if preand post transposes are invoked to start with and end up with standard row-ordering) format involves that internal “twist,” or phase factor, in step [Transpose and twist the matrix]. A twodimensional FFT does not involve the phase-factor twisting step; instead, one simply takes FFTs of all rows in place, then all columns in place.

Of course, with respect to repeated applications of Algorithm 9.5.7 the e cient option is simply this: Always store signals and their transforms in

the columnwise format. Furthermore, one can establish a rule that for signal

√

lengths N = 2n, we factor into matrix dimensions as W = H = N = 2n/2 for n even, but W = 2H = 2(n+1)/2 for n odd. Then the matrix is square or almost square. Furthermore, for the inverse FFT, in which everything proceeds as above but with F F T −1 calls and the twisting phase uses g+JK , with a final division by N , one can conveniently assume that the width and height for this inverse case satisfy W = H or H = 2W , so that in such as convolution problems the output matrix of the forward FFT is what is expected for the inverse FFT, even when said matrix is nonsquare. Actually, for convolutions per se there are other interesting optimizations due to J. Papadopoulos, such as the use of DIF/DIT frameworks and bit-scrambled powers of g; and a very fast large-FFT implementation of Mayer, in which one never transposes, using instead a fast, memory-e cient columnwise FFT stage; see [Crandall et al. 1999].

One interesting byproduct of this approach is that one is moved to study the basic problem of matrix transposition. The treatment in [Bailey 1989] gives an interesting small example of the algorithm in [Fraser 1976] for e cient transposition of a stored matrix, while the paper [Van Loan 1992, p. 138] indicates how active, really, is the ongoing study of fast transpose. Such an algorithm has applications in other aspects of large-integer arithmetic, for example see Section 9.5.7.

We next turn to a development that has enjoyed accelerated importance since its discovery by pioneers A. Dutt and V. Rokhlin. A core result in their seminal paper [Dutt and Rokhlin 1993] involves a length-D, nonuniform FFT of the type

D−1

where all we know a priori about the (possibly nonuniform) frequencies ωj is that they all lie in [0, D). This form for Xk is to be compared with the

486 Chapter 9 FAST ALGORITHMS FOR LARGE-INTEGER ARITHMETIC

standard DFT (9.20) for root g = e−2πi/D; in the latter case we have the uniform scenario, ωj = j. The remarkable development—already a decade old but as we say emerging in importance—is that such nonuniform FFTs can be calculated to absolute precision in

operations. In a word: This nonuniform FFT method is “about as e cient” as a standard, uniform FFT. This e cient algorithm has found application in such disparate fields as N -body gravitational dynamics (after all, multibody gravity forces can be evaluated as a kind of convolution) and Riemann zetafunction computations (see, e.g., Exercise 1.62).

For the following algorithm display, we have departed from the literature in several ways. First, we force indicial sums like (9.23) to run for j, k [0, D − 1], for book consistency; indeed, many of the literature references involve equivalent sums for j or k [−D/2, D/2−1]. Secondly, we have chosen an algorithm that is fast and robust (in the sense of guaranteed accuracy even under radical behavior of the input signal), but not necessarily of minimum complexity. Robustness is, of course, important for rigorous calculations in computational number theory. Third, we have chosen this particular algorithm because it does not rely upon special-function evaluations such as Gaussians or windowing functions. The penalty for all of this streamlining is that we are required to perform a certain number of standard, length-D FFTs, this number of FFTs depending only logarithmically on desired accuracy

In what follows, the “error” means that the true DFT (9.23) Xk and the calculated one Xk from the algorithm below di er according to

We next present an algorithm that computes Xk in (9.23) to within error< 2−b, that is to b-bit precision, in

operations. The algorithm is based on the observation that

Such an inequality allows us to rigorously bound the error for this algorithm. (See Exercise 9.54.)

Algorithm 9.5.8 (Nonuniform FFT). Let x be a signal of length D ≡ 0 (mod 8), and assume real-valued (not necessarily integer) frequencies (ωj [0, D) : j [0, D−1]). For a given bit-precision b (so the relative error is < 2−b), this algorithm returns an approximation Xk to the true DFT (9.23).

1.[Initialize]

"#

Algorithm 9.5.8 is written above in rather compact form, but a typical implementation on a symbolic processor looks much like this pseudocode. Note that the signal union at the end—being the left-right concatenation of length-(D/8) signals—can be e ected in some programming languages about as compactly as we have. Incidentally, though the symbolics mask somewhat the reason why the algorithm works, it is not hard to see that the Taylor expansion of e−2πi(µj +θj )k/D in powers of θj , together with adroit manipulation of indices, brings success. It is a curious and happy fact that decimating the transform-signal length by the fixed factor of 8 su ces for all possible input signals x to the algorithm. Such is the utility of the inequality (9.25). In summary, the number of standard FFTs we require, to yield b-bit accuracy, is about 16b/ lg b. This is a “worst-case” FFT count, in that practical applications often enjoy far better than b-bit accuracy when a particular b parameter is passed to Algorithm 9.5.8.

Since the pioneering work of Dutt–Rokhlin, various works such as [Ware 1998], [Nguyen and Liu 1999] have appeared, revealing somewhat better accuracy, or slightly improved execution speed, or other enhancements. There has even been an approach that minimizes the worst-case error for input signals of unit norm [Fessler and Sutton 2003]. But all the way from the Dutt–Rokhlin origins to the modern fringe, the basic idea remains the same: Transform the calculation of X to that of obtaining a set of uniform and standard FFTs of only somewhat wasteful overall length.

488 Chapter 9 FAST ALGORITHMS FOR LARGE-INTEGER ARITHMETIC

We close this FFT section by mentioning some new developments in regard to “gigaelement” FFTs that have now become possible at reasonable speed. For example, [Crandall et al. 2004] discusses theory and implementation for each of these gigaelement cases:

Length-230, one-dimensional FFT (e ected via Algorithm 9.5.7); 215 × 215, two-dimensional FFT;

210 × 210 × 210, three-dimensional FFT.

With such massive signal sizes come the di cult yet fascinating issues of fast matrix transposition, cache-friendly memory action, and vectorization of floating-point arithmetic. The bottom line as regards performance is that the one-dimensional, length-230 case takes less than one minute on a modern hardware cluster, if double-precision floating-point is used. (The twoand three-dimensional cases are about as fast; in fact the two-dimensional case is usually fastest, for technical reasons.)

For computational number theory, these new results mean this: On a hardware cluster that fits into a closet, say, numbers of a billion decimal digits can be multiplied together in roughly a minute. Such observations depend on proper resolution of the following problem: The errors in such “monster” FFTs can be nontrivial. There are just so many terms being added/multiplied that one deviates from the truth, so to speak, in a kind of random walk. Interestingly, a length-D FFT can be modeled as a random walk in D dimensions, having O(ln D) steps. The paper [Crandall et al. 2004] thus reports quantitative bounds on FFT errors, such bounds having been pioneered by E. Mayer and C. Percival.

9.5.3Convolution theory

Let x denote a signal (x0, x1, . . . , xD−1), where for example, the elements of x could be the digits of Definitions (9.1.1) or (9.1.2) (although we do not a priori insist that the elements be digits; the theory to follow is fairly general). We start by defining fundamental convolution operations on signals. In what follows, we assume that signals x, y have been assigned the same length (D) of elements. In all the summations of this next definition, indices i, j each run over the set {0, . . . , D − 1}:

Definition 9.5.9. The cyclic convolution of two length-D signals x, y is a signal denoted z = x × y having D elements given by

while the negacyclic convolution of x, y is a signal v = x ×− y having D elements given by

and the acyclic convolution of x, y is a signal u = x ×A y having 2D elements given by

for n {0, . . . , 2D − 2}, together with the assignment u2D−1 = 0. Finally, the half-cyclic convolution of x, y is the length-D signal x ×H y consisting of the first D elements of the acyclic convolution u.

These fundamental convolutions are closely related, as is seen in the following result. In such statements we interpret the sum of two signals c = a + b in elementwise fashion; that is, cn = an + bn for relevant indices n. Likewise, a scalar-signal product qa, with q a number and a a signal, is the signal (qan). We shall require the notion of the splitting of signals (of even length) into halves, so we denote by L(a), H(a), respectively, the lower-indexed and higher-indexed halves of a. That is, from c = a b the natural, left-right, concatenation of two signals of equal length, we shall have L(c) = a and H(c) = b.

Theorem 9.5.10. Let signals x, y have the same length D. Then the various convolutions are related as follows (it is assumed that in the relevant domain to which signal elements belong, 2−1 exists):

1

x ×H y = 2 ((x × y) + (x ×− y)).

Furthermore,

1

x ×A y = (x ×H y) 2 ((x × y) − (x ×− y)).

Finally, if the length D is even and xj , yj = 0 for j ≥ D/2, then

L(x) ×A L(y) = x × y = x ×− y.

These interrelations allow us to use certain algorithms more universally. For example, a pair of algorithms for cyclic and negacyclic can be used to extract both the half-cyclic or the acyclic, and so on. In the final statement of the theorem, we have introduced the notion of “zero padding,” which in practice amounts to appending D zeros to signals already of length D, so that the signals’ acyclic convolution is identical to the cyclic (or the negacyclic) convolution of the two padded sequences.

The connection between convolution and the DFT of the previous section is evident in the following celebrated theorem, wherein we refer to the dyadic operator , under which a signal z = x y has elements zn = xnyn:

Theorem 9.5.11 (Convolution theorem). Let signals x, y have the same length D. Then the cyclic convolution of x, y satisfies

x × y = DF T −1(DF T (x) DF T (y)),

490 Chapter 9 FAST ALGORITHMS FOR LARGE-INTEGER ARITHMETIC

which is to say

D−1

(x × y)n = D1 XkYk gkn.

k=0

Armed with this mighty theorem we can e ect the cyclic convolution of two signals with three transforms (one of them being the inverse transform), or the cyclic convolution of a signal with itself with two transforms. As the known complexity of the DFT is O(D ln D) operations in the field, the dyadic product implicit in Theorem 9.5.11, being O(D), is asymptotically negligible.

A direct and elegant application of FFTs for large-integer arithmetic is to perform the DFTs of Theorem 9.5.11 in order to e ect multiplication via acyclic convolution. This essential idea—pioneered by Strassen in the 1960s and later optimized in [Sch¨onhage and Strassen 1971] (see Section 9.5.6)—runs as follows. If an integer x is represented as a length-D signal consisting of the (base-B) digits, and the same for y, then the integer product xy is an acyclic convolution of length 2D. Though Theorem 9.5.11 pertains to the cyclic, not the acyclic, we nevertheless have Theorem 9.5.10, which allows us to use zeropadded signals and then perform the cyclic. This idea leads, in the case of complex field transforms, to the following scheme, which is normally applied using floating-point arithmetic, with DFT’s done via fast Fourier transform (FFT) techniques:

Algorithm 9.5.12 (Basic FFT multiplication). Given two nonnegative integers x, y, each having at most D digits in some base B (Definition 9.1.1), this algorithm returns the base-B representation of the product xy. (The FFTs normally employed herein would be of the floating-point variety, so one must beware of roundo error.)

1. [Initialize]

Zero-pad both of x, y until each has length 2D, so that the cyclic convolution of the padded sequences will contain the acyclic convolution of the unpadded ones;

Y= DF T (y);

3.[Dyadic product]

Z = X Y ;

4.[Inverse transform] z = DF T −1(Z);

5.[Round digits]

6. [Adjust carry in base B] carry = 0;

for(0 ≤ n < 2D) { v = zn + carry;

zn = v mod B; carry = v/B ;

}

7. [Final digit adjustment]

Delete leading zeros, with possible carry > 0 as a high digit of z; return z;

This algorithm description is intended to be general, conveying only the principles of FFT multiplication, which are transforming, rounding, carry adjustment. There are a great many details left unsaid, not to mention a great number of enhancements, some of which we address later. But beyond these minutiae there is one very strong caveat: The accuracy of the floating-point arithmetic must always be held suspect. A key step in the general algorithm is the elementwise rounding of the z signal. If floating-point errors in the FFTs are too large, an element of the convolution z could get rounded to an incorrect value.

One immediate practical enhancement to Algorithm 9.5.12 is to employ the balanced representation of Definition 9.1.2. It turns out that floatingpoint errors are significantly reduced in this representation [Crandall and Fagin 1994], [Crandall et al. 1999]. This phenomenon of error reduction is not completely understood, but certainly has to do with the fact of generally smaller magnitude for the digits, plus, perhaps, some cancellation in the DFT components because the signal of digits has a statistical mean (in engineering terms, “DC component”) that is very small, due to the balancing.

Before we move on to algorithmic issues such as further enhancements to the FFT multiply and the problem of pure-integer convolution, we should mention that convolutions can appear in number-theoretical work quite outside the large-integer arithmetic paradigm. We give two examples to end this subsection; namely, convolutions applied to sieving and to regularity results on primes.

Consider the following theorem, which is reminiscent of (although obviously much less profound than) the celebrated Goldbach conjecture:

Theorem 9.5.13. Let N = 2 · 3 · 5 · · · pm be a product of consecutive primes. Then every su ciently large even n is a sum n = a+b with each of a, b coprime to N .

It is intriguing that this theorem can be proved, without too much trouble, via convolution theory. (We should point out that there are also proofs using CRT ideas, so we are merely using this theorem to exemplify applications of discrete convolution methods (see Exercise 9.40).) The basic idea is to consider a special signal y defined by yj = 1 if gcd(j, N ) = 1, else yj = 0, with the signal given some cuto length D. Now the acyclic convolution y ×A y will tell us precisely which n < D of the theorem have the a + b representations, and furthermore, the n-th element of the acyclic is precisely the number of such representations of n.