- •1 Introduction
- •1.1 What makes eigenvalues interesting?
- •1.2 Example 1: The vibrating string
- •1.2.1 Problem setting
- •1.2.2 The method of separation of variables
- •1.3.3 Global functions
- •1.3.4 A numerical comparison
- •1.4 Example 2: The heat equation
- •1.5 Example 3: The wave equation
- •1.6 The 2D Laplace eigenvalue problem
- •1.6.3 A numerical example
- •1.7 Cavity resonances in particle accelerators
- •1.8 Spectral clustering
- •1.8.1 The graph Laplacian
- •1.8.2 Spectral clustering
- •1.8.3 Normalized graph Laplacians
- •1.9 Other sources of eigenvalue problems
- •Bibliography
- •2 Basics
- •2.1 Notation
- •2.2 Statement of the problem
- •2.3 Similarity transformations
- •2.4 Schur decomposition
- •2.5 The real Schur decomposition
- •2.6 Normal matrices
- •2.7 Hermitian matrices
- •2.8 Cholesky factorization
- •2.9 The singular value decomposition (SVD)
- •2.10 Projections
- •2.11 Angles between vectors and subspaces
- •Bibliography
- •3 The QR Algorithm
- •3.1 The basic QR algorithm
- •3.1.1 Numerical experiments
- •3.2 The Hessenberg QR algorithm
- •3.2.1 A numerical experiment
- •3.2.2 Complexity
- •3.3 The Householder reduction to Hessenberg form
- •3.3.2 Reduction to Hessenberg form
- •3.4 Improving the convergence of the QR algorithm
- •3.4.1 A numerical example
- •3.4.2 QR algorithm with shifts
- •3.4.3 A numerical example
- •3.5 The double shift QR algorithm
- •3.5.1 A numerical example
- •3.5.2 The complexity
- •3.6 The symmetric tridiagonal QR algorithm
- •3.6.1 Reduction to tridiagonal form
- •3.6.2 The tridiagonal QR algorithm
- •3.7 Research
- •3.8 Summary
- •Bibliography
- •4.1 The divide and conquer idea
- •4.2 Partitioning the tridiagonal matrix
- •4.3 Solving the small systems
- •4.4 Deflation
- •4.4.1 Numerical examples
- •4.6 Solving the secular equation
- •4.7 A first algorithm
- •4.7.1 A numerical example
- •4.8 The algorithm of Gu and Eisenstat
- •4.8.1 A numerical example [continued]
- •Bibliography
- •5 LAPACK and the BLAS
- •5.1 LAPACK
- •5.2 BLAS
- •5.2.1 Typical performance numbers for the BLAS
- •5.3 Blocking
- •5.4 LAPACK solvers for the symmetric eigenproblems
- •5.6 An example of a LAPACK routines
- •Bibliography
- •6 Vector iteration (power method)
- •6.1 Simple vector iteration
- •6.2 Convergence analysis
- •6.3 A numerical example
- •6.4 The symmetric case
- •6.5 Inverse vector iteration
- •6.6 The generalized eigenvalue problem
- •6.7 Computing higher eigenvalues
- •6.8 Rayleigh quotient iteration
- •6.8.1 A numerical example
- •Bibliography
- •7 Simultaneous vector or subspace iterations
- •7.1 Basic subspace iteration
- •7.2 Convergence of basic subspace iteration
- •7.3 Accelerating subspace iteration
- •7.4 Relation between subspace iteration and QR algorithm
- •7.5 Addendum
- •Bibliography
- •8 Krylov subspaces
- •8.1 Introduction
- •8.3 Polynomial representation of Krylov subspaces
- •8.4 Error bounds of Saad
- •Bibliography
- •9 Arnoldi and Lanczos algorithms
- •9.2 Arnoldi algorithm with explicit restarts
- •9.3 The Lanczos basis
- •9.4 The Lanczos process as an iterative method
- •9.5 An error analysis of the unmodified Lanczos algorithm
- •9.6 Partial reorthogonalization
- •9.7 Block Lanczos
- •9.8 External selective reorthogonalization
- •Bibliography
- •10 Restarting Arnoldi and Lanczos algorithms
- •10.2 Implicit restart
- •10.3 Convergence criterion
- •10.4 The generalized eigenvalue problem
- •10.5 A numerical example
- •10.6 Another numerical example
- •10.7 The Lanczos algorithm with thick restarts
- •10.8 Krylov–Schur algorithm
- •10.9 The rational Krylov space method
- •Bibliography
- •11 The Jacobi-Davidson Method
- •11.1 The Davidson algorithm
- •11.2 The Jacobi orthogonal component correction
- •11.2.1 Restarts
- •11.2.2 The computation of several eigenvalues
- •11.2.3 Spectral shifts
- •11.3 The generalized Hermitian eigenvalue problem
- •11.4 A numerical example
- •11.6 Harmonic Ritz values and vectors
- •11.7 Refined Ritz vectors
- •11.8 The generalized Schur decomposition
- •11.9.1 Restart
- •11.9.3 Algorithm
- •Bibliography
- •12 Rayleigh quotient and trace minimization
- •12.1 Introduction
- •12.2 The method of steepest descent
- •12.3 The conjugate gradient algorithm
- •12.4 Locally optimal PCG (LOPCG)
- •12.5 The block Rayleigh quotient minimization algorithm (BRQMIN)
- •12.7 A numerical example
- •12.8 Trace minimization
- •Bibliography
Lecture Notes on
Solving Large Scale Eigenvalue Problems
Prof. Dr. Peter Arbenz
D-INFK
ETH Z¨urich
Email: ARBENZ@INF.ETHZ.CH
with modifications by
Prof. Dr. Daniel Kressner
D-MATH ETH Z¨urich
Email: KRESSNER@MATH.ETHZ.CH
Spring semester 2012
ii
Contents
1 Introduction |
1 |
1.1What makes eigenvalues interesting? . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Example 1: The vibrating string . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.1Problem setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.2The method of separation of variables . . . . . . . . . . . . . . . . . 5
1.3Numerical methods for solving 1-dimensional problems . . . . . . . . . . . . 6
1.3.1Finite di erences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3.2The finite element method . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3.3 Global functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3.4A numerical comparison . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4 Example 2: The heat equation . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.5Example 3: The wave equation . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.6 The 2D Laplace eigenvalue problem . . . . . . . . . . . . . . . . . . . . . . 13
1.6.1The finite di erence method . . . . . . . . . . . . . . . . . . . . . . . 13
1.6.2 The finite element method (FEM) . . . . . . . . . . . . . . . . . . . 16
1.6.3A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.7 Cavity resonances in particle accelerators . . . . . . . . . . . . . . . . . . . 21
1.8Spectral clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.8.1The graph Laplacian . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
1.8.2Spectral clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
1.8.3Normalized graph Laplacians . . . . . . . . . . . . . . . . . . . . . . 27
1.9Other sources of eigenvalue problems . . . . . . . . . . . . . . . . . . . . . . 27
Bibliography |
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 |
2 Basics |
29 |
2.1Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.2Statement of the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.3Similarity transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.4Schur decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.5The real Schur decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.6 Normal matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.7Hermitian matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.8Cholesky factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.9The singular value decomposition (SVD) . . . . . . . . . . . . . . . . . . . . 42
2.10 |
Projections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
43 |
2.11 |
Angles between vectors and subspaces . . . . . . . . . . . . . . . . . . . . . |
45 |
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
49 |
|
iii
iv |
CONTENTS |
3 The QR Algorithm |
51 |
3.1The basic QR algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.1.1Numerical experiments . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.2 The Hessenberg QR algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.2.1A numerical experiment . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.2.2Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.3The Householder reduction to Hessenberg form . . . . . . . . . . . . . . . . 59
3.3.1Householder reflectors . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.3.2 Reduction to Hessenberg form . . . . . . . . . . . . . . . . . . . . . 59
3.4Improving the convergence of the QR algorithm . . . . . . . . . . . . . . . . 61
3.4.1A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.4.2 QR algorithm with shifts . . . . . . . . . . . . . . . . . . . . . . . . 63
3.4.3A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.5The double shift QR algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.5.1A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.5.2The complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.6 The symmetric tridiagonal QR algorithm . . . . . . . . . . . . . . . . . . . 72
3.6.1Reduction to tridiagonal form . . . . . . . . . . . . . . . . . . . . . . 72
3.6.2The tridiagonal QR algorithm . . . . . . . . . . . . . . . . . . . . . . 73
3.7Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
75 |
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
76 |
4 Cuppen’s Divide and Conquer Algorithm |
77 |
4.1The divide and conquer idea . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.2Partitioning the tridiagonal matrix . . . . . . . . . . . . . . . . . . . . . . . 78
4.3Solving the small systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4.4 |
Deflation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
79 |
|
4.4.1 Numerical examples . . . . . . . . . . . . . . . . . . . . . . . . . . . |
80 |
4.5 |
The eigenvalue problem for D + ρvvT . . . . . . . . . . . . . . . . . . . . . |
81 |
4.6Solving the secular equation . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4.7A first algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.7.1A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.8The algorithm of Gu and Eisenstat . . . . . . . . . . . . . . . . . . . . . . . 89
4.8.1A numerical example [continued] . . . . . . . . . . . . . . . . . . . . 90
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
93 |
|
5 LAPACK and the BLAS |
95 |
|
5.1 |
LAPACK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
95 |
5.2 |
BLAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . |
96 |
5.2.1 Typical performance numbers for the BLAS . . . . . . . . . . . . . . 97
5.3Blocking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.4 LAPACK solvers for the symmetric eigenproblems . . . . . . . . . . . . . . 100
5.5Generalized Symmetric Definite Eigenproblems (GSEP) . . . . . . . . . . . 102
5.6An example of a LAPACK routines . . . . . . . . . . . . . . . . . . . . . . . 102
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
CONTENTS |
v |
6 Vector iteration (power method) |
111 |
6.1Simple vector iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6.2Convergence analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
6.3A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
6.4 The symmetric case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
6.5Inverse vector iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
6.6The generalized eigenvalue problem . . . . . . . . . . . . . . . . . . . . . . . 125
6.7Computing higher eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . . 125
6.8 Rayleigh quotient iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
6.8.1A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
Bibliography . . . . . . . . . . . . . . . . . . . . . |
. . . . . . . . . . . . . . . . . 129 |
7 Simultaneous vector or subspace iterations |
131 |
7.1Basic subspace iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
7.2 Convergence of basic subspace iteration . . . . . . . . . . . . . . . . . . . . 131
7.3Accelerating subspace iteration . . . . . . . . . . . . . . . . . . . . . . . . . 136
7.4Relation between subspace iteration and QR algorithm . . . . . . . . . . . . 141
7.5 Addendum . |
. . . |
. |
. . . |
. |
. . . |
. . . |
. |
. . . |
. |
. . . |
. . . . |
. |
. . . |
. . . . |
. 144 |
Bibliography . . . |
. . . |
. |
. . . |
. |
. . . |
. . . |
. |
. . . |
. |
. . . |
. . . . |
. |
. . . |
. . . . |
. 144 |
8 Krylov subspaces |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
145 |
8.1Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
8.2 |
Definition and basic properties . . . . . . . . |
. . . . |
. . . . |
. . . . |
. . |
. . |
. |
146 |
8.3 |
Polynomial representation of Krylov subspaces |
. . . |
. . . . |
. . . . |
. . |
. . |
. |
147 |
8.4Error bounds of Saad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Bibliography . . . . . . . . . . . . . . |
. . . . . . . . . . . . . . . . . . . . . . . . 153 |
9 Arnoldi and Lanczos algorithms |
155 |
9.1An orthonormal basis for the Krylov space Kj (x) . . . . . . . . . . . . . . . 155
9.2 |
Arnoldi algorithm with explicit restarts |
. . . |
. . . . |
. . . . |
. . . . |
. . . . |
. |
157 |
9.3 |
The Lanczos basis . . . . . . . . . . . |
. . . . |
. . . . |
. . . . |
. . . . |
. . . . |
. |
158 |
9.4The Lanczos process as an iterative method . . . . . . . . . . . . . . . . . . 160
9.5An error analysis of the unmodified Lanczos algorithm . . . . . . . . . . . . 167
9.6 Partial reorthogonalization . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
9.7Block Lanczos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
9.8 |
External selective reorthogonalization . . . . |
. . . . . . . . . . . . . . . . . 175 |
Bibliography . . . . . . . . . . . . . . . . . . . . . |
. . . . . . . . . . . . . . . . . 176 |
|
10 Restarting Arnoldi and Lanczos algorithms |
177 |
|
10.1 |
The m-step Arnoldi iteration . . . . . . . . . |
. . . . . . . . . . . . . . . . . 177 |
10.2 |
Implicit restart . . . . . . . . . . . . . . . . . |
. . . . . . . . . . . . . . . . . 178 |
10.3 |
Convergence criterion . . . . . . . . . . . . . |
. . . . . . . . . . . . . . . . . 180 |
10.4The generalized eigenvalue problem . . . . . . . . . . . . . . . . . . . . . . . 181
10.5A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
10.6Another numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
10.7The Lanczos algorithm with thick restarts . . . . . . . . . . . . . . . . . . . 192
10.8Krylov–Schur algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
10.9 The rational Krylov space method |
. . |
. |
. . . |
. . . . |
. . . . |
. . . . |
. . . . |
. |
196 |
Bibliography . . . . . . . . . . . . . . |
. . . |
. |
. . . |
. . . . |
. . . . |
. . . . |
. . . . |
. |
197 |
vi |
|
|
CONTENTS |
11 The Jacobi-Davidson Method |
199 |
||
11.1 |
The Davidson algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 |
||
11.2 |
The Jacobi orthogonal component correction |
. . . . . . . . . . . . . . . . . 200 |
|
|
11.2.1 |
Restarts . . . . . . . . . . . . . . . . . |
. . . . . . . . . . . . . . . . . 203 |
|
11.2.2 |
The computation of several eigenvalues |
. . . . . . . . . . . . . . . . 203 |
|
11.2.3 |
Spectral shifts . . . . . . . . . . . . . |
. . . . . . . . . . . . . . . . . 204 |
11.3 |
The generalized Hermitian eigenvalue problem |
. . . . . . . . . . . . . . . . 206 |
|
11.4A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
11.5The Jacobi–Davidson algorithm for interior eigenvalues . . . . . . . . . . . . 210
11.6Harmonic Ritz values and vectors . . . . . . . . . . . . . . . . . . . . . . . . 211
11.7 Refined Ritz vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
11.8The generalized Schur decomposition . . . . . . . . . . . . . . . . . . . . . . 215
11.9JDQZ: Computing a partial QZ decomposition . . . . . . . . . . . . . . . . 215
11.9.1 |
Restart . . . . . . . . . . . . . . . . . . . . |
. . . . . . . . . . . . . . 217 |
11.9.2 |
Deflation . . . . . . . . . . . . . . . . . . . |
. . . . . . . . . . . . . . 217 |
11.9.3 |
Algorithm . . . . . . . . . . . . . . . . . . . |
. . . . . . . . . . . . . . 218 |
11.10Jacobi-Davidson for nonlinear eigenvalue problems |
. . . . . . . . . . . . . . 218 |
|
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . |
. . . . . . . . . . . . . . 221 |
|
12 Rayleigh quotient and trace minimization |
223 |
|
12.1Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
12.2The method of steepest descent . . . . . . . . . . . . . . . . . . . . . . . . . 224
12.3The conjugate gradient algorithm . . . . . . . . . . . . . . . . . . . . . . . . 225
12.4Locally optimal PCG (LOPCG) . . . . . . . . . . . . . . . . . . . . . . . . . 229
12.5The block Rayleigh quotient minimization algorithm (BRQMIN) . . . . . . 232
12.6The locally-optimal block preconditioned conjugate gradient method (LOBPCG)232
12.7A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
12.8Trace minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
Chapter 1
Introduction
Before we start with the subject of this notes we want to show how one actually arrives at large eigenvalue problems in practice. In the following, we restrict ourselves to problems from physics [4, 10, 7] and computer science.
1.1What makes eigenvalues interesting?
In physics, eigenvalues are usually connected to vibrations. Objects like violin strings, drums, bridges, sky scrapers can swing. They do this at certain frequencies. And in some situations they swing so much that they are destroyed. On November 7, 1940, the Tacoma narrows bridge collapsed, less than half a year after its opening. Strong winds excited the bridge so much that the platform in reinforced concrete fell into pieces. A few years ago the London millennium footbridge started wobbling in a way that it had to be closed. The wobbling had been excited by the pedestrians passing the bridge. These are prominent examples of vibrating structures.
But eigenvalues appear in many other places. Electric fields in cyclotrones, a special form of particle accelerators, have to oscillate in a precise manner, in order to accelerate the charged particles that circle around its center. The solutions of the Schr¨odinger equation from quantum physics and quantum chemistry have solutions that correspond to vibrations of the, say, molecule it models. The eigenvalues correspond to energy levels that molecule can occupy.
Many characteristic quantities in science are eigenvalues:
•decay factors,
•frequencies,
•norms of operators (or matrices),
•singular values,
•condition numbers.
In the sequel we give a number of examples that show why computing eigenvalues is important. At the same time we introduce some notation.
1