E-Book Overview
As one of the classical statistical regression techniques, and often the first to be taught to new students, least squares fitting can be a very effective tool in data analysis. Given measured data, we establish a relationship between independent and dependent variables so that we can use the data predictively. The main concern of Least Squares Data Fitting with Applications is how to do this on a computer with efficient and robust computational methods for linear and nonlinear relationships. The presentation also establishes a link between the statistical setting and the computational issues.
In a number of applications, the accuracy and efficiency of the least squares fit is central, and Per Christian Hansen, Víctor Pereyra, and Godela Scherer survey modern computational methods and illustrate them in fields ranging from engineering and environmental sciences to geophysics. Anyone working with problems of linear and nonlinear least squares fitting will find this book invaluable as a hands-on guide, with accessible text and carefully explained problems.
Included are
• an overview of computational methods together with their properties and advantages
• topics from statistical regression analysis that help readers to understand and evaluate the computed solutions
• many examples that illustrate the techniques and algorithms
Least Squares Data Fitting with Applications can be used as a textbook for advanced undergraduate or graduate courses and professionals in the sciences and in engineering.
E-Book Content
Least Squares Data Fitting with Applications
Per Christian Hansen Víctor Pereyra Godela Scherer
Least Squares Data Fitting with Applications Per Christian Hansen Department of Informatics and Mathematical Modeling Technical University of Denmark
Víctor Pereyra Energy Resources Engineering Stanford University
Godela Scherer Department of Mathematics and Statistics University of Reading
Contents Foreword
ix
Preface
xi
Symbols and Acronyms
xv
1 The 1.1 1.2 1.3 1.4 1.5
Linear Data Fitting Problem Parameter estimation, data approximation Formulation of the data fitting problem Maximum likelihood estimation The residuals and their properties Robust regression
1 1 4 9 13 19
2 The 2.1 2.2 2.3
Linear Least Squares Problem Linear least squares problem formulation The QR factorization and its role Permuted QR factorization
25 25 33 39
3 Analysis of Least Squares Problems 3.1 The pseudoinverse 3.2 The singular value decomposition 3.3 Generalized singular value decomposition 3.4 Condition number and column scaling 3.5 Perturbation analysis
47 47 50 54 55 58
4 Direct Methods for Full-Rank Problems 4.1 Normal equations 4.2 LU factorization 4.3 QR factorization 4.4 Modifying least squares problems 4.5 Iterative refinement 4.6 Stability and condition number estimation
65 65 68 70 80 85 88
v
vi
CONTENTS 4.7
Comparison of the methods
89
5 Direct Methods for Rank-Deficient Problems 5.1 Numerical rank 5.2 Peters-Wilkinson LU factorization 5.3 QR factorization with column permutations 5.4 UTV and VSV decompositions 5.5 Bidiagonalization 5.6 SVD computations
91 92 93 94 98 99 101
6 Methods for Large-Scale Problems 6.1 Iterative versus direct methods 6.2 Classical stationary methods 6.3 Non-stationary methods, Krylov methods 6.4 Practicalities: preconditioning and stopping criteria 6.5 Block methods
105 105 107 108
7 Additional Topics in Least Squares 7.1 Constrained linear least squares problems 7.2 Missing data problems 7.3 Total least squares (TLS) 7.4 Convex optimization 7.5 Compressed sensing
121 121 131 136 143 144
8 Nonlinea