GMRES Methods for Least Squares Problems

Abstract

The standard iterative method for solving large sparse least squares problems $\min\|\mbox{\boldmathb}-A\mbox{\boldmathx}\|_2$, $A\in\mathbf{R}^{m\times n}$, is the CGLS method, or its stabilized version, LSQR, which is mathematically equivalent to applying the conjugate gradient method to the normal equation $A^{\mbox{\tiny T}}A\mbox{\boldmathx}=A^{\mbox{\tiny T}}\mbox{\boldmathb}$. We consider alternative methods using a matrix $B\in\mathbf{R}^{n\times m}$ and applying the generalized minimal residual (GMRES) method to $\min\|\mbox{\boldmathb}-AB\mbox{\boldmathz}\|_2$ or $\min\|B\mbox{\boldmathb}-BA\mbox{\boldmathx}\|_2$. We give a sufficient condition concerning B for the GMRES methods to give a least squares solution without breakdown for arbitrary $\mbox{\boldmathb}$, for overdetermined, underdetermined, and possibly rank-deficient problems. We then give a convergence analysis of the GMRES methods as well as the CGLS method. Then, we propose using the robust incomplete factorization (RIF) for B. Finally, we show by numerical experiments on overdetermined and underdetermined problems that, for ill-conditioned problems, the GMRES methods with RIF give least squares solutions faster than the CGLS and LSQR methods with RIF, and are similar in performance to the reorthogonalized CGLS with RIF.