An <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow><mml:msub><mml:mi>l</mml:mi> <mml:mi>∞</mml:mi></mml:msub> </mml:mrow> </mml:math> Eigenvector Perturbation Bound and Its Application to Robust Covariance Estimation.

Abstract

In statistics and machine learning, we are interested in the eigenvectors (or singular vectors) of certain matrices (e.g. covariance matrices, data matrices, etc). However, those matrices are usually perturbed by noises or statistical errors, either from random sampling or structural patterns. The Davis-Kahan sin <i>θ</i> theorem is often used to bound the difference between the eigenvectors of a matrix A and those of a perturbed matrix <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow><mml:mover><mml:mi>A</mml:mi> <mml:mo>˜</mml:mo></mml:mover> <mml:mo>=</mml:mo> <mml:mi>A</mml:mi> <mml:mo>+</mml:mo> <mml:mi>E</mml:mi></mml:mrow> </mml:math> , in terms of <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow><mml:msub><mml:mi>l</mml:mi> <mml:mn>2</mml:mn></mml:msub> </mml:mrow> </mml:math> norm. In this paper, we prove that when <i>A</i> is a low-rank and incoherent matrix, the <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow><mml:msub><mml:mi>l</mml:mi> <mml:mi>∞</mml:mi></mml:msub> </mml:mrow> </mml:math> norm perturbation bound of singular vectors (or eigenvectors in the symmetric case) is smaller by a factor of <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:msqrt> <mml:mrow><mml:msub><mml:mi>d</mml:mi> <mml:mn>1</mml:mn></mml:msub> </mml:mrow> </mml:msqrt> </mml:mrow> </mml:math> or <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:msqrt> <mml:mrow><mml:msub><mml:mi>d</mml:mi> <mml:mn>2</mml:mn></mml:msub> </mml:mrow> </mml:msqrt> </mml:mrow> </mml:math> for left and right vectors, where <i>d</i> ₁ and <i>d</i> ₂ are the matrix dimensions. The power of this new perturbation result is shown in robust covariance estimation, particularly when random variables have heavy tails. There, we propose new robust covariance estimators and establish their asymptotic properties using the newly developed perturbation bound. Our theoretical results are verified through extensive numerical experiments.