Publication | Open Access
Orthogonal Random Features
104
Citations
0
References
2016
Year
EngineeringMachine LearningRandom Fourier FeaturesBiometricsGaussian Kernel ApproximationImage AnalysisData ScienceData MiningPattern RecognitionRandom MappingStochastic GeometryStatisticsIntriguing DiscoveryKnowledge DiscoveryProbability TheoryComputer ScienceDimensionality ReductionSparse RepresentationHigh-dimensional MethodReproducing Kernel MethodOrthogonal Random FeaturesRandom MatrixKernel Method
We present an intriguing discovery related to Random Fourier Features: in Gaussian kernel approximation, replacing the random Gaussian matrix by a properly scaled random orthogonal matrix significantly decreases kernel approximation error. We call this technique Orthogonal Random Features (ORF), and provide theoretical and empirical justification for this behavior. Motivated by this discovery, we further propose Structured Orthogonal Random Features (SORF), which uses a class of structured discrete orthogonal matrices to speed up the computation. The method reduces the time cost from $\mathcal{O}(d^2)$ to $\mathcal{O}(d \log d)$, where $d$ is the data dimensionality, with almost no compromise in kernel approximation quality compared to ORF. Experiments on several datasets verify the effectiveness of ORF and SORF over the existing methods. We also provide discussions on using the same type of discrete orthogonal structure for a broader range of applications.