Support Vector Machines for Classification and Regression

TLDR

Support Vector Machines, introduced by Vapnik, are popular for their attractive features and empirical performance, extend from classification to regression, and use Structural Risk Minimisation to bound VC dimension and improve generalisation. The study aims to leverage SVMs’ Structural Risk Minimisation to achieve superior generalisation in statistical learning. Our results demonstrate that SVMs’ SRM principle outperforms the ERM principle used by conventional neural networks.

Abstract

The foundations of Support Vector Machines (SVM) have been developed by Vapnik and are gaining popularity due to many attractive features, and promising empirical performance. The formulation embodies the Structural Risk Minimisation (SRM) principle, which in our work has been shown to be superior to traditional Empirical Risk Minimisation (ERM) principle employed by conventional neural networks. SRM minimises an upper bound on the VC dimension (generalisation error), as opposed to ERM which minimises the error on the training data. It is this difference which equips SVMs with a greater ability to generalise, which is our goal in statistical learning. SVMs were developed to solve the classification problem, but recently they have been extended to the domain of regression problems.