Correct and still wrong: The relationship between sampling strategies and the estimation of the generalization error

Ronny Hänsch, Andreas Ley, Olaf Hellwich

2017 · 37 citations · 3 references

Concepts

Abstract

The automatic generation of semantic maps from remotely sensed imagery by supervised classifiers has seen much effort in the last decades. The major focus has been on the improvement of the interplay between feature operators and classifiers, while experimental design and test data generation has been mostly neglected. This paper shows that sampling strategies applied to partition the available reference data into train and test sets have a large influence on the quality and reliability of the estimated generalization error. It illustrates and discusses problems of common choices for sampling schemes, i.e. the violation of the independence assumption and the illusion of the availability of global knowledge in the training data. Furthermore, a novel sampling strategy is proposed which circumvents these problems and achieves a less biased estimate of the classification error.

References

3