Are Convolutional Neural Networks or Transformers more like human\n vision?

Abstract

Modern machine learning models for computer vision exceed humans in accuracy\non specific visual recognition tasks, notably on datasets like ImageNet.\nHowever, high accuracy can be achieved in many ways. The particular decision\nfunction found by a machine learning system is determined not only by the data\nto which the system is exposed, but also the inductive biases of the model,\nwhich are typically harder to characterize. In this work, we follow a recent\ntrend of in-depth behavioral analyses of neural network models that go beyond\naccuracy as an evaluation metric by looking at patterns of errors. Our focus is\non comparing a suite of standard Convolutional Neural Networks (CNNs) and a\nrecently-proposed attention-based network, the Vision Transformer (ViT), which\nrelaxes the translation-invariance constraint of CNNs and therefore represents\na model with a weaker set of inductive biases. Attention-based networks have\npreviously been shown to achieve higher accuracy than CNNs on vision tasks, and\nwe demonstrate, using new metrics for examining error consistency with more\ngranularity, that their errors are also more consistent with those of humans.\nThese results have implications both for building more human-like vision\nmodels, as well as for understanding visual object recognition in humans.\n