The Utility Metric: A Novel Method to Assess the Overall Performance of Discrete Brain–Computer Interfaces

Abstract

A relevant issue in a brain-computer interface (BCI) is the capability to efficiently convert user intentions into correct actions, and how to properly measure this efficiency. Usually, the evaluation of a BCI system is approached through the quantification of the classifier performance, which is often measured by means of the information transfer rate (ITR). A shortcoming of this approach is that the control interface design is neglected, and hence a poor description of the overall performance is obtained for real systems. To overcome this limitation, we propose a novel metric based on the computation of BCI Utility. The new metric can accurately predict the overall performance of a BCI system, as it takes into account both the classifier and the control interface characteristics. It is therefore suitable for design purposes, where we have to select the best options among different components and different parameters setup. In the paper, we compute Utility in two scenarios, a P300 speller and a P300 speller with an error correction system (ECS), for different values of accuracy of the classifier and recall of the ECS. Monte Carlo simulations confirm that Utility predicts the performance of a BCI better than ITR.