E-Track: Eye Tracking with Event Camera for Extended Reality (XR) Applications

Abstract

Eye tracking is an essential functionality to enable extended reality (XR) applications. However, the latency and power constraints of an XR headset are tight. Unlike fix-rate frame-based RGB cameras, the event camera senses brightness changes and generates asynchronous sparse events with high temporal resolution. Although the event camera exhibits suitable characteristics for eye tracking in XR systems, processing an event-based data stream is a challenging task. In this paper, we present an event-based eye-tracking system that extracts pupil features. It is the first system that operates only with an event camera and requires no additional sensing hardware. We first propose an event-to-frame conversion method that encodes the events triggered by eye motion into a 3-channel frame. Secondly, we train a Convolutional Neural Network (CNN) on 24 subjects to classify the events representing the pupil. Finally, we employ a region of interest (RoI) mechanism that tracks pupil location and reduces the amount of CNN inference by 96%. Our eye-tracking pipeline is able to locate the pupil with an error of 3.68 pixels at 160 mW system power.