You-Do, I-Learn: Discovering Task Relevant Objects and their Modes of Interaction from Multi-User Egocentric Video

Abstract

We present a fully unsupervised approach for i) discovery of task-relevant objects and ii) how these objects have been used. Given egocentric video from multiple operators, the approach can discover objects with which the users interact, both static objects such as a coffee machine as well as movable ones such as a cup. Importantly, the common modes of interaction for discovered objects are also found. We investigate using appearance, position, motion and attention, and present results using each and a combination of relevant features. Results show that the method is capable of discovering 95% of task-relevant objects on a variety of daily tasks such as initialising a printer, preparing a coffee or setting up a gym machine. In addition, the approach enables the automatic generation of guidance video on how these objects have been used before.