SIGMA: An Open-Source Interactive System for Mixed-Reality Task Assistance Research – Extended Abstract

Abstract

We introduce an open-source system called Sigma (short for “Situ-ated Interactive Guidance, Monitoring, and Assistance”) as a plat-form for conducting research on task-assistive agents in mixed-reality scenarios. The system leverages the sensing and rendering affordances of a head-mounted mixed reality device in conjunction with large language and vision models to guide users step by step through procedural tasks. By open-sourcing the system, we aim to lower the barrier to entry, accelerate research in this space, and chart a path towards community-driven end-to-end evaluation of large language, vision, and multimodal models in the context of real-world interactive applications.