2024 · 12 citations · 34 references
ArchitectureDesignAugmented Reality DesignInteraction DesignUser Experience DesignAugmented RealityContext-aware Augmented RealityImmersive MediaInteraction Design For Augmented RealityMultimodal InteractionAugmented Reality AuthoringContextLarge Language ModelsUser Context InferenceUser Study Evaluation
Sound plays a crucial role in enhancing user experience and immersiveness in Augmented Reality (AR). However, current platforms lack support for AR sound authoring due to limited interaction types, challenges in collecting and specifying context information, and difficulty in acquiring matching sound assets. We present SonifyAR, an LLM-based AR sound authoring system that generates context-aware sound effects for AR experiences. SonifyAR expands the current design space of AR sound and implements a Programming by Demonstration (PbD) pipeline to automatically collect contextual information of AR events, including virtual-content-semantics and real-world context. This context information is then processed by a large language model to acquire sound effects with Recommendation, Retrieval, Generation, and Transfer methods. To evaluate the usability and performance of our system, we conducted a user study with eight participants and created five example applications, including an AR-based science experiment, and an assistive application for low-vision AR users.
34
Example-guided physically based modal sound synthesis
Zhimin Ren, Hengchin Yeh, Ming C. Lin · ACM Transactions on Graphics · 2013 · 455 citations
Natural Sciences, Physics, Acoustics +11
Visual to Sound: Generating Natural Sound for Videos in the Wild
Yipin Zhou, Zhaowen Wang, Fang Chen et al. · 2018 · 211 citations
Computing, Vision Science, Acoustics +12