Subjective Evaluation of Visual Quality and Simulator Sickness of Short 360$^\circ$ Videos: ITU-T Rec. P.919

TLDR

Immersive technologies such as AR, VR, and 360° video have advanced rapidly, yet quality assessment methods have lagged behind. The study aims to assess and validate subjective quality evaluation methods for 360° video through cross‑lab tests conducted by the VQEG’s Immersive Media Group. The authors evaluated audiovisual quality, simulator sickness, and exploration behavior using short 10–30 s 360° sequences, examining factors such as assessment method, duration, HMD type, coding degradations, and sickness metrics, and released the annotated dataset for public use. Results confirm that ACR and DCR are valid for 360° video, 10‑second clips (with or without audio) suffice for coding artifact assessment, any compliant commercial HMD can be used, and faster sickness‑assessment methods outperform the SSQ, informing ITU‑T Recommendation P.919.

Abstract

Recently an impressive development in immersive technologies, such as Augmented Reality (AR), Virtual Reality (VR) and 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> video, has been witnessed. However, methods for quality assessment have not been keeping up. This paper studies quality assessment of 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> video from the cross-lab tests (involving ten laboratories and more than 300 participants) carried out by the Immersive Media Group (IMG) of the Video Quality Experts Group (VQEG). These tests were addressed to assess and validate subjective evaluation methodologies for 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> video. Audiovisual quality, simulator sickness symptoms, and exploration behavior were evaluated with short (from 10 seconds to 30 seconds) 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> sequences. The following factors&#x2019; influences were also analyzed: assessment methodology, sequence duration, Head-Mounted Display (HMD) device, uniform and non-uniform coding degradations, and simulator sickness assessment methods. The obtained results have demonstrated the validity of Absolute Category Rating (ACR) and Degradation Category Rating (DCR) for subjective tests with 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> videos, the possibility of using 10-second videos (with or without audio) when addressing quality evaluation of coding artifacts, as well as any commercial HMD (satisfying minimum requirements). Also, more efficient methods than the long Simulator Sickness Questionnaire (SSQ) have been proposed to evaluate related symptoms with 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> videos. These results have been instrumental for the development of the ITU-T Recommendation P.919. Finally, the annotated dataset from the tests is made publicly available for the research community.