Human-in-the-Loop Machine Learning to Increase Video Accessibility for Visually Impaired and Blind Users

TLDR

Video accessibility is essential for blind and visually impaired people, yet professional descriptions are costly and time‑consuming, and volunteer‑created descriptions vary in quality and can intimidate novices. The study introduces a Human‑in‑the‑Loop Machine Learning system that automates text generation and scene segmentation, enabling volunteers to edit outputs and collaborate with machines to produce high‑quality descriptions with low entry barriers. The system automates video text generation and scene segmentation, then presents the results for volunteer describers to edit. First‑time describers found the HILML system significantly faster and easier to use, and blind and visually impaired users rated its description quality and topic understanding significantly higher than the human‑only condition.

Abstract

Video accessibility is crucial for blind and visually impaired individuals for education, employment, and entertainment purposes. However, professional video descriptions are costly and time-consuming. Volunteer-created video descriptions could be a promising alternative, however, they can vary in quality and can be intimidating for novice describers. We developed a Human-in-the-Loop Machine Learning (HILML) approach to video description by automating video text generation and scene segmentation and allowing humans to edit the output. The HILML approach facilitates human-machine collaboration to produce high quality video descriptions while keeping a low barrier to entry for volunteer describers. Our HILML system was significantly faster and easier to use for first-time video describers compared to a human-only control condition with no machine learning assistance. The quality of the video descriptions and understanding of the topic created by the HILML system compared to the human-only condition were rated as being significantly higher by blind and visually impaired users.