RetAssist: Facilitating Vocabulary Learners with Generative Images in Story Retelling Practices

Abstract

Reading and repeatedly retelling a short story is a common and efective approach to learning the meanings and usages of target words. However, learners often struggle with comprehending, recalling, and retelling the story contexts of these target words. Inspired by the Cognitive Theory of Multimedia Learning, we propose a computational workfow to generate relevant images paired with stories. Based on the workfow, we work with learners and teachers to iteratively design an interactive vocabulary learning system named RetAssist. It can generate sentence-level images of a story to facilitate the understanding and recall of the target words in the story retelling practices. Our within-subjects study (N=24) shows that compared to a baseline system without generative images, RetAssist signifcantly improves learners’ fuency in expressing with target words. Participants also feel that RetAssist eases their learning workload and is more useful. We discuss insights into leveraging text-to-image generative models to support learning tasks. © 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.