Comparing human-made and AI-generated teaching videos: An experimental study on learning effects

Abstract

In the age of generative AI, can teaching videos be efficiently and effectively generated by large language models? In this study, the authors used generative AI tools to develop four short teaching videos for a management course and then compared them with human-generated videos on the same subjects in an online experiment. In an across-subject experimental design, 447 participants completed two treatment conditions presenting different mixes of AI-generated and human-made videos. The participants were asked to rate their learning experiences after each video and had their learning outcomes tested in a multiple-choice exam at the end of the session (N = 1788 video treatments). The findings show that human-generated videos provided a statistically significant but small advantage to participants in terms of learning experience, indicating that the participants still prefer to be taught by human teachers. However, a comparison of exam results between the experimental groups implies that the participants eventually acquired knowledge about the content to a similar degree. Given these findings and the ease with which AI-generated teaching videos can be created, this study concludes that AI-generated teaching videos will likely proliferate. • This study compares the learning effects of AI-generated versus human-made teaching videos. • In an online experiment, 447 participants watched four teaching videos, filled out a survey, and took an exam. • Participants prefer human-made teaching videos in terms of learning experience. • When watching AI-generated videos, participants achieved equally high learning outcomes. • Teaching videos can quickly be made using generative AI tools and can be expected to proliferate.