Where is the evidence?

TLDR

Debates in medicine and psychology highlight problems with research evidence quality, such as the file drawer effect and incentive structures that favor quantity over quality, raising questions about the evidence base for learning analytics. The paper introduces the LACE Evidence Hub to map learning analytics research to four propositions—supporting learning, teaching, widespread deployment, and ethical use—and examines evidence in the university sector. The authors use the LACE Evidence Hub to collate and analyze studies, focusing on evidence for teaching and learner support, and identify weaknesses in the available research. Only a small amount of evidence (7 % negative, 123 studies) supports the propositions, indicating learning analytics is not immune to the same pressures, and the authors recommend substantial improvements to the evidence base.

Abstract

Where is the evidence for learning analytics? In particular, where is the evidence that it improves learning in practice? Can we rely on it? Currently, there are vigorous debates about the quality of research evidence in medicine and psychology, with particular issues around statistical good practice, the 'file drawer effect', and ways in which incentives for stakeholders in the research process reward the quantity of research produced rather than the quality. In this paper, we present the Learning Analytics Community Exchange (LACE) project's Evidence Hub, an effort to relate research evidence in learning analytics to four propositions about learning analytics: whether they support learning, support teaching, are deployed widely, and are used ethically. Surprisingly little evidence in this strong, specific sense was found, and very little was negative (7%, N=123), suggesting that learning analytics is not immune from the pressures in other areas. We explore the evidence in one particular area in detail (whether learning analytics improve teaching and learners support in the university sector), and set out some of the weaknesses of the evidence available. We conclude that there is considerable scope for improving the evidence base for learning analytics, and set out some suggestions of ways for various stakeholders to achieve this.