Measuring research quality using the journal impact factor, citations and ‘Ranked Journals’: blunt instruments or inspired metrics?

Abstract

This paper examines whether three bibliometric indicators—the journal impact factor, citations per paper and the Excellence in Research for Australia (ERA) initiative's list of ‘ranked journals’—can predict the quality of individual research articles as assessed by international experts, both overall and within broad disciplinary groupings. The analysis is based on data obtained from a Mock Research Quality Framework (RQF) exercise conducted by Monash University during 2006–07 in which external assessors rated research articles for their quality using a five-point scale. Although a significant relationship exists between all three bibliometric variables and the overall Mock RQF assessor quality scores, only a relatively small amount of the variance (generally <20 per cent) could be explained. There is some evidence that the relationship is stronger within some disciplinary groupings than others. The findings suggest caution should be exercised when using these indicators as proxies for research quality.