Publication ratings versus peer ratings of universities

Abstract

Abstract This paper presents a quantitative comparison of peer versus bibliometric procedures for rating the quality of U.S. universities. The peer ratings used are the Roose‐Andersen rating of the quality of graduate faculty in 10 scientific fields. The bibliometric ratings used are (1) the number of university papers in each field; (2) the average “quality” of the papers based on their citation rates, expressed as influence per paper in each field; and (3) total influence, the product of number of papers and influence per paper. The bibliometric ratings are based on 127,000 university papers, from 450 journals in 10 fields for 1965 to 1973. Roose‐Andersen ranks and scores are found to correlate most highly with the total influence of the university's papers, followed closely by correlations with the total number of papers, and much less closely with the average influence per paper. A partial correlation and regression analysis indicate that the Roose‐Andersen scores have two additive components: bibliometric size and bibliometric quality. Further analysis explores the extent to which a university's prestige in one department correlates with the assessment of other departments within that university, and the extent to which the university's overall bibliometric size correlates with the assessment of departments within that university. It is shown that the ratings of departments within a university are not independent, and that these dependencies are associated with the bibliometric size of the university. University ranks and scores in different fields are shown to be much more highly correlated when based on peer assessment than when based on bibliometric measures.