Citation Analysis May Severely Underestimate the Impact of Clinical Research as Compared to Basic Research

TLDR

Citation analysis is widely used to assess research performance, yet citation practices differ markedly across medical research areas. The authors aim to determine whether citation‑based bibliometric indicators can validly compare research units in different medical areas and introduce a visualization method to expose such differences. Their method extracts terms from titles and abstracts of a large publication set, visualizes the structure of 32 medical fields, and highlights how research areas differ in average citation impact, focusing on Cardiac & cardiovascular systems, Clinical neurology, and Surgery. The findings reveal pronounced within‑field heterogeneity: clinical intervention research shows lower citation impact than basic and diagnostic research, and both popular and more sophisticated bibliometric indicators fail to correct for these differences, leading to an underestimation of clinical research impact.

Abstract

Citation analysis has become an important tool for research performance assessment in the medical sciences. However, different areas of medical research may have considerably different citation practices, even within the same medical field. Because of this, it is unclear to what extent citation-based bibliometric indicators allow for valid comparisons between research units active in different areas of medical research.A visualization methodology is introduced that reveals differences in citation practices between medical research areas. The methodology extracts terms from the titles and abstracts of a large collection of publications and uses these terms to visualize the structure of a medical field and to indicate how research areas within this field differ from each other in their average citation impact.Visualizations are provided for 32 medical fields, defined based on journal subject categories in the Web of Science database. The analysis focuses on three fields: Cardiac & cardiovascular systems, Clinical neurology, and Surgery. In each of these fields, there turn out to be large differences in citation practices between research areas. Low-impact research areas tend to focus on clinical intervention research, while high-impact research areas are often more oriented on basic and diagnostic research.Popular bibliometric indicators, such as the h-index and the impact factor, do not correct for differences in citation practices between medical fields. These indicators therefore cannot be used to make accurate between-field comparisons. More sophisticated bibliometric indicators do correct for field differences but still fail to take into account within-field heterogeneity in citation practices. As a consequence, the citation impact of clinical intervention research may be substantially underestimated in comparison with basic and diagnostic research.