Which Type of Citation Analysis Generates the Most Accurate Taxonomy of Scientific and Technical Knowledge?

TLDR

Price foresaw that a citation‑based taxonomy of science and technology would be useful for policy, yet determining its accuracy remains challenging and prior work is sparse. The study compares the accuracies of topic‑level taxonomies derived from direct citation, bibliographic coupling, and co‑citation clustering. The authors evaluated these taxonomies using a gold standard of highly cited articles (≥100 references) to assess reference concentration. Direct citation clustering yields more accurate topic‑level taxonomies than bibliographic coupling or co‑citation, while discipline‑level journal‑based taxonomies are highly inaccurate and should be avoided.

Abstract

In 1965, Price foresaw the day when a citation‐based taxonomy of science and technology would be delineated and correspondingly used for science policy. A taxonomy needs to be comprehensive and accurate if it is to be useful for policy making, especially now that policy makers are utilizing citation‐based indicators to evaluate people, institutions and laboratories. Determinining the accuracy of a taxonomy, however, remains a challenge. Previous work on the accuracy of partition solutions is sparse, and the results of those studies, although useful, have not been definitive. In this study we compare the accuracies of topic‐level taxonomies based on the clustering of documents using direct citation, bibliographic coupling, and co‐citation. Using a set of new gold standards—articles with at least 100 references—we find that direct citation is better at concentrating references than either bibliographic coupling or co‐citation. Using the assumption that higher concentrations of references denote more accurate clusters, direct citation thus provides a more accurate representation of the taxonomy of scientific and technical knowledge than either bibliographic coupling or co‐citation. We also find that discipline‐level taxonomies based on journal schema are highly inaccurate compared to topic‐level taxonomies, and recommend against their use.