Rigor in Data Synthesis: A Case Study of Reliability in Meta-analysis

Abstract

The subject of this study was the reliability among persons coding information from primary research sources for use in a metaanalysis. Meta-analysis is a quantitative procedure for synthesizing data in primary research reports (Glass, 1976; 1977). As such, it is one of a number of quantitative procedures proposed as alternatives to the traditional, verbal review of literature. This study rests on the assumption that meta-analysis in particular, and quantitative data synthesis procedures in general, should be examined for sources of invalidity as rigorously as is the process of generating primary research. Pillemer and Light (1980) defined data synthesis as, using formal procedures for combining the results from several experiments (p. 177). Until recently, the traditional review of literature has been the primary formal mechanism for data synthesis. However, in the context of burgeoning numbers of primary research sources, alternate quantitative procedures have been proposed (see Pillemer & Light, 1980, for a review). Among the different methods that have been proposed for data synthesis, meta-analysis has been applied most widely. With regard to primary research, numerous texts call attention to sources of invalidity which investigators must try to avoid (Campbell & Stanley, 1966). However, only recently has it been suggested that quantitative data synthesis also is subject to threats to its validity, and that data synthesis studies should be examined for sources of invalidity as carefully as primary research (Glass, McGaw, & Smith, 1981; Cooper, Note 1). Although few empirical efforts have focused directly on assessing the validity of meta-analysis, critics have expressed several objections. First, they have called attention to the lack of a statistical