Methodological Quality as a Component of Meta-Analysis

Abstract

Abstract An issue ,associated with meta-analytic reviews is whether or not the effect sizes of the studies included should be weighted in proportion to the "quality" of the research. In this preliminary investigation, it was found that neither advanced graduate students taking a research methodology course nor professional researchers with expertise in the substantive domain under consideration exhibited much agreement in their rank orderings of studies' methodological quality. Suggestions for improving the reliability of methodological-quality assessments are included, along with weighting schemes that could be applied should inter-judge agreement be achieved.