Reviewing studies with diverse designs: the development and evaluation of a new tool

TLDR

Tools for assessing research quality are often design‑specific, making it difficult to evaluate a body of research that addresses the same question using diverse approaches. This paper aims to develop and preliminarily evaluate a quality assessment tool that can be applied to a methodologically diverse set of research articles. The 16‑item QATSDD was tested for reliability and validity by health services researchers in psychology, sociology, and nursing, and qualitative feedback on its comprehension, content, perceived value, and usability was collected from mixed‑methods researchers. The tool’s components were confirmed to represent “good research technique,” face validity was established by nine researchers, inter‑rater reliability was substantial (κ = 71.5 %) and test‑retest reliability was good to substantial, demonstrating that QATSDD is reliable and valid for assessing diverse studies and can help reviewers standardize quality assessments.

Abstract

Abstract Rationale, aims & objective Tools for the assessment of the quality of research studies tend to be specific to a particular research design (e.g. randomized controlled trials, or qualitative interviews). This makes it difficult to assess the quality of a body of research that addresses the same or a similar research question but using different approaches. The aim of this paper is to describe the development and preliminary evaluation of a quality assessment tool that can be applied to a methodologically diverse set of research articles. Methods The 16‐item quality assessment tool (QATSDD) was assessed to determine its reliability and validity when used by health services researchers in the disciplines of psychology, sociology and nursing. Qualitative feedback was also gathered from mixed‐methods health researchers regarding the comprehension, content, perceived value and usability of the tool. Results Reference to existing widely used quality assessment tools and experts in systematic review confirmed that the components of the tool represented the construct of ‘good research technique’ being assessed. Face validity was subsequently established through feedback from a sample of nine health researchers. Inter‐rater reliability was established through substantial agreement between three reviewers when applying the tool to a set of three research papers (κ = 71.5%), and good to substantial agreement between their scores at time 1 and after a 6‐week interval at time 2 confirmed test–retest reliability. Conclusions The QATSDD shows good reliability and validity for use in the quality assessment of a diversity of studies, and may be an extremely useful tool for reviewers to standardize and increase the rigour of their assessments in reviews of the published papers which include qualitative and quantitative work.