Exploring the reliability of social and environmental disclosures content analysis

TLDR

The study explores inter‑coder reliability of social and environmental disclosure content analysis in annual reports and discusses implications and challenges of formal reliability measurement. Using Hackston and Milne’s sentence‑based coding instruments and decision rules, three coders analyzed 49 annual reports over five rounds to assess co‑agreement levels. Results show that inexperienced coders can reliably produce aggregate disclosure totals with minimal training, but detailed sub‑category coding requires at least 20 reports of training before outputs are trustworthy.

Abstract

This paper reports the results of an exploratory study of inter‐coder reliability of annual report social and environmental disclosures content analysis. Using the sentence‐based coding instruments and decision rules from Hackston and Milne (1996), this study reports the co‐agreement levels reached by three coders over five rounds of testing 49 annual reports. The study also provides a commentary on the implications of formal reliability analysis for past and future social and environmental disclosures content analyses, and the complexities of formal reliability measurement. The overall findings suggest that the coded output from inexperienced coders using the Hackston and Milne approach with little or no prior training can be relied on for aggregate total disclosures analysis. For more detailed sub‐category analysis, however, the findings suggest a period of training for the less experienced coders with at least 20 reports appears necessary before their coded output could be relied on.