Assessing Agreement: An Examination of the Interrater Reliability of Portfolio Assessment in Rochester, New York

Abstract

This article reports on the interrater reliability of language arts portfolio assessment in the primary grades of the Rochester, New York, school system. This assessment was based on a sample of approximately 400 primary-grade language arts portfolios that were rated by two raters: the student's classroom teacher and an external reviewer. Reliabilities for student portfolios ranged from .58 for Grade 1 reading portfolios to .79 for kindergarten reading portfolios. Many of the reading portfolios could not be rated by the external reviewer, revealing problems both with the portfolio policies of the district and with the classroom-level implementation. The findings showed that primary-grade classroom teachers can reliably rate their own students' work, but tensions exist if a single portfolio system is intended both to provide classroom teachers with information about their students and to supply accountability information to administrators.