Educational portfolios in the assessment of general practice trainers: reliability of assessors

Abstract

OBJECTIVES: This paper reports a project that assessed a series of portfolios assembled by a cohort of participants attending a course for prospective general practice trainers. DESIGN: The reliability of judgements about individual 'components', together with an overall global judgement about performance were studied. SETTING: NHSE South & West, King Alfred's College, Winchester and Institute of Community Studies, Bournemouth University. SUBJECTS: Eight experienced general practice trainers recruited from around Wessex, which incorporates Hampshire, Dorset, Wiltshire and the Isle of Wight. RESULTS: The reliability of individual assessor's judgements (i.e. their consistency) was moderate, but inter-rater reliability did not reach a level which could support making a safe summative judgement. The levels of reliability reached were similar to other subjective assessments and perhaps reflected individuality of personal agendas of both the assessed and the assessors, and variations in portfolio structure and content. CONCLUSIONS: Suggestions for approaches in future are made.