Realworld Evaluation: Working under Budget, Time, Data, and Political Constraints

Abstract

REALWORLD EVALUATION: WORKING UNDER BUDGET, TIME, DATA, AND POLITICAL CONSTRAINTS Michael Bamberger, Jim Rugh and Linda Mabry Thousand Oaks CA: Sage Publications 2006, PB xxxii + 468 pp, AUD 44.95, ISBN 1412909465The last two years have seen a flowering of fine books on evaluation. These have icluded the 7th edition of the world's top seller in the field (Rossi, Lipsey and Freeman's 2004 Evaluation: A Systematic Approach); the 3rd edition of Australian scholar John Owen's 2006 Program Evaluation: Forms and Approaches; Davidson's 2005 Evaluation Methodology Basics: The Nuts and Bolts of Sound Evaluation (which may well become a core reference for students); and Mathieson's wonderful new 2005 Encyclopedia of Evaluation. These, and other new publications I could mention, support the contention that evaluation has become not only a profession, but also a discipline.Bamberger, Rugh and Mabry's 2006 RealWorld Evaluation: Working Under Budget, Time, Data, and Political Constraints is the latest offering from Sage Publication's evaluation booklist. Bamberger is a private consultant and a former Senior Sociologist at the World Bank; Rugh is with CARE International; and Mabry a Washington State University academic. They have much experience in conducting evaluations and teaching evaluation concepts and practice in developing countries and elsewhere, and this solid experience permeates the book.The title conveys the way in which this book differs from so many others apparently covering similar ground: the numerous how-to evaluation resource books. The authors start with the real world of program evaluators, the world in which there is often insufficient money, time, data and political support to undertake sound evaluations. What, then, is the alternative to the types of evaluation research we would undertake - were there sufficient money, time, data and political support? The authors point out that, for far too many evaluators, the alternative is a 'quick and dirty' study; meaning one which lacks methodological rigour and ignores threats to the validity of the study's findings.'RealWorld evaluation' is defined as 'An approach developed for evaluations operating under budget, time, data and political constraints intended to maximise the rigor of methods and the validity of findings' (p. 439). RealWorld evaluation, as portrayed in this volume, has seven steps:(1) planning and scoping the evaluation;(2-5) directly addressing the budget, time and data constraints, along with the political influences;(6) strengthening the evaluation design and the validity of the conclusions; and(7) helping clients to use the evaluation's processes and products.These are the features, the authors argue, which make this book different from the many others covering program and policy evaluation.An example may assist. The chapter covering 'Critical information is missing or difficult to collect: addressing data constraints', deals seriatim with the following strategies: reconstructing baseline data, identifying and constructing comparison groups, problems of non-equivalent comparison groups, and collecting data on sensitive topics and from difficult-to-reach groups. In each area, specific research strategies to overcome the constraints are described and discussed. …