Situating evaluation in scenarios of use

Abstract

We report on the use of scenario-based methods for evaluating collaborative systems. We describe the method, the case study where it was applied, and provide results of its efficacy in the field. The results suggest that scenario-based evaluation is effective in helping to focus evaluation efforts and in identifying the range of technical, human, organizational and other contextual factors that impact system success. The method also helps identify specific actions, for example, prescriptions for design to enhance system effectiveness. However, we found the method somewhat less useful for identifying the measurable benefits gained from a CSCW implementation, which was one of our primary goals. We discuss challenges faced applying the technique, suggest recommendations for future research, and point to implications for practice.