Strategies for Design Science Research Evaluation

Abstract

Seminal works in the application of design science research (DSR) in IS emphasize the importance of evaluation. However, discussion of evaluation activities and methods is limited and typically assumes an ex post perspective, in which evaluation occurs after the construction of an IS artifact. Such perspectives can assume that the evaluation is an empirical process and its methods can be selected in the same way as empirical research methods. In this paper, we analyze a broader range of evaluation strategies, which includes ex ante (prior to artifact construction) evaluation. This broader view is developed as a strategic DSR evaluation framework, which expands evaluation choices for IS DSR researchers, and also adds emphasis to strategies for evaluating design processes in addition to design products, using well-known quality criteria as an important asset. The framework encompasses both ex ante and ex post orientations as well as naturalistic settings (e.g., case studies) and artificial settings (e.g., lab experiments) for DSR evaluation. The framework proposed offers a strategic view of DSR evaluation that is useful in analyzing published studies, and also in surfacing the evaluation opportunities that present themselves to IS DSR researchers.