On the differences between theoretical and applied system dynamics modeling

Abstract

Many research methodology handbooks distinguish between theoretical (often called “fundamental”) and applied research (Hedrick et al., 1993; Bryman and Bell, 2011; Easterby-Smith et al., 2012; Saunders et al., 2012; Babbie, 2013; Sekaran and Bougie, 2016). The main difference is that the purpose of theoretical research is to increase understanding (ideally, by developing new theory or scrutinizing existing theory), while applied research is “deliberately intended to bring about social change” (Babbie, 2013, p. 18) and to solve concrete real-world problems. Although most research lies somewhere in the middle of the continuum between theoretical and applied research, the distinction is important because it has implications for research design and for evaluating the research. When the aim of research is essentially to increase understanding, it requires different decisions in the design of the research projects compared to when the aim is to propose polices. Further, these two types of research can only be adequately assessed when their outcomes are evaluated in the light of what the research intends to achieve. System dynamics (SD) is a versatile method and is used in very different settings, from highly applied (e.g., Cooper, 1980; McCarthy et al., 2014; Ghaffarzadegan et al., 2017) to highly theoretical (e.g., Sastry, 1997; Sterman and Wittenberg, 1999; Gambardella et al., 2017). More generally, there are three broad categories of SD research: methodological, theoretical, and applied modeling. Although most research projects have elements from two or even three of these categories, we can assume this classification without loss of generality. For the purpose of this article, we focus on two of these categories: theoretical modeling and applied modeling, and the continuum they span—thus acknowledging that many theoretical models have applied implications and some applied models have theoretical generalizations. Although many related aspects have been discussed earlier (see, among others, de Gooyert, 2018; Größler, 2008; Lane and Schwaninger, 2008; Rahmandad, 2015; Repenning, 2003; Schwaninger and Grösser, 2008; Sterman, 1992), a systematic comparison of applied and theoretical SD modeling has, to the best of our knowledge, not been conducted before. Such a comparison is relevant because, in analogy to research in general, it has implications for designing SD research and assessing the quality of SD research. We argue that, although most SD research is somewhere in between applied and theoretical research, the distinction is important to be able to adequately assess the contributions that applied and theoretical SD research make, since they each require a different set of quality criteria. Therefore, below we provide a first attempt of making such a comparison. In the rest of this article, we first elaborate on the general distinction between theoretical and applied research. Then, we discuss the implications for SD research design, followed by a discussion of the implications for assessing SD research, and some concluding remarks. General research methodology handbooks acknowledge the difference between applied and fundamental/theoretical research (Hedrick et al., 1993; Bryman and Bell, 2011; Easterby-Smith et al., 2012; Saunders et al., 2012; Babbie, 2013; Sekaran and Bougie, 2016), although the terminology varies. These handbooks all use the term “applied research,” but fundamental research is also referred to as “basic” or “pure” research (Saunders et al., 2012, p. 12).1 Table 1 summarizes some typical differences between fundamental and applied research as presented by Saunders et al. (2012, p. 12), who base their summary on their own experience, on Easterby-Smith et al. (2012), and on Hedrick et al. (1993). Some handbooks build on the distinction between mode 1 and mode 2 knowledge production as put forward by Gibbons et al. (1994), arguing that mode 1 resembles fundamental research whereas mode 2 is more applied (Bryman and Bell, 2011; Saunders et al., 2012). Mode 1 knowledge production here refers to what is called the “traditional,” disciplinary way of conducting science, driven by an academic agenda, and written for an academic audience. Mode 2 knowledge production then refers to research carried out by multidisciplinary teams that take a real-world problem as their starting point (Gibbons et al., 1994). The core competence of the scientist is explanatory research, researching and explaining what is—the actual. However, for professionals such as doctors, lawyers and engineers, it is field problem solving (FPS). Professionals are interested in changing “the actual” into “the preferred”. By “field problem” we mean a situation in reality, which in the view of some influential stakeholders can or should be improved, such as a sick person, a polluted water well or an unreliable logistical system. (Van Aken et al., 2012, p. 4). Blumberg et al. conclude that: “Both applied and pure research are, then, problem-solving-based. Applied research is, however, directed much more to making immediate managerial decisions” (Blumberg et al., 2011, p. 11). However, as Sekaran and Bougie clarify, fundamental research often precedes applied research: “the objective of engaging in basic research is primarily to equip oneself with additional knowledge of certain phenomena and problems that occur in several organizations and industries with a view to finding solutions, the knowledge generated from such research is often applied later for solving organizational problems” (Sekaran and Bougie, 2016, p. 7). The distinctions above are about research in general, and not about SD modeling specifically. Below we discuss whether it is useful to claim a similar difference between applied and theoretical SD modeling. Much SD research aims at both increasing understanding and supporting change through discussing policy implications. However, there are several arguments that lead to making a distinction between applied and theoretical SD research, with the purpose of discussing the implications of these differences regarding research design and evaluation. First, the SD community consists of both academics and practitioners. Research that academics carry out varies: some are more inclined towards fundamental research whereas others focus on applied research. However, practitioners rarely write for a purely academic audience. Therefore, having both practitioners and academics in the SD community results in both applied and theoretical research. Second, the difference is reflected in the prizes that the System Dynamics Society awards. There is a specific System Dynamics Application Award (System Dynamics Society, 2018), and the name of the award already suggests that theoretical research is not eligible, while applied research is. The Jay W. Forrester Award is given to “the best contribution to the field …” (System Dynamics Society, 2018), and its description does not distinguish between contributions from fundamental research, from applied modeling, or from methodology research. In practice, however, the majority of the Forrester awards have been awarded to articles published in mainstream and high-quality academic journals. Those journals usually require a strong focus on theoretical contributions, which make it less likely for mainly applied research to be eligible. Third, SD studies are published in very different journals. Some of these journals, like the mainstream journals mentioned above, focus strongly on a theoretical contribution, so mostly fundamental research stands a chance of getting published (e.g., Sastry, 1997; Sterman and Wittenberg, 1999). Other journals have a stronger focus on practical implications, and applied research is published in such journals (e.g., Jalali and Kaiser, 2018). System Dynamics Review publishes both types of articles (recent examples of applied research published in System Dynamics Review are Ghaffarzadegan et al., 2017, and Kapmeier and Gonçalves, 2018). All in all, we conclude that there are enough reasons to justify a distinction between theoretical and applied SD modeling. We argue that the distinction is important because it has implications for research design and for evaluation of the studies. For theoretical SD research, the outcomes are new, adapted, or refuted theories of dynamic phenomena. When the focus is on practical problem solving, emphasis is put on the implementation of improved policies derived from the modeling project. Models that focus on providing a theoretical contribution are usually developed to explain phenomena rather than to achieve change. The best they can provide (from a practical perspective) is the identification of factors, linkages, and policy formulations that might be interesting to investigate if there is later a need for change in a real system. Thus applied projects normally follow all major steps shared by SD textbooks (problem articulation, formulation of dynamic hypothesis, formulation of simulation model, testing, policy design, and evaluation; Sterman, 2000, p. 85); however, not all these steps are always required for theoretical contributions—in particular, the policy design step. Expanding knowledge is an incremental process and theoretical contributions can, for example, focus on adjusting and testing existing theories (e.g., Sastry, 1997; Sterman and Wittenberg, 1999; Black et al., 2004) or on formulating a model to develop propositions that can be tested in future research (e.g., Azoulay et al., 2010; Rahmandad and Repenning, 2016). Thus theoretical SD research sometimes comprises only two or three of the steps that are required for practical contributions. Some theoretical SD research projects have the objective to provide an explanation for an observed instance of a phenomenon and for such studies collecting data is an important element (e.g., Rudolph et al., 2009; Walrave et al., 2011; Rahmandad and Repenning, 2016), but this does not necessarily need to be the case. Since data might not—or only to a limited degree—be available, there are many examples of theoretical SD studies that do not use data (e.g., Sastry, 1997; Sterman and Wittenberg, 1999; Gary, 2005; Rahmandad, 2012). Other fundamental studies do not collect data themselves but build on causal relationships that were the result of earlier empirical studies (e.g., Rudolph and Repenning, 2002; Black et al., 2004). Applied SD is about specific instances of problems that require collecting empirical data on that specific instance, while theoretical SD can be about a phenomenon in its general form. When the goal is to generalize, data on a specific case can even be misleading, because “the particular curves of past history are only a special case” (Forrester, 2007, p. 364). The focus of theoretical research is on generalizability; the focus of applied research is on solving a specific instance of a problem, and therefore collecting primary data has a more prominent role in applied SD research, not least because simulation output and empirical data are often directly compared. Theoretical SD research consists of modeling a parsimonious representation of a problem. This has important implications for the decision of what elements are included in the model. When modeling a practical problem, variables should be added if they contribute to solving a problem and to achieving change, even if each variable and its relationship with the rest of the system are already perfectly understood. When modeling a theoretical problem, however, a variable that is already well understood (including its implications for the behavior of the rest of the model) can be left out if it does not advance existing theory (and no interaction effects with the variables included are hypothesized). Sometimes omitting such variables is actually required because retaining such variables in the model makes it more complex, making it harder to present a clear, convincing model that will constitute a significant contribution to the body of knowledge (as follows from Occam’s razor, the principle of parsimony; see also Repenning, 2003). A theoretical problem can be about just one or a few main aspects of a phenomenon; thus, focusing on theoretical contributions can lead to highly stylized or “conceptual” models (see, for example, Gary, 2005; Rudolph and Repenning, 2002; Sastry, 1997), without the negative connotation of being “impressionistic” (Homer, 1996). Most system dynamicists agree that all models are wrong, but that some are useful (Box, 1976; Sterman, 2002). The usefulness of a theoretical SD study, however, is different from the usefulness of an applied SD study. For theoretical SD research, the purpose is making a knowledge contribution. A main question is whether the model accurately fits the scientific debate it is contributing to. This implies that the model is positioned in the language of the ongoing scientific debate and established theories, regardless of whether this language is known to decision makers in the field (Repenning, 2003; Rahmandad, 2015). Evaluating both applied and theoretical SD research comes with its own challenges. Evaluating theoretical studies requires extensive knowledge of the scientific literature that the study is aiming to contribute to. Evaluating applied studies requires extensive background knowledge on the details of the problem. Sharing such knowledge can be hindered by confidentiality concerns. Besides, applied SD models can become very large because of the desire to do justice to the many facets of a real-world problem. These differences are also reflected in the usage of some standard tests for SD models (Senge and Forrester, 1980; Barlas, 1996). For instance, as noted by Barlas (1992), policy tests do not apply to theoretical SD modeling, while for applied SD research this is where a key focus is. Another example is the family member test (Sterman, 2000, p. 860), which is not needed in theoretical but critical in applied modeling. When theoretical SD models are developed in accordance with the characteristics outlined above, they naturally pass this test since they are built as generalized reflections of a phenomenon. However, for applied SD models the family member test becomes highly relevant and can be used for discussing a model’s validity. The main point of this note is that making a distinction between applied and theoretical SD modeling can be useful because they come with their own research design decisions and their own sets of quality criteria. At one extreme, applied SD answers the question of a problem-owner who wants to change a situation. This often requires following all the steps of textbook SD, including collecting empirical data on a specific case, and expanding the boundaries of the system until all variables that are hypothesized to impact the system’s behavior are included. Theoretical SD modeling, at the other extreme, can be much more focused on a small incremental step in a larger research agenda. Trying to combine too many aspects in a fundamental modeling study can prevent a clear and convincing theoretical contribution to an ongoing scientific debate. Therefore, theoretical SD studies do not necessarily follow all the steps of textbooks, may not always collect empirical data, and can have restrictive system boundaries intentionally leaving out certain variables. The general rules still apply: it all depends on the purpose of the project and the usefulness of SD to achieve that purpose. Applied and theoretical studies have different purposes, either solving a client’s problem or contributing to understanding a phenomenon. We hope this note helps to clarify the distinction by starting to make explicit how these two types of studies are different and how each type of study and related publications should be evaluated. We stress that our comparison of applied and theoretical SD research intentionally emphasizes the differences between these two categories of modeling, while many studies lie on a continuum between purely applied and purely theoretical. In fact, many studies consist of elements of both categories. But even then, these different elements require differences in research design and evaluation, so our suggestions should still be useful for doing SD research and publishing it. This note provides a comparison of applied and theoretical SD modeling. We stress that we only offered some initial examples about possible differences. We call for more systematic research of published applied and theoretical SD studies to shed more light on their differences and resulting implications, in terms of research design and the criteria used to evaluate the research.