Reporting guidelines for simulation‐based research in social sciences

Abstract

Reproducibility is central to the progress of science, and simulation-based research is no exception. Only when research results are independently reproducible can different scholars verify the results reported by others, build on each other's work, and convince the public of the reliability of their results (Laine et al., 2007). Given the widespread use of computational methods in different branches of science, many scientists have called for more transparency in documenting computational research to allow reproducibility (Schwab et al., 2000; Code, 2010; Peng, 2011). Simulation-based research in the social sciences has been on the rise over the last few decades (Gilbert and Troitzsch, 2005), yet a set of reporting guidelines that ensure reproducibility and more efficient and effective communication among researchers is lacking. As a result, many research reports lack the information required to reproduce the simulation models they discuss or the specific simulation experiments they report. In this paper we provide an initial set of reporting guidelines for simulation-based research (RGSR) in the social sciences, with a focus on common scenarios in system dynamics research. We discuss these guidelines separately for reporting models, reporting simulation experiments, and reporting optimization results. The guidelines are further divided into minimum and preferred requirements, distinguishing between factors that are indispensable for reproduction of research and those that enhance transparency. We also provide guidelines to improve visualization of research to reduce the costs of reproduction. Finally we offer suggestions to enhance the adoption of these guidelines. To illustrate the challenge of documentation and reproducibility, we reviewed all the articles published in System Dynamics Review in the years 2010 and 2011. Of 34 research articles, 27 reported results from a simulation model. Of these 27, the majority (16; 59%) did not include model equations, two (7%) contained partial equations, and the rest reported the complete model, either in the text (3; 11%), in an online appendix (5; 19%), or by referencing another publication (1; 4%). Similarly, only eight articles (30%) included the parameter values needed to replicate the base case. Only six (22%) included complete units for all the equations, with three offering partial coverage of units. Finally, the information needed to replicate a reported graph (e.g. scenario and parameter settings) was missing in eight studies and could not be verified without attempting full reproduction in another five. These findings are consistent with recent research that has examined model quality and documentation in System Dynamics Review and International System Dynamics Conference proceedings (Groesser and Tschupp, 2012). Despite a long tradition emphasizing model transparency, attention to modeling process, and reproducibility of results (Forrester, 1961; Sterman, 2000) the system dynamics literature is falling short of the goals for full reproducibility to which it aspires. Similar challenges to reproducibility are reported in a variety of disciplines and journals (Dewald et al., 1986; Hubbard and Vetter, 1996; Ioannidis, 2005; McCullough et al., 2006, 2008; Koenker and Zeileis, 2009). In response, some fields developed guidelines for the reporting of models and simulation results, such as minimum information required in the annotation of biochemical models (MIRIAM) (Le Novere et al., 2005), minimum information about a simulation experiment (MIASE) in systems biology (Waltemath et al., 2011), IIE computational research reporting guidelines (Lee et al., 1993), standards for describing agent-based models (Grimm et al., 2006), and guidelines for mathematical programmers for reporting computational experiments (Jackson et al., 1991). Others have called for reproducibility of all computational research (Code, 2010; Peng, 2011) and some go further, calling for provision of the full computational environment that produces published results (Donoho et al., 2009; Morin et al., 2012). Here we propose standard reporting guidelines for simulation-based research in the social sciences to enhance reproducibility. We focus on common scenarios encountered in the field of system dynamics, but the guidelines should be informative for other modeling approaches as well. Table 1 defines the key concepts we use in this paper. The scope of these guidelines is limited to the reporting of the model and simulation results and does not attempt to specify best modeling or analysis practices: reasonable people can disagree about how a system should be modeled, but, we argue, all should document their work in such a way that it is fully reproducible by others.1 Simulation-based research reports results of simulation and optimization experiments on a model. Therefore the guidelines that follow are discussed separately for general visualization, reporting a model, reporting simulation experiments, and reporting optimization experiments. An optimization experiment often consists of many simulation runs, and as such will follow the requirements outlined for simulation experiments. However, optimization experiments also include additional reporting requirements that are discussed separately. For each type of information we identify minimum and preferred reporting requirements, where minimum requirements are essential for research reproducibility, and preferred requirements are recommended for enhanced communication and transparency. To provide a concrete example of these reporting requirements, we introduce a simple model, presented following the requirements we propose. The model is illustrative and the numerical results reported here do not have any real-world significance. The model builds on the classical Bass diffusion model (Bass, 1969; as implemented in Sterman, 2000) and incorporates first-order autocorrelated noise (Sterman, 2000) in the adoption rate (AR) around the expected values.1 Figure 1 provides a graphical representation of the model. Given that the model is small, we include the model parameters in the diagram; for example, we show that adoption from word of mouth (WOM) depends on three constants: the adoption fraction i, the contact rate c and total population n. We note, however, that there are circumstances in which authors may choose not to include all parameters in such diagrams. Specifically, for larger models and/or contexts where the purpose of the diagram is to communicate the overall feedback or stock and flow structure it is often appropriate to include in the diagram only those parameters that are important for the presentation and discussion, omitting others, for example, Figure 1 in Mojtahedzadeh (2012). While the full documentation of larger models, which will typically be presented in an appendix or online supplement, should show diagrams that correspond exactly to the full model, diagrams showing the structure of large models should generally be avoided in the main presentation—authors should avoid cluttering their papers with complex “gazinta diagrams” (after “goes into”; see Matthews and Matthews, 2008) indistinguishable from a plate of spaghetti. Table 2 specifies the model using the preferred requirements for model reporting. The complete model is also available in the online appendix for independent assessment and reproduction. Reproducibility requires standards for the presentation of the model structure and for the numerical results that are generated by the model. Model-reporting requirements should therefore be followed whenever a simulation model is discussed or any results reported. Table 2 follows the PMRR for the example model above. One could have achieved MMRR-compliant documentation by only including the formulations. The equations are represented using letters and short abbreviations for the variable names, as is standard and appropriate in, for example, scientific journals. Alternatively one could use longer, more explanatory variable names, so-called “friendly algebra” (Morecroft et al., 1991), available in Figure 1, in explaining the equations in the text or in an appendix; the choice depends on the intended audience for the work. If only a subset of equations is discussed in the main text, full documentation, including all parameter values, must also be made available in an appendix or online supplement. A simulation experiment consists of setting up the model and conducting one or multiple simulation runs that generate numerical results. Simulation runs may differ in their parameter settings, i.e. belong to different scenarios, or in their driving random number streams, i.e., different realizations of the same scenario. The following reporting requirements apply to results reported from any simulation run(s). Table 3 reports the results of a sensitivity analysis on the diffusion model described above. The analysis changes two of the model parameters over three values each (a full factorial analysis yielding a total of nine scenarios) and runs multiple iterations for each of these scenarios, calculating the sensitivity of two outcome variables of interest (peak adoption rate (PAR) and peak time (PT)) to these parameters. The table footnotes provide the MSRR and PSRR information. Optimization experiments can be applied to deterministic or stochastic models and are used for policy optimization, calibration (estimating parameters of interest by minimizing some function of the error between the simulation and data), dynamic programming, and finding equilibria in multi-player games, among others. The following information should be provided to enable reproduction of optimization experiments. We consider an optimization problem that builds on the diffusion model above to find the advertising policy that produces a desired peak adoption rate (PAR). Minimum reporting requirements follow, with preferred requirements provided in endnote ii and the Appendix. In light of the stochastic nature of this model, we need to simulate the model multiple times and find an approximate value for the expected PAR. To do so we generate 1000 iterations of the model (using the subscript functionality in Vensim; see Appendix) and calculate the mean PAR in that sample. That value is then compared to a goal for PAR, which we set to 600,000 persons per year. We use Vensim's built-in optimization module, which uses a modified Powell conjugate search algorithm (taken from numerical recipes in C (Press, 1992) but modified to include additional constraints) to search values of “Advertising Effectiveness a” between 0 and 2 and find the value that minimizes the squared error between the PAR mean at the final time and the goal (of 600,000 people/year). We find the optimal value for Advertising Effectiveness a = 0.361 leads to the desired mean PAR. The optimal value was found after 706 simulations to accommodate 45 random restarts of the search in the parameter space.2 We thank Mohammad Jalali for excellent research assistance, and David Lane, Rogelio Oliva, David Ford and participants in the 2012 International System Dynamics Conference for helpful comments and suggestions. The desired Peak Adoption Rate is set to 600,000 people per year. Supporting information may be found in the online version of this article. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.