The Effects of Performance Information on Public Resource Allocations: A Study of Chile's Performance-Based Budgeting System

Abstract

ABSTRACT This article reports the findings of a mixed-methods study of the effects of performance-related considerations on the allocation of public resources in Chile. The study considered aspects and stages of the budgetary process that most previous research on the relationship between performance assessments and the budget has tended to disregard. By virtue of this distinctive approach, the study identified many instances of performance considerations' influence that many other studies would have failed to capture. Because of its strictly exploratory character, the study could not establish how typical the different types of effects identified are. But it suggests that the effects most neglected in the literature may account for a great portion, if not for most, of the performance assessments' impact on the quality of public spending. ACKNOWLEDGEMENT I would like to thank Professors Sonia Ospina and Shanna Rose, from New York University, for their invaluable support throughout the development of this study. I am also grateful to Marcela Guzmán, who at the time was the chief of DIPRES's Management Control Division, for her crucial cooperation with the project. Notes Note: ∗Significant at the 0.10 level; ∗∗Significant at the 0.05 level; ∗∗∗Significant at the 0.01 level. Robust standard errors reported. The coefficients for agency fixed effects were excluded, to make the table manageable. Note: ∗Significant at the 0.10 level; ∗∗Significant at the 0.05 level; ∗∗∗Significant at the 0.01 level. Robust standard errors reported. Note: ∗Significant at the 0.10 level; ∗∗Significant at the 0.05 level; ∗∗∗Significant at the 0.01 level. Robust standard errors reported. Note: ∗Significant at the 0.10 level; ∗∗Significant at the 0.05 level; ∗∗∗Significant at the 0.01 level. Robust standard errors reported. †The table does not examine the relationship between the agencies' MIP compliance status and budget decreases because only one of them experienced a reduction in its personnel budget. Note: ∗Significant at the 0.10 level; ∗∗Significant at the 0.05 level; ∗∗∗Significant at the 0.01 level. Robust standard errors reported. Note: ∗Significant at the 0.10 level; ∗∗Significant at the 0.05 level; ∗∗∗Significant at the 0.01 level. Robust standard errors reported. Note: ∗Significant at the 0.10 level; ∗∗Significant at the 0.05 level; ∗∗∗Significant at the 0.01 level. Robust standard errors reported. The management control cycle involves a variety of activities, including planning, coordinating the activities of various parts of the organization, communicating and evaluating information, deciding what action (if any) should be taken, and influencing people to change their behavior (Anthony and Govindarajan Citation2007, 7; see also Simons Citation1995). See, for instance, GASB (Citation2002), Zaltsman (Citation2006), and Mackay (Citation2004). The program grants a 5 percent salary bonus to agencies that comply with 90 percent or more of their MIP targets, and a 2.5 percent bonus to those that comply with between 75 and 89 percent of their targets. Instrumental use involves the direct application of a piece of information to a concrete decision. The evaluation utilization literature distinguishes it from other forms of information use, such as persuasive and conceptual use. Persuasive use implies relying on the information to justify or advocate for decisions that were made on other grounds, whereas conceptual use does not entail a direct link between particular pieces of information and specific decisions: it occurs through the generalizations and conceptual perspectives that the information generates (Weiss Citation1977; Weiss Citation1985; Leviton and Hughes Citation1981). In addition, in the specific context of Chile's PBB system, Zaltsman (Citation2008) identifies a fourth type of performance information use that he refers to as "retributive use." This form of utilization involves the use of M&E information for rewarding or penalizing different levels of performance. See, for instance, Petrei (Citation1998) and OECD (Citation2004). This is a problem affecting PBB systems in many parts of the world. As Schick (Citation2007, 125) explains, "few governments have robust cost accounting systems that enable them to measure marginal costs by distinguishing between fixed and variable costs, as well as between marginal and average costs." For a discussion on the pragmatic approach to mixed-methods research, see Greene et al. (Citation1999). The interviews' sample included the current and former national budget directors, the head of DIPRES's Management Control Division and the senior officials directly responsible for the evaluation component, the chiefs of DIPRES's line budget divisions, high-ranking officials from several line ministries and agencies, three members of Congress, and current and former evaluators. In the case of the focus groups, the participants were program evaluators, authorities of evaluated programs, budget officials from agencies and line ministries that underwent evaluations, DIPRES budget analysts, and members of DIPRES's Management Control Division's staff. A summary of these interviews and focus groups findings was published as an appendix of the World Bank study's final report (Banco Mundial Citation2005). Throughout the paper, I reserve the term "appropriation" for the allocation decisions sanctioned through the national budget law, and use the phrase "resource allocation" more generally to refer to any decision on how to assign (or propose to assign) resources. It is important to note that many of the allocation choices that different budget actors make during budget preparation do not make their way into the budget law. The following evaluation recommendations, extracted from the evaluation-based pledges of the eight agencies that make up the study's sample, illustrate this point: assessing the convenience of decentralizing the program based on a study that was already completed; adjusting the program's current targeting criteria; creating a data base based on information that was already available; redefining the program's strategic guidelines based on the evaluation's results; identifying and tracking the program's administrative costs; and conducting a feasibility assessment of the introduction of a customized user fee scheme. The data can be accessed at www.dipres.cl. The analysis focused on both the inter-annual differences between the funding levels proposed in the president's budget bill for year t and the appropriations that had been approved for year t−1, and between the appropriation levels approved for both years. But, as discussed in the findings section, the results turned out to be virtually the same regardless of which of the two was taken as the dependent variable. This is why, for the sake of simplicity, this section makes no distinction between the two variables, and refers to them generically as "budget changes." In an attempt to control for the potential influence of the other performance assessments, I also run these models with the other M&E assessment variables as covariates. Since this did not result in any substantial change in the significance levels or signs of the main predictors' coefficients, I only report the findings of the most parsimonious version of the models. A previous version of the analysis included the agencies' budget execution rates as of year t − 2, a proxy for the proportion of the agencies' budget that was made up of "rigid" allocations (i.e., entitlement programs, constitutional requirements, debt repayments, etc.), and a dummy that singled out the agencies that were involved in the implementation of programs that the government defined as "prioritary" in each specific year. But, since neither proved to be significantly associated with the outcome variables, they were not included in the models presented in the findings section. Similarly, a separate analysis replicated the regression models as a two-level analysis that controlled for potential line ministry effects. Since the results of that analysis were substantially the same, the paper introduces the regression models in their original ordinary least squares (OLS) form. Taking the budgets of the agencies that had at least one M&E assessment record and complete budgetary data for the 2002–2005 period as an example, the correlation between the funding levels proposed in the president's budget bill and the ones approved in the national budget law is virtually perfect (Pearson's r = 1.000), with 86.3 percent of the observations presenting exactly the same value. While, in theory, all the agencies in the sample are supposed to participate in the PI and MIP assessment cycles, in practice some of them have taken more time than others to do so. This may have been a consequence of differences in their level of "readiness" to implement the organizational routines that those assessment activities require, or in their eagerness for having their performance evaluated. On the other hand, the agencies do not participate in the decision as to what programs or agencies will be subject to evaluation studies; that decision lies with DIPRES and Congress. But it seems reasonable to expect that the relatively few agencies that are selected each year for in-depth evaluation present certain distinctive characteristics (e.g., presumed weak performance, political sensitivity, etc.) that make them more likely targets for this type of assessment than the rest. For example, while for some of the agencies that underwent evaluation assessments the evaluations concerned less than one percent of the budget, for others they involved their totality. Nevertheless, in this part of the analysis, the performance-related variables treat all agencies equally. That being said, the group F-statistic for % PIs Complied With t−1, Number of PIs t−1 and the interaction term remains significant, suggesting that there is in fact a correlation between this subset of variables and percent budget change. The fact that none of the individual coefficients is statistically significant may be due to the fact that % PIs Complied With t−1 and Number of PIs t−1 are correlated with each other (i.e., the larger the number of indicators, the more difficult to comply with all of their targets). This may explain why the standard errors of their individual coefficients are too large to reject the null hypothesis. Additional informationNotes on contributorsAriel Zaltsman Ariel Zaltsman (ariel.zaltsman@nyu.edu) is currently a research fellow at the Inter-American Development Bank's Office of Evaluation and Oversight, and specializes in program monitoring and evaluation, and performance-based budgeting and management. In previous years, he worked as a consultant for the World Bank, the Latin American Center for Development Administration (CLAD) and Argentina's Ministry of Economy. Mr. Zaltsman earned his Ph.D in public administration from New York University.