Biases in estimating the balance between model-free and model-based learning systems due to model misspecification

Abstract

Accumulating studies support the existence of dual learning systems in decision making: one is the model-free learning system, which updates values based solely on experience; and the other is the model-based learning system, which calculates values using a complex environmental structure. A two-stage decision task and its computational model have been widely used to distinguish the effects of these systems on choices. However, the computational model is often used as a tool without doubting its assumptions. In this study, we examined the possible biases in model parameter estimation due to model misspecification of a computational model. In particular, we focused on two features related to choice behavior, the existence of which was implied by the actual choice data but has not been assumed in the widely used computational models. One feature is the forgetting process, which assumes a change in unchosen option values. The other feature is gradual perseveration, which assumes that actions are positively autocorrelated with multiple preceding actions. We simulated cases in which these features relate to the choice process, but the obtained data were fit using a model that does not assume these features. We revealed that such misspecification of a fitting model can cause systematic biases in the estimation of the relative contributions of the model-free and model-based systems, implying that previous findings using the standard computational model might have been distorted by some biases. The possibility of estimation biases discussed in this study is important because the assumptions of the forgetting process and gradual perseveration, which can be combined with any reinforcement learning model, are not included in most existing models. In addition, the discussed mechanisms of the biases are widely related to basic model parameters. Using experimental data from the two-stage decision task (N = 39), we examined the associations between obsessive compulsivity and the weighting parameter between the model-free and model-based systems. Although a negative association has been consistently reported, we could not replicate this association when we used the model with forgetting and gradual perseveration. Instead, we found that individuals with greater obsessive–compulsive tendencies had both lower learning rates and lower forgetting rates than those with lesser obsessive–compulsive tendencies.