Why the A/AN prediction effect may be hard to replicate: a rebuttal to Delong, Urbach, and Kutas (2017)

Abstract

In our recent publication "How robust are prediction effects in language comprehension?Failure to replicate article-elicited N400 effects" (Ito, Martin, & Nieuwland, 2016a), we report two experiments which failed to replicate existing event-related potential (ERP) evidence for prediction as reported in C. D. Martin et al. (2013), whose study resembled DeLong, Urbach, and Kutas (2005; from hereon DUK05).DeLong, Urbach, and Kutas (2017; from hereon, DUK17) recently published a commentary which depicts our publication as a case of poor scholarship that makes "no substantive contribution to the literature on what factors may matter for prediction and when."DUK17 warn that the readers of our work "will be led into serious error."In this rebuttal, we first present evidence that is inconsistent with the arguments of DUK17 regarding our own experiments, and then we briefly discuss other indications why it might be hard to observe and thus replicate the A/AN prediction effect.To address DUK17's arguments regarding our own study, (1) we discuss why the observed null noun effect in non-native speakers in our Experiment 1 is in fact not a failure to replicate, in contrast to what DUK17 state, and we discuss how DUK17 conflate noun effects stemming from plausibility and semantic congruence in their discussion of bilingual ERP studies, (2) we report a linear mixed-effects model analysis on both of our experiments that fails to replicate the graded effect of cloze probability on article-elicited ERPs as observed in DUK05, and (3) we report the results of Bayesian analyses showing evidence in favour of the null hypothesis that article-cloze has no effect on article-elicited ERPs.Then, (4) we turn to the replicability of the landmark findings reported by DUK05, reviewing previous attempts to replicate their findings, and (5) end with a very brief discussion of the relevance of prediction effects reported in other articles.We emphasise that we believe that prediction could play an important role in language comprehension, and we also do not see prediction as an all-or-nothing phenomenon.In concord with DUK17, we think that previously reported ERP effects elicited by articles marked for gender agreement or animacy provide evidence that favours prediction accounts over integration accounts.But the focus of our 2016 publication, and of our points in this rebuttal, is replicability of the prediction effects reported for the English indeterminate articles a/ an, which abide by phonotactic but not agreement rules, and which have been argued to be evidence of the preactivation of phonological information during reading and language processing more generally.We also wish to state why we performed our experiments in the way that we did.The experiments were a replication attempt of the C. D. Martin et al. study, not that of DUK05.The items were presented as filler materials for another study on prediction (Ito, Corley, Pickering, Martin, & Nieuwland, 2016;Ito, Martin, & Nieuwland, 2016b), which used isolated sentences that often contained a semantic anomaly.To have a more uniform set of items in the complete experiment, we changed the two-sentence items of the C. D. Martin et al. study into single-sentence items, and re-normed them.We first ran the experiment using an stimulusonset asynchrony (SOA) of 500 ms because that is the SOA that C. D. Martin et al. described in their Methods section.Later, we discovered that this SOA was incorrectly reported and was, in fact, 700 ms.We therefore repeated the experiment with the longer SOA of 700 ms.This was very important, because non-native participants tend to read more slowly than native