Using eye-tracking and EEG to study the mental processing demands during learning of text-picture combinations

Abstract

Using and combining eye-tracking and EEG frequency band power as process measures, in the current study we were interested in the mental processing demands during learning of text-picture combinations that either enabled or prohibited text-picture integration (TPI). In the mismatch condition, the textual and pictorial information being dissimilar, TPI was not possible, whereas in the match and the partial-match condition, the textual and pictorial information being identical respective complementary, TPI was possible. We expected mental processing demands to be higher in the mismatch condition, when pictorial and textual information had to be processed and memorized as separate representations, compared to the match and partial-match conditions when TPI was possible. As expected, on virtually all process measures we observed increased mental processing demands when two mental representations had to be processed and memorized compared to the two conditions where TPI was possible. The EEG alpha and theta frequency band power data corroborated and extended the eye-tracking measures of mental processing demands. In addition, we performed a fixation-related EEG frequency band power analysis that also corroborated the results of the classic stimulus-locked EEG frequency band power analysis, exemplifying the use of this former methodology in the context of complex multimedia task materials.