Two Visual Systems and Their Eye Movements: a Fixation-Based Event-Related Experiment with Ultrafast fMRI Reconciles Competing Views

Introduction

Studies of active vision in naturalistic environments show the existence of two classes of eye movements manifested in ambient and focal visual fixations [1–5]. This finding seems to corroborate with the known anatomical separation between two main cortical pathways (“streams”) of visual processing related primarily either to localization (dorsal system) or to identification of objects (ventral system). Direct verification of this connection proved to be difficult due to an insufficient resolution of the conventional noninvasive brain-imaging methods. Another hypothesis, recently proposed by Mills and colleagues [6], attributed the same behavioral observations to the lateralization of mechanisms for global and local modes of visual processing in the structures of right and left hemispheres, correspondingly. Thus, there are two tentative neurocognitive explanations for the same particularities of eye movement patterns in free image viewing.

Which of these explanations is the correct one? The answer to this question would only be possible when eye movements and brain activity can be measured with a sufficiently high temporal as well as spatial resolution. Such a measurement was not the case in previous research. For example, Mills and colleagues [6] used the functional transcranial Doppler ultrasound method with its exemplarily low temporal and spatial resolution. Therefore they relied on our finding that the exploration of complex visual scenes begins with an ambient mode of processing — short visual fixations and long-range saccades, which within a few seconds proceed to a more attentive, or focal mode of processing characterized by longer fixation and short-range saccades [3]. According to their results, this ambient-to-focal shift temporally overlaps with a shift of brain energy consumption from the right to the left hemisphere suggesting a right-hemispheric dominance in the early, ambient, phase of scene perception [6]¹.

¹Mills et al. [6] used two tasks, visual search and recognition, in their study. The interhemispheric shift of blood supply was apparent only in memory task. Unfortunately the construction of both tasks can be questioned — the targets of search tasks were too small to be detected and the recognition memory task included no memory tests at all.

Another option to access ambient and focal visual processing modes is to simply look at duration of visual fixations and the amplitudes of surrounding saccades whereby the length of the fixations is the single dominating variable [4, 5]. However, to use this option one would again need imaging tools with temporal resolution on a subsecond scale.

An attempt to create a method to explore mechanisms behind ambient and focal visual processing has been made in the framework of EU NEST Project PERCEPT [7]. The idea was to modify the event-related paradigm where visual fixations would be considered as ‘events’ instead of physical stimulation, such as the presentation of visual or auditory stimuli. In its first application, the FIBER (fixation-based event-related) paradigm with functional magnetic resonance imaging (fMRI) scanning demonstrated in a free viewing experiment that fixations on different objects in different task contexts resulted in distinct cortical patterns of activation [8]. By using multivariate pattern analysis, the authors of this study showed that the blood-oxygen-level dependent (BOLD) signals revealed information about the task context of fixations and about the binary category of objects (houses vs. faces) being inspected during these fixations. There was only one follow-up study with the FIBER paradigm [9]. It aimed at exactly the question of brain mechanisms behind ambient and focal visual fixations but the obtained results were inconsistent with the previous findings. The authors described three nonlateralized regions of activation one of which was in the primary visual cortex while two others were located in the deep ventral parts of posterior cortex without relevance to known mechanisms of active vision (see, e.g., [10]). This inconsistency could be explained by the low temporal resolution of both eye tracking (50 Hz) and the standard fMRI scanning protocol with repetition time (TR) 2.0 s used in the experiments.

In a parallel development, there were efforts to speed up the fMRI with new schemes of parallel scanning. Employing these schemes generally showed that the BOLD responses may convey more finely graded timing information than usually believed however on the costs of a decline in spatial accuracy [11, 12]. We recently adopted an alternative approach to ultrafast scanning developed at the University of Minnesota, which is free from the detriment in spatial accuracy and offers whole-brain sampling resolutions of about 0.5 s².

²Multi-band Accelerated EPI Pulse Sequence, Release 016a from 19.12.2017 (curtesy Center for Magnetic Resonance Research, University of Minnesota).

Although this is still insufficient to directly describe eye movement dynamics, the technique is a major step on the way to revealing mechanisms of active vision especially when combined with the FIBER paradigm. In addition to the use of fast eye tracking, such a combination was the unique feature of the present study. Its aims were twofold. First, we wished to test whether previous FIBER results can be replicated after these changes of fMRI protocol³.

³For a preliminary report on the method, see [13].

The second and main goal of the study was the measurement of brain functional activity in relation to ambient and focal visual fixations. The division of ambient and focal fixations was operationalized on the basis of their durations as in some earlier experiments [4, 5].

Materials and Methods

Participants. Overall 13 healthy subjects (age 21 to 31 years, right handed, 8 females) with normal or corrected to normal vision and without known history of neurological deceases participated in the experiment. Initially 15 subjects were scanned but 2 of them had to be excluded due to calibration issues of the eye tracker. Each participant gave informed consent prior to the experiment. Ethical approval for this study was provided by the local ethical committee of the National Research Center “Kurchatov Institute”.

Stimuli and equipment. To pass visual stimuli into an MRI scanner room during the functional research we used a projection system as schematically shown in Figure 1. Via a 45 degree tilted mirror, placed on top of the head coil, the subject was able to see the entire presentation display. The distance between the eyes and the screen was 75 cm. A second mirror relayed the image of the eye to the infrared camera of the eye tracker mounted at the head of the scanner bed. The paradigm was developed using NBS Presentation software. Parameters of presentation including the size of a translucent display were identical to those of the earlier FIBER study [8]. The display of 1024×768 pixels subtended a visual angle of 32×25 degrees for the entire screen. The house and face objects inside the stimulus display covered approximately 150×150 pixels. A typical stimulus configuration for free viewing conditions is shown in Figure 2. An Apple MacBook Pro (Apple, Cupertino, USA) was used to drive the stimulus display presentation.

Figure 1. Schematic view of the experiment with simultaneous registration of BOLD-signal and eye movements

Figure 2. Typical stimulus displays in the free viewing condition: (a) multiobject presentation; (b) a control test-object

For the eye tracking, a MRI-compatible eye tracker EyeLink 1000 Plus (SR Research, Canada) was used with a sampling frequency of 250 Hz. Visual fixations were recorded and categorized according to the type of object inspected and the task at hand (see below). Each subject was placed to Magnetom Verio 3T (Siemens, Germany) MRI scanner with 32-channel MRI head coil. During the experiment, we simultaneously recorded fMRI data by a combination of FIBER paradigm [8] and ultrafast Multi-band Accelerated EPI Pulse Sequence. The scanning process had two stages: capturing high-resolution anatomical data and recording functional data by a parallel scanning protocol with ultrafast EPI-sequence (TR=720 ms, TE=33 ms, 56 slices, slice thickness = 2 mm, spatial resolution in each slice = 2×2 mm).

Procedure. During the scanning session, we collected functional data in two experimental paradigms: “localizer” and “multiobject viewing”. The localizer part was a standard experiment with passive observation (block paradigm). Subjects viewed pictures of houses or faces in the center of the screen while BOLD-activity was tested in search of cortical areas predominately involved in perception of houses or faces [8, 14–16]. Figure 3 shows a scheme of the experiment for functional localization. The experiment consisted of 4 blocks, each including 2 parts. The first part of a block contained 15 stimuli with pictures of faces and a fixation cross. The second part of a block contained 15 stimuli with pictures of houses and a fixation cross. The time of each stimulus was 750 ms, fixation cross time — 10 s, blank interval — 500 ms, total trial time — 11.25 s. Structural MRI and resting state BOLD-activity were registered individually with closed eyes during 10 min preceding each localizer experiment. Overall length of the experiment was about 15 min.

Figure 3. Scheme representation of the localizer experiment

In the multiobject experiment, subjects inspected a display containing spatially-fixed circular arrangement of 3 house and 3 face objects with the instruction to look at houses (house task, HT), at faces (face task, FT), or at houses and faces (all task, AT) with the purpose of a later recognition. A control test-object was shown after each multiobject stimulus presentation. Subjects had to answer by pressing “YES” and “NO” buttons to indicate whether the object was shown among the objects on the previous stimulus display (akin in Sternberg short-term recognition memory test). Random guessing baseline was 50%.

Figure 4 shows a scheme of the experiment with multiobject viewing. The experiment consisted of 4 blocks. Each block consisted of 4 screens.

Figure 4. Scheme representation of the multiobject experiment with house (HT), face (FT), and free viewing (AT) conditions (see text for explanation)

Screen 1: Instruction (10 s). Subjects were given 3 types of tasks: “Look at faces” (FT), “Look at houses” (HT), or “Look at all objects” (AT).

Screen 2: Stimulus display consisting of 3 images of houses and 3 of faces, placed randomly in a circle at equal distances in 6 fixed positions (Figure 2 (a)). Stimulus duration varied pseudorandomly from 8 to 18 s. Due to the varying duration of the presentation, the blocks also varied in their total duration.

Screen 3: Control test-stimulus (3 s). The stimulus was a single house or face image (depending on the task) which was placed at the center of display.

Screen 4: Fixation cross (10 s).

Each block included 3 groups of presentation cyclically repeated 4 times and differed by the type of task. Numbers 2 and 3 correspond to “Screen 2” and “Screen 3” in a block. Each block ended with the fixation cross. Eye tracking was employed only during “Screen 2”. This experiment was preceded by standard 9 points eye tracker calibration and lasted for about 20 min.

Data processing. For all subjects in the multiobject experiment, we acquired saccades and fixations data that were afterwards regularized using EyeLink Data Viewer (SR Research, Canada) software. Pictures of eyetracks preceding and following such regularization are shown in Figure 5. Each fixation was additionally marked depending on the category of the foveated object and actual instruction (type of task). Some fixations were excluded from the further modelling of hemodynamic responses (see below) according to the following criteria: 1) fixations shorter than 80 ms, 2) fixations longer than 1000 ms, and 3) fixations that were not clearly attributable either to houses or faces, e.g. located at the center of display presentation of the 6 objects. In our analysis of brain mechanisms behind ambient and focal visual processes, we considered a larger set of fixations by also including fixations not clearly attributable either to houses or faces. We further differentiated ambient and focal fixations on the basis of fixation length parameter using for decision approximately the median-threshold of 280 ms as it was done earlier [5]. To ensure that subjects followed the instruction, we automatically registered correctness of their responses in the recognition-memory test.

Figure 5. An example of eye tracking data in free viewing task (AT): (a) raw eyetrack; (b) regularized eyetrack

BOLD-data from localizer and multiobject experiments for each subject were preprocessed using SPM8 software (Wellcome Trust Centre for Neuroimaging, London, UK) in Matlab R2018a (Mathworks, Natick, MA). Preprocessing consisted of realignment to correct for subject movements, co-registration to align all functional data to subject’s anatomical volume, normalization to convert all images to Montreal Neurological Institute (MNI) space and spatial smoothing with a Gaussian kernel of 8 mm (full width at half maximum).

From the data of the localizer experiment, two regions of interest (ROIs) were defined which were predominately involved in perception of houses or faces. In fMRI data analysis of multiobject experiment, we reconstructed the hemodynamic responses to study activity patterns in both ROIs revealed in the localizer experiment and in early visual cortex areas. The latter were defined as spherical regions of 10 mm radius in the visual cortex of each hemisphere around location with MNI coordinates –34 –71 –16 in the left hemisphere and 28 –71 –16 in the right hemisphere.

Differences in the BOLD responses evoked by each fixation category (fixations on houses or on faces) were investigated by modelling their associated hemodynamic responses. At the single subject level, a model was defined using both the onsets and the durations of fixations of both categories corrected for a typical delay of the BOLD hemodynamic response function (HRF). These models were estimated in SPM8 (Restricted Maximum Likelihood estimation) using the informed basis set represented by HRF amplitude, derivative and dispersion [8, 17]. In the final reconstruction of brain structures involved in ambient and focal visual processing, we used individual resting-state data as baseline.

Results

Eye movements and memory performance. During the multiobject experiment, we registered nearly 80,000 visual fixations. Their proportion according to task at hand and exclusion criteria is shown in Table 1. Most of the excluded fixations belonged to the group of “not clearly attributable either to houses or faces”. At the end of our study, they were again taken into account in the analysis of brain mechanisms behind ambient and focal processing, the proportion of considered fixations then increased to more than 92%. The data summarized in Table 1 show that subjects generally followed task instruction by looking predominately at the objects of appropriate category. In the group of “wrong” fixations, we found 3 times more fixations on faces during the HT than on houses in the FT. This result evidences a strong bias for fixating faces compared to houses. A similar tendency can be seen even in the balanced AT condition.

Table 1. Mean number — and respective percents — of visual fixations used in fMRI analysis of multiobject experiment (data for one “averaged” subject from the group of 13 participants)

In order to demonstrate directly that the subjects performed the required tasks, we examined their scores on the Sternberg recognition-memory task. Average performance was 85% (±10% SD) correct for both the HT and FT and 63% (±12% SD) for the AT. The difference in performance is not surprising given that the AT condition requires remembering double the number of items. One can conclude from the data that subjects performed the tasks appropriately. These scores were not used in the further fMRI analyses.

In view of a large number of earlier studies, the time course of relation between fixation lengths and saccade amplitudes during the first several seconds after presentation of complex images is of particular interest. Figure 6 shows the results of such analysis. Contrary to the bulk of previous reports [3, 6, 8, 18], our data show no trends of growing fixation durations and falling saccadic amplitude. This unusual result could be related to the full spatial predictability of the multiobject displays in the present study. It also provides the reason for our selection of fixation length as the critical parameter to distinguish between ambient and focal modes of visual processing.

Figure 6. Fixation duration (left) and saccadic amplitude (right) over stimulus presentation time with a running average across 50 ms both

fMRI data. The localizer experiments led us to the description of the same two bilateral areas predominately involved in perception of faces or houses as in pioneering works by Kanwisher and colleagues [14, 16] and in the referent FIBER study [8] namely fusiform face area (FFA) and parahippocampal place area (PPA). Their MNI centers of coordinates were as follows: FFA (left: –40 –50 –19; right: 42 –49 –17); PPA (left: –24 –49 –9; right: 26 –47 –10). The four ROIs were defined from the T-maps of the localizer experiment, using MarsBar [19] and uncorrected p-values of 0.01. Together with earlier defined regions of primary visual cortex, this resulted in 6 (or 3 bilaterally located) ROIs which were used in modelling hemodynamic response depending on the category of the foveated objects and the task at hand. The results of such modelling averaged across subjects and hemisphere localization are presented in Figure 7.

Figure 7. Modelled hemodynamic responses to house fixations (red) and face fixations (blue) with standard error of the mean (gray) in three task conditions for three ROIs: PPA, FFA, and early visual cortex (averaged models for all subjects and both hemispheres)

In the last phase of our analysis, we looked for changes of the whole-brain functional activity in relation to ambient and focal visual modi of visual processing operationalized on the basis of fixation durations. Eye movement information for FIBER computing was used irrespective of fixating or not fixating particular objects. The contrasts with the resting state baseline data were drawn using a more stringent level of significance criterion than before: p-values <0.0001 (uncorrected). Our analyses revealed several regions in posterior parts of the brain that were activated in comparison to the resting state. No inhibited regions were discovered. The MNI coordinates of identified regions with numbers of voxels involved are listed in the Table 2. The same results are presented in a graphical form in Figure 8.

Table 2. Brain regions showing a significant increase in FIBER whole-brain fMRI activity associated with ambient and focal visual fixations (all significant changes to the resting-state baseline data, p<0.0001, uncorrected)

Figure 8. Brain activity maps in the multiobject free viewing experiment: (a) ambient mode of visual processing (short fixations); (b) focal mode of visual processing (long fixations)

Discussion

In the present study, we followed two interrelated goals. First, we aimed at a substantial improvement of temporal resolution in BOLD-signal measurement up to a level sufficient for multimodal fMRI and eye tracking experiments. Our second goal was to describe brain functional activity in relation to ambient and focal fixations in free viewing complex visual displays. Using eye movement information as parameters for the FIBER fMRI analysis combined with ultrafast scanning protocol, we confirmed reports by Marsman, Renken, Velichkovsky, et al. [8] that the fMRI signal contains information to identify the task and executed and the type of object inspected by the subject. Moreover, in some respects we got a more elaborated pattern of results than in the previous study. This can be explained by a number of details that multiplied sensitivity of our method: a 5 times higher temporal resolution of the eye tracking (250 vs. 50 Hz), the whole-brain fMRI scanning within 720 ms (instead of 2 s) and the use of 32-channels MRI head coil (instead of one with 8 channels)⁴.

⁴Recently, one can observe a re-birth of interest to the FIBER type of analysis however up to date only in reading studies and without employment of ultrafast scanning protocols [20, 21].

First of all, it can be seen that brain processing of both categories of objects — houses and faces — is rather dissimilar. Data on eye movements (see Table 1) show a strong bias to fixate faces and not houses while modelling of hemodynamic responses in the FT as well as in the HT conditions (see Figure 7) demonstrates a faster increase of reaction to faces in comparison to houses. This special character of face processing is widely discussed in the literature, for instance, in EEG event-related potentials studies that reliably demonstrated a face-specific reaction as early as 170 ms after stimulus presentation [22].

Similar arguments were once defined to consider the FFA as a prototypical module of brain cognitive organization [16]. However other aspects of results seem to challenge the modular interpretation of face processing, which should be mandatory task-independent and associated with a particular brain structure [23, 24]. One of the discrepancies is that similar hemodynamic responses to face images can be observed also in the PPA and, rather counter-intuitively, in the early visual cortex. One would need additional hypotheses to understand this broad similarity such as the assumption that hemodynamic responses have a travelling-wave nature [25, 26]. The next inconsistency is more difficult to explain. As one can see, the AT condition reverses the relations between the hemodynamic responses to faces and houses so that now the latter start to produce faster and stronger reactions at all studied brain locations including the early visual cortex. Such a reversal under the influence of a task is impossible in strictly modular architecture working as a mosaic of “cognitive reflexes”. In our view, the reversal can be related to subject’s attempts of counterbalancing the prioritization of faces in the AT condition that explicitly requires equal attention to both object categories.

If indeed the BOLD-signals may “travel” across different brain locations, one would expect the observed broad similarity of hemodynamic responses in dependence on the task at hand at intervals of several seconds. In contrast, the core specialization of particular brain structures would become less evident in such long time segments. Thus, for a sharp spatial localization one needs a maximum of temporal resolution. In the second part of our study, we applied ultrafast multi-band scanning protocol together with FIBER analysis to reveal brain mechanisms behind ambient and focal visual fixations as events on a subsecond scale.

There are currently two alternative explanations for these brain mechanisms: one stresses the division of dorsal and ventral pathways in perception [4], another emphasizes interhemispheric differences in global and local modes of processing [6]. The results of this analysis summarized in Table 2 and Figure 8 show that both hypotheses are to some extent confirmed. In line with our early proposal, ambient fixations were accompanied by an activation of structures associated with the dorsal visual pathway, while focal fixations correlated with that of the ventral pathway. At the same time, the second hypothesis is also proved to be correct: the activated structures of the dorsal pathway were localized in the right hemisphere and those of the ventral stream mainly — albeit not exclusively — in the left hemisphere.

More specifically, short fixations associated with ambient mode of processing correlate with bilateral activation of early visual cortex (BA17 left and right) and a number of structures that increasingly demonstrate rightward lateralization. The asymmetry is already present in the superior occipital gyrus (BA19): 53 activated voxels on the left vs. 316 on the right side. Area 19 is considered to be the differentiation point of the two visual streams [27] and probably receives inputs from the retina via the superior colliculus and pulvinar. Furthermore, only right-hemisphere lateralized parts of the cuneus and the superior parietal lobule are activated. The latter is considered as the core region of the dorsal stream involved with multimodal functions of the parietal lobe in general, such as reaching and grasping behaviour [28]. In contrast, long visual fixations associated with focal mode of processing are connected to activation of the fusiform gyrus in the left hemisphere. The fusiform gyrus is a classical structure of the ventral stream. A novel result is also a locus of activation within the lobule VI of the right cerebellum. One has to note that the right cerebellar lobule VI is involved in the tasks which have a cognitive component and simultaneously activate regions of the left hemisphere [29]. Overall, cross-lateral connectivity seems to be a frequent case in building functional systems of neocortical areas and that of the cerebellum [30].

The current reconciliation of competing views on the brain mechanisms behind ambient and focal fixations poses at least three further research questions. The first one is about the way these mechanisms actually work: it is necessary to clarify cause-and-effect connections among the described structures, e.g. by applying the dynamic causal modelling [10, 31, 32]. The second question is about general biological and computational reasons behind this and other forms of hemispheric asymmetries [33]. Lastly, an unexplored issue remains the role of brain structures located upstream of the dorsal and ventral visual streams. In a speculative way, one can suppose that ventromedial vs. ventrolateral locations of activity clusters that were attributed to ambient and focal modi of eye movements in one of the studies [9] could be related to a projection of the dorsal and ventral visual pathways on the deep cortical layers around hippocampal formation [34], where they also become to play a role in different forms of memory [35]. A low temporal resolution of previous studies may be the reason that only these late components of brain activity involved with free visual behavior were revealed.

Conclusion

In the present study, we realized the use of fixations in free viewing as events to model brain activation in an experiment using ultrafast multi-band scanning protocol and FIBER paradigm of data analysis. This allowed us for the first time to demonstrate contrasting lateralization of brain mechanisms belonging to dorsal and ventral streams in charge of visual perception and eye movements in free processing of multiobject images. Therefore the new multimodal imaging methodology could make a definitive contribution to the exploring brain mechanisms behind ambient and focal modes of active vision. A next step in this line of research would be a modelling of effective connections among neural networks involved with active vision. The conclusion also poses a number of further questions about a possible relation between two modes of active vision and other structural and functional asymmetries found at different levels of the human brain functional organization.

Acknowledgment. This work was partially supported by the NRC “Kurchatov Institute” (grant No.1359). The development of algorithms for classification of eye movements in fMRI studies was funded by the German Research Foundation (DFG, Deutsche Forschungsgemeinschaft) as part of Germany’s Excellence Strategy — EXC 2050/1 — Project ID 390696704 — Cluster of Excellence “Centre for Tactile Internet with Human-in-the-Loop” (CeTI) of Technische Universität Dresden.

We thank Dr. V. Verkhlutov for sharing with us his knowledge on the travelling-wave processes in the human brain.

Conflicts of interest. The authors have no conflicts of interest to disclose.