Individual Stable Patterns of Human Brain Rhythms as a Reflection of Mental Processes

Introduction

Recognition of perception and thinking based on brain signals. The method of recognizing mental states from brain signals is often called “brain reading”. Notably, this term was first attributed to the technology of restoring the subjective content of the human brain by using functional MRI (fMRI) [1], and not EEG (as we proposed 8 years before the term appeared in the literature [2, 3]).

At the moment, the research into “brain reading” can be divided into several narrower specialized areas.

Recognition of categories of perceived objects by fMRI. In this specific field, the study of Haxby is the best known [4]. The subjects confined to a fMRI machine were presented with visual stimuli of several categories (houses, shoes, furniture, faces, etc.). By analyzing the activities of the higher visual and temporal associative regions of the cerebral cortex, the authors found differences between the activities caused by stimuli of different categories. Furthermore, each category of stimuli caused a specific multicomponent pattern of the MRI signal; it was impossible to find a single component related to the given category of stimuli. The authors were also able to determine a category of the object, which the subject is currently viewing. This recognition process was performed with a reliability of >95%. The quoted study proved it was possible to determine the subject’s cognitive state in real time using a brain signal (with an accuracy of up to the time resolution of the fMRI method — about 10 s).

A group of scientists led by Pietrini [5] performed similar experiments, but with the tactile presentation of stimuli to healthy subjects and to people who were born blind. It was found that tactile stimuli could also be well discriminated by categories, and that each category had its own brain activation pattern. It was also noted that the blind subjects used additional visual cortex to identify the tactile stimuli.

Shinkareva developed a technique to use the BOLD signal for the recognition of subjects’ perception of images of tools or dwellings [6], thus confirming the results of Haxby.

The further development of brain reading research was a technologically sophisticated work [7]. In this, the subjects were presented with video fragments and then, using the fMRI signal and the Gabor filters, the entire video image was restored. Although the quality of the restoration was not ideal, the study aroused an interest and was then actively quoted.

Recognition of the nature of mental activity using fMRI. Most of all, ideologically close to our research were studies of Mitchell and his colleagues, in particular, the results described in their article of 2004 [8]. The authors asked the subjects to perform various cognitive tasks. For example, they showed a picture and then (after some time) a sentence. The subjects were asked to guess whether the sentence related to the picture or not. The tasks were presented repeatedly with parallel fMRI measurements.

The researchers aimed to distinguish between the time intervals whereby the subject performed various cognitive actions, for example, viewing a picture or comprehending a sentence. The mental states were recognized on a pseudo-real time scale with an accuracy of the time resolution of the method, which in this case was about 5 s. The support-vector machine turned out to be the best trainable classifier. The relative classification error for the above example of tasks was 0.11 (0 is the ideal classification; 0.5 is a random one).

The authors proposed that in the future this tool would allow one to track a person’s “trajectory of thought” and use it for training, cognitive research, and medicine (neurology and psychiatry). In [9], the same researchers were able to predict the patterns of brain activation upon presentation of verbal stimuli that had not been used previously.

The brain–computer interface (BCI) paradigm. The main purpose of the BCI is the control of external devices (trolley, exoskeleton, computer, etc.) by using electrical signals from the brain but not using muscle force. BCIs can be invasive (intracranial electrodes) or non-invasive (surface EEG). It is even more important that there can be so-called synchronous and asynchronous interfaces. The first are based on recognition of the evoked brain activity: for example, the “typewriter” BCI, which is based on the P300 component of evoked potential. In a BCI of the second type (asynchronous) a person arbitrarily changes his/her thoughts; the system, by analyzing the simultaneous EEG, guesses the thoughts and takes necessary actions. It is obvious that synchronous BCIs have limited and highly specialized fields of use (for example, establishing contact with a completely immobilized patient). On the contrary, asynchronous BCIs (if well developed) can find a wide and universal use in medicine and other areas of science and technology (for example, in industry or military).

Thus, non-invasive asynchronous BCIs are also, in essence, “brain reading” devices. As the main mission of the BCI is to control external devices, the mental states are often represented by imaginary movements [10, 11]. In these systems, the power indices of the motor cortex rhythms in specific frequency ranges [10] or the event-related desynchronization [11] are used. In some cases, the accuracy of recognizing an isolated imaginary movement reaches 98% [11].

The BCI, based on imaginary movements, is being actively developed in Russia by a team led by Professor Frolov. The use of additional methods for processing the original signal (for example, the independent component analysis) has increased the reliability and sensitivity of the system [12, 13].

Recognizing the type of mental activity by EEG. There are just few reports about recognizing the mental states by using EEG — without linking it to applied tasks, as in the case of BCI [14, 15]. The study [14] has much in common with our research. The subjects maintained five consecutive mental states: rest, imagination of movements with the right or left hand, mental rotation of a cube, subtraction of numbers. A compact artificial neural network was used as the classifier to perform classification into three classes; the results were successful in some cases. Thus, the rest, the cube rotation and the imaginary hand movement differed from each other with a reliability of about 90%. The states in the “rest–subtraction–hand” triad also differed quite well. At the same time, the cube rotation and the subtraction tasks could not be reliably distinguished. Notably, in our report published five years earlier [2], the rotation and the mental arithmetic tasks differed from each other with an average reliability of 87% in six subjects.

In [15], short episodes of solving an arithmetic problem against the background of the current EEG were detected using multifractal analysis with about 100% reliability.

Quantifying the mental activity and creating the respective spaces. Psychological quantification (scaling) has been used for decades; it has always been based on transformations of subjects’ answers (subjective psychological scaling) or on behavioral tests (objective psychological scaling). As a result, the so-called perceptual spaces were obtained.

The approach to psychological scaling is based on the assumption that the judgment about a similarity or difference between two mental phenomena can be expressed in the form of distance between the points reflecting these phenomena in some space. The more similar the two mental phenomena, the closer to each other in space are the points representing them, and vice versa.

Scaling based on psychophysical experiments and creating a perception space was demonstrated in studies [16–20], and scaling based on questionnaires — in [21]. Notably, the above-mentioned studies were conducted in Russia in the 1970s and 1980s — in the heydays of instrumental psychology in the Soviet Union. The most of credit goes, to E.N. Sokolov.

Of the current international publications on mental state mapping, the study based on fMRI is worth mentioning [22]. The authors depicted a semantic tree imposed on the straightened image of the cerebral cortex.

Thus, in the above section, we briefly reviewed the major publications, ideologically and/or methodically related to ours.

Methods

General design of the studies. For more than 20 years, we conducted dozens of experiments, in which hundreds of subjects took part; yet all the experimental schemes have had common features.

1. In one experimental series, all subjects followed the same scenario. The subject performed several tasks, and for each of them, there were dozens of stereotypical variants. All tasks were mixed and presented in a random order. The difficulty of the tasks was chosen such that to keep the execution time around 10–20 s and not to exceed the percentage of errors beyond 30%. In experiments with emotions, the stimulation parameters were somewhat different from the above due to a specific nature of the process. On average, an experiment with one subject lasted for 1.5–2 h with one or two breaks.

2. During the entire experiment, EEG recording was carried out using 19 electrodes set according to an extended system of 10–20%. Here, we used the standard EEG recording parameters: the digitization frequency was 250 Hz, with signal filters of 0.1 to 70 Hz, the notch filter was 50 Hz, and the electrodes’ impedance was <10 kΩ. Throughout these years, we used EEG equipment of various kinds: Biotop (Japan), Medicor (Hungary), ATES Medica (Italy–Russia), and Medicom-MTD (Russia). Geographically, the first series of experiments were performed in Japan, at the Brain Functions Laboratory (Kawasaki); then, most of the experimentation was done at the Laboratory of the Human Higher Nervous Activity at the Institute of Higher Nervous Activity and Neurophysiology of the Russian Academy of Sciences in Moscow. The fact that the experiments conducted in different countries and laboratories using different equipment, gave fundamentally the same results is an additional argument for the validity of our results; it also suggests that we observe a universal and stable phenomenon that depends little on the methodological details.

3. In parallel with the EEG, notes were recorded along a separate channel and an electrooculogram was also recorded via two channels. With the help of a regression procedure (described in [23]), corrections for oculomotor artifacts were introduced. These corrections enabled our subjects to make free eye movements, specifically, we did not ask them not to blink.

4. All data processing was performed on the individual basis, separately for each subject.

EEG preprocessing. Prior to analyzing the EEG data, we preprocessed all EEG records.

1. From the continuous EEG recording, we selected epochs corresponding to the fulfillment of tasks, and (in some cases) epochs from the inter-stimuli intervals corresponding to the state of operative rest. In experiments with emotions, EEG segments corresponding to the presentation of emotionally significant stimuli were chosen.

2. We calculated the squares of the Fourier transform module for individual EEG records, loosely termed by us the “single power spectra”. The Fourier transform is chosen because the frequency spectrum provides an adequate and illustrative assessment of the rhythmic character of the EEG. The size of the EEG analysis window was 16 or 32 s, except for experiments with real-time (cognitive BCI and cognovisor, see below), where it was 2 or 4 s. The range of analyzed frequencies in the cognitive experiments was from 5 to 20 Hz; in experiments with emotions, the lower frequency level dropped to 1.6 Hz (to “capture” the delta rhythm). The rhythms of the upper beta range, as well as the gamma range, were not considered in this experimentation series.

3. Additionally, averaged EEG power spectra, characteristic of each task or emotion, were created; that gave us a general visual assessment of differences between the EEG patterns.

EEG-based classification of mental states using an artificial neural network. As a classifier used for the recognition of mental states by EEG, we used a simple artificial neural network of the Perceptron type, described in [24]. A schematic image of this network is shown in Figure 1. At the input, the network is fed with samples of single power spectra in all electrode sites, lined up in a row. The output of the network encodes the class that has been recognized (the number of output elements equals the number of classes). The network is trained by one data set (training sample) and is tested by other data (control sample).

---

Figure 1. Scheme of an artificial neural network [23, 24]

---

The efficiency of classification was calculated as the index of correct recognitions (ICR) in the control data sample. The threshold of a truly non-random classification was calculated. Having two classes with 30 stereotypical tasks in each class and with a significance of 0.05 (conventional in biology), the threshold for a truly non-random classification is 65%. In other words, if the ICR is higher than 65%, then the probability of a random result is lower than 0.05. Turning to the psychophysiological interpretation of these results, we suggest that if the ICR exceeds 65%, then the existence of specific rhythmic patterns in specific mental states is considered proven.

We chose the Perceptron network because it is simple, reliable, and easy for interpretation.

Creating cognitive (emotional) spaces. The key point in forming a space of the recorded data is the right choice of metrics. Since we analyze EEG rhythmic patterns, the metric must be suitable to measure the distance between the patterns. A short distance would indicate that the rhythm patterns for the two compared mental states are similar, and a long distance — that they are different. In our studies, it was necessary to confirm or disprove the hypothesis, that similar psychological states generate similar patterns of EEG rhythms and, conversely, different mental states gave rise to strongly different patterns.

The method of calculating the distances [25, 26] is based on the assessment of the significance of the difference between the spectral data according to the Mann–Whitney U test for a statistical series of single power spectra. Then, the number of significant differences is normalized per the total number of spectral counts. The resulting index is a measure of the pattern differences (i.e. distance) ranging from 0 to 1.

After the distances between the patterns are calculated, their inter-relations were analyzed. To that end, we constructed a map, on which symbols depicting various patterns of EEG rhythms (and, accordingly, various mental states) were located on a plane so that the distances between the symbols reflected the experimentally measured indices of the pattern differences. This kind of problems can be solved using methods of multidimensional scaling. We used one of the simplest and most popular among them — the Sammon projection [27]. As a result, a “constellation” of mental states was visualized on the map.

Cognitive BCI. This technology is based on the hypothesis that in the process of continuously performed cognitive activity, the subject gets adjusted to this activity, and his/her EEG acquires a characteristic rhythmic pattern. By using biofeedback for training, the subject learns to maintain certain rhythm patterns so that his/her performance is expected to improve. In the below experiments, animation of task images was used for feedback: the subject was given a hint in case the real-time classifier detected the desired rhythm pattern in the EEG [28].

Cognovisor. Based on the accumulated data, a device prototype (cognovisor) was created; it allows for real-time monitoring of the thinking process and displaying it in the form of a “set point of consciousness” moving in an individual cognitive space. In cognovisor, the distances between the patterns of EEG rhythms are calculated using an artificial neural network.

The cognovisor prototype was tested in experiments with the presentation of eight types of cognitive tasks of two types — spatial and verbal. On the basis of the EEG pre-recording, an individual cognitive space of the given subject was built prior to applying the cognovisor. Then, the distance from the current EEG pattern to the patterns of the eight known mental states was calculated in real time. The set point was placed on the individual cognitive space map in accordance with the calculated distances. The technology is described in detail in [29].

Cognitive and emotional stimuli. In the experiments, numerous verbal-logical and spatial-imagery stimuli were presented to the subjects. We also used tasks for the mental math abilities. Most cognitive stimuli were presented visually on a computer screen; in other experiments, four audial cognitive stimuli equivalent to the visual ones were also presented. In Figure 2, two typical visual stimuli (verbal and spatial) are shown; the subjects successfully classified these stimuli into two categories. The first task was to choose a figure from three gray figures, which after stretching or compressing will add the white figure to a square. The second task was an anagram, in which the subject was tasked to guess the word by rearranging the letters.

---

Figure 2. Examples of spatial and verbal tasks: (a) complement the top white shape with one of the gray shapes to make a square; (b) solve the anagram [28]

---

The emotional stimuli were presented in various sensory modalities: olfactory, tactile, audial, and visual. In each sensory modality, there were several stimuli that differed from each other by the subjective degree of pleasantness — from pleasant to unpleasant. In Figure 3, some of the stimuli are shown.

---

Figure 3. Examples of emotionally significant stimuli The variety of the stimuli is described in two scales: the evolutionary antiquity of the sense organ (the horizontal scale) and the pleasantness — unpleasantness (the vertical scale). Forty stereotypical (pleasant and unpleasant) visual and auditory stimuli were presented for 8 s each, and the tactile and olfactory stimuli were presented 4 times for 30 s each. The visual stimuli were borrowed from the International Affective Picture System (IAPS) database, and the auditory stimuli — from the International Affective Digitized Sounds (IADS) database. In the Figure, the unpleasant visual stimulus is redacted for ethical reasons; yet it was presented unchanged to the subjects who gave their informed consent

---

The subjects. In each series of experiments, there were several dozens of subjects (25–35 on average, except for the first experiments in Japan, in which only 5 people took part). As a rule, the subjects were young university students, but middle-aged people participated as well. The ratio of men to women was about 2:1. In the 1992–1993 experiments, the subjects were all Japanese; in the later experiments, the subjects were citizens of the Russian Federation. Before starting the tests, all the examined persons were briefed on the experimental procedures and problems that might arise; the participants were then convinced that the procedure was completely safe. Before being tested, the subjects underwent a training course for solving the problems outside the test chamber and without setting the electrodes; during the training, they performed 5–20 tasks of each type. In the due time, the study was approved by the Ethics Committee of the Institute of Higher Nervous Activity and Neurophysiology; the subjects signed their informed consent forms.

We emphasize once again that despite the differences in methodology and the diversity in subjects’ ethnicity, the results of our many experiments are very close to each other — both quantitatively and qualitatively.

Results

Differences in averaged EEG power spectra. Differences in the rhythmic patterns of EEG, as a rule, are clearly visible from the averaged power spectra. The spectra themselves are highly individual, but for each given individual they sustain (apparently) for the life-time with very slow changes as evidenced from long-term experiments in the same subjects over 10–15 years.

A few signatures of the thinking patterns are typical for many subjects; for example, those shown in Figure 4. At the bottom, four types of tasks are depicted: 1) road junction: you need to drive from point O to one of the points — A, B or C, without breaking traffic rules; 2) logical judgment: based on the first statement, determine whether the second is true; 3) shape assembling: determine, which of the three shapes in the bottom can be assembled from the fragments shown above; 4) complex words: instead of points, insert letters that form the end of one word and the beginning of another. Obviously, assignments 1 and 3 are spatial, and 2 and 4 are verbal (the first ones are marked in green, the second — in red). The left panel of Figure 4 shows averaged EEG power spectra in a subject performing tasks 1 and 2, and the right panel shows tasks 3 and 4. A signature of spatial thinking can be clearly seen: it is the rhythm of ~11 Hz in the central and front leads (more on the right); for the verbal thinking, the rhythm with a frequency of ~8 Hz in the same leads (more on the left) is characteristic.

---

Figure 4. Averaged EEG power spectra of a subject performing four types of tasks Frequency range: 5 to 18 Hz; specific spectral power range: 0 to 18 µV2/Hz. See text for details

---

Recognition of the type of thinking from EEG spectra using an artificial neural network. In all our experiments, the ICR (with a classification into two or three classes) was much higher than the threshold of a truly non-random classification. In most cases, the average value for all tasks and all subjects was 87% (with the threshold of a truly non-random classification of 65%). Below, we report the specific values of the ICR obtained in various experimental series.

1. Brain Functions Laboratory, Japan, 1992–1993 [2]. The stimuli: spatial tasks (as in Figure 2 (a)) and tasks on mental arithmetic (addition of two multiplication products). The state of operative rest was also presented for recognition; thus, the classification was carried out in three classes. The average ICR was 90, 84, 96% for the three classes, respectively. The average ICR for the mental tasks was 87%. The rest state was best recognized by the occipital-parietal alpha rhythm.

2. Institute of Higher Nervous Activity and Neurophysiology, 1998 [30]. The stimuli: a logical judgement (as in Figure 4) and a spatial task (as in Figure 2 (a)). The average ICR was 84 and 91%, respectively, with an average value of 88%.

3. Institute of Higher Nervous Activity and Neurophysiology, 2007 [31]. The stimuli: eight types of tasks (Figure 5); of those, four — spatial tasks (including the “cube section” task, not mentioned above) and four verbal ones. The average ICR values for the group of subjects have the following meaning:

1) two specific types of tasks related to different types of thinking are recognized with an ICR of 86%;

2) two types of thinking (spatial vs verbal) with an ICR of 76%. In this experiment, the presented tasks were not those used in the training session. For example, the training included anagrams and complementary shapes, but the test included complex words and a road junction. With the ICR value exceeding the threshold of non-randomness, it can be argued that spatial and verbal thinking had common EEG rhythmic signatures, which are invariant for a specific type of mental activity;

3) a variety of tasks within one type of thinking can also be recognized: verbal vs verbal — with a reliability of 73%, and spatial vs spatial — 72%. This suggests that specific types of cognitive activity have their own specific signatures, manifested in specific EEG rhythm patterns although these individual signatures are minor as compared to the fundamental types of thinking.

---

Figure 5. Eight tasks of two types (four spatial and four verbal) and their ICR values See text for explanation

---

4. Institute of Higher Nervous Activity and Neurophysiology, 2015 [32]. The independent component analysis was used to determine the optimal set of components needed to produce the best possible classification. As a result, the ICR value increased from 87 to 89% (less than we expected).

The sensory modalities of the stimuli and the cognitive rhythmic patterns. For this experimentation, we developed 4 types of tasks to be presented in the visual mode and their 4 analogues for the audial presentation [33]. The list of tasks included: 1) a modified “cube section” task; 2) a simple planimetric task; 3) a search for one of four redundant words (the solution is not obvious, since some words have a double meaning); 4) composing a sentence with words beginning with the letters of the presented word.

Averaged power spectra were evaluated visually and analytically. According to the results, the EEG rhythmic patterns reflect the task content regardless of the way of presentation (audial or visual). Then, we concluded that the sensory modality of the stimulus had little impact on the EEG activity (Figure 6).

---

Figure 6. Averaged EEG power spectra of a subject performing a “cube section” task (green curve) and a “compose a sentence” task (red curve): (a) visual mode; (b) audial mode; frequency range: 5 to 20 Hz; the bars below the graphs denote the areas where one curve significantly differs from the other; the arrows mark the signatures of thinking patterns invariant for the two sensory modalities [33]

---

Creating cognitive space maps. In these studies (described in detail in [25, 26]), we aimed to show that the EEG rhythms patterns were not just specific for different types of cognitive activity, but also related to each other as much as did the psychological properties of the tasks.

For these experiments conducted in 30 healthy subjects aged 18 to 55, a group of special stimuli was developed; those differed by the spatial, figurative and verbal properties. A total of 6 types of stimuli were created, each contained 60 stereotypical tasks as follows (Figure 7): 1) a puzzle with crossing lines (insert the missing fragment from those shown below); 2) a puzzle with crossing word (same as 1, but with words instead of lines); 3) find a picture in the bottom row that is not related to any picture on the top; 4) the same, but some of the pictures are replaced by words; 5) find a specific word in the list of four that is not associated with any on the top; 6) find an abstract word that is not associated with any on the top. The “concrete words” denote well-imaginable objects. The “abstract words” denote general concepts.

---

Figure 7. Tasks with gradually changing degrees of spatiality, imagery, and verbality The tasks are described in detail in the text. The colors (according to the rainbow color sequence) encode the psychological character of the tasks: blue — spatiality, red — verbality, and green — imagery

---

The degree of spatiality decreases from tasks of the 1^st type to tasks of the 6^th type, and the degree of verbality, on the contrary, increases. In addition, tasks of the 3^rd type have a significant degree of figurativeness; tasks of the 4^th and 5^th types take a place between tasks of the 3^rd and 6^th types.

According to the method described above, a map of cognitive space was constructed. In addition to constructing a map based on electrophysiological data, we also asked a number of experts to comment about the stimuli used in this study. The experts — 20 professional psychologists — gave their assessment of the “spatiality”, “figurativeness”, and “verbality” using a 10-point scale. The technique is described in detail in [26].

Figure 8 presents two maps of cognitive spaces averaged over all subjects. The first map was obtained from the EEG rhythm analysis by averaging the individual maps. The second map presented in Figure 8 was obtained by averaging the scores suggested by the experts. The maps matched well: the vector correlation coefficient was 0.98. Therefore, we conclude that the space of rhythmic EEG patterns is isomorphic to the space of psychological characteristics of the types of cognitive activity during which these patterns are recorded.

---

Figure 8. Objectively and subjectively determined cognitive spaces Cognitive space maps obtained by expert opinions of psychologists (a) and by quantitative analysis of brain rhythms (b). Top — types of tasks (the same as in Figure 7, and with the same color codes)

---

Sensory-emotional space. Several studies demonstrated the dependence of EEG rhythms on emotions [34–36]. Along with that, in behavioral linguistic experiments, an association of the emotional valence of words with sensory modalities has been shown [37, 38]. According to the latter authors, this may be due to the fact that adjectives, which constitute the main part of the affective lexicon and describe various sensations, are closely related to the specific sensory experience.

Based on our studies in patients suffering from the oppression of consciousness, as well as studies in children presented with emotionally significant stimuli in different sensory modalities, we came to the following conclusions [39, 40]: 1) under suppressed consciousness, perception of strongly negative emotional stimuli by patients is improved as compared to neutral stimuli; accordingly, their impact on the EEG is more pronounced than that of neutral stimuli, which are often not perceived at all; 2) the presentation of stimuli via the ancient sensory organs (touch and smell) causes a greater response (both behavioral and electrophysiological); 3) in children, the stimuli sent via more ancient sensory organs, as well as more affective stimuli, cause a response at an earlier age.

We then addressed a question: whether opposite emotions associated with different sensory modalities are reflected in the EEG rhythms in an orderly and regular way? To get the answer, we constructed a mental space to analyze the EEG data obtained in the above experiments where emotional stimuli of different signs were presented in four sensory modalities (see Figure 3) [41].

The results are shown in Figure 9. Here, a map averaged over 20 subjects is shown. Two features emerged from the map: 1) the negative stimuli were located on the left of the positive ones; 2) the stimuli presented via the evolutionarily older organs of sense were located lower.

---

Figure 9. Map of the sensory-emotional space averaged over 20 subjects The horizontal axis of the space is a measure of emotions; the vertical axis indicates the evolutionary antiquity of the sense organ

---

Thus, we succeeded in constructing a two-dimension space map: the horizontal axis reflected the emotions and the vertical axis — the evolutionary age of the sensory modality. We termed the resulting space “sensory-emotional”.

Summarizing this section, based on the experiments with emotionally significant stimuli, we once again confirmed that the space of rhythmic EEG patterns was isomorphic to the mental space, in this case — sensory-emotional.

Rhythmic EEG patterns in schizophrenia. A mental illness impairs cognitive functions, which is especially noticeable in schizophrenia. Since, as we have shown, thinking is well reflected in the rhythmic EEG patterns, a question was asked whether mental abnormalities developing in schizophrenia would manifest in EEG rhythmic patterns.

To answer the question, a group of patients diagnosed with schizophrenia was examined with the help of the rhythmic EEG patterns recognition technique based on an artificial neural network [42–44]. The data were then compared with the results obtained in healthy subjects.

The patients were presented with the same tasks as were the healthy subjects, although in a simplified form. Anagrams were reduced to four letters; in the search for a complimentary fragment, the requirement to scale the shapes was omitted.

We found that in some types of schizophrenia, the ICR of mental states was significantly reduced as compared with that in healthy subjects. In patients with positive symptoms of schizophrenia, no decrease in the quality of the recognition with the artificial neural network was observed. These were patients with a single episode of the disease or with sporadic (once in a few years) acute attacks with a pronounced affective component. Along with that, the interictal period was characterized by excellent remission: the patients took a job and maintained social connections; also they were critical of their illness and compliant in terms of psychiatric care.

A decrease in the ICR was observed in patients with a severe form of the disease. These patients experienced frequent hospitalizations, incomplete remissions with residual symptoms, flattening of affect, and poor social adjustment.

Using a detailed statistical analysis, we then found that the ICR decrease was associated with a greater than usual variability of the EEG patterns characterizing a specific type of thinking. The increased instability of EEG patterns could be explained by an imbalanced salient mechanism: patients could not concentrate on tasks, their thinking was unstable, random, poorly focused on results, distracted by obsessive ideas and paradoxical associations.

Deformation of the sensory-emotional space in depressed consciousness. As mentioned above, by presenting emotionally significant stimuli in different sensory modalities to healthy subjects, a map of the sensory-emotional space can be constructed. This map reflects both the sensory aspect of perception and the emotional valence of the stimuli. However, when consciousness is depressed (for example, in patients with severe brain injury), the map has become modified so to solely reflect the physical parameters of the stimuli, but not their emotional content [45].

Cognitive BCI. In this study [28], we tested a technique aimed at improving the cognitive activity and accelerating the training process. In the control group, the subjects participated in three identical experiments (with an interval of several days between them) where they were presented with tasks, similar to those shown in Figure 2.

In the experimental group, the second experiment included feedback: after the rhythmic EEG patterns typical of a certain activity appeared, the subject was given a hint (for example, the figures in the spatial task were turned so that the problem became easier to solve). The reference patterns were obtained in the first experiment. In the third experiment, the behavioral parameters of cognitive activity (time and correctness of the solution) were assessed in both groups; in addition, the index of neural efficiency was assessed in accordance with the idea of the Austrian school of Klimesch [46].

In these experiments, we found that the feedback based on EEG rhythm patterns facilitated the learning process required for solving anagrams. In the experimental group, the anagram solution time was significantly reduced by 30%, while in the control group there was no significant change in the solution time. For spatial tasks, no behavioral effect was detected. An improved neural efficiency was found for both types of problems only in the experimental group.

Thus, we demonstrated the possibility of an effective cognitive BCI.

Cognovisor. The cognovisor [29] works in a two-step mode: preliminary and main. At the preliminary stage, the previously recorded EEG of a subject is used to build the cognitive space and calculate the “Perceptron” classifying weights. In the main stage, the cognovisor works to track the current mental state in real time and display it as a “set point of consciousness” on the map of an individual cognitive space.

Figure 10 schematically represents the work of the cognovisor with alternating tasks of 8 types. In these actions, the “set point of consciousness” is moving across the screen so to result in an animated picture. In Figure 10, this set point is marked with a cross in four consecutive positions related to solving different types of tasks.

---

Figure 10. How the cognovisor works The circles denote spatial tasks, the rhombi — verbal tasks; the symbol sizes reflect the degree of spatiality or verbality; below depicted the tasks corresponding to the symbols on the cognovisor panel

---

In this experiment, we observed an interesting phenomenon. Often, the set point comes to the correct cognitive state (corresponding to the task being solved), but then unexpectedly returns to the previous task for a short time, and then moves to the current one. This “behavior” correlates with the self-reports of the subjects, who say they sometimes return to the previous task in order to complete it, if they did not have time enough to complete it at the first attempt or they were not sure of the correct answer. Hence, this simple device prototype makes it possible to see a hidden feature of thinking.

Discussion

Brain rhythms — a substrate of consciousness. The above results, in our opinion, reflect the common phenomenon, namely: the brain rhythms do not just take part in the mental activity, but form its basis and can be called “a substrate of consciousness”. Accordingly, the rhythmic EEG patterns arising in the process of mental activity are highly specific, stabile (probably lifelong), and individual. Moreover, the rhythmic patterns do not only differ from each other in different mental states, they orderly inter-correlate so to form a space that is isomorphic to the space of the human psyche.

The role of patterns. The human psyche is a multicomponent and multifactorial phenomenon. Any thought or feeling is a result of many interacting neural processes. Therefore, things like cognitive tasks or various emotions generate complex patterns of brain activation. To identify such a complex mental process, it is hardly possible to rely on a single discriminating parameter of brain signals [4, 5].

Despite its complexity, such a multicomponent pattern by itself is a stable unit. The human psyche, for all its complexity, is a well-ordered mechanism, which operates according to well-defined laws and has a certain structure. Therefore, the mental state (cognitive or emotional) of a healthy person is a highly organized, stable, and individually adjusted set of processes that has been developed during human evolution and ontogenesis. This set is reflected in the specific pattern of brain signals. In mental diseases, this well-organized structure may be damaged, which can be seen from the discoordinated patterns of EEG signals, which become unstable and disconnected from each other.

Comparison with other studies on brain reading. Most of the reports cited in the Introduction described the recognition of perceived objects of various categories. In our work, complex cognitive and emotional processes are recognized. In our own studies, we interpret the obtained results as reflecting a common phenomenon. It’s worth noting that some of our studies were conducted prior to those published by others and quoted in the Introduction.

Possible practical applications of the techniques. With their further development, the cognovisor and cognitive BCI can be used for training, professional selection, diagnosis of mental disorders, and, as we believe, in rehabilitation of patients with brain diseases. Creating mental space maps can become a useful tool in diagnosing depression of consciousness and in predicting the course of diseases, as well as in monitoring the psychological status of children with developmental disabilities.

Problems to solve. There are two major unanswered questions regarding the described approaches: 1) What is behind the extremely high inter-person variability of cognitive and emotional patterns of EEG rhythms? 2) What are the neurophysiological mechanisms that generate the observed rhythms?

To answer the first question, we can suggest several possibilities.

1. The reason lies in the variability of skull and brain anatomy between different individuals. To us, this hypothesis seems the least plausible. Morphological differences could lead to a quantitative difference in individual patterns, but not to a completely different profile, which we actually observe.

2. The reason lies in the difference between various cognitive styles. This hypothesis seems more plausible. However, simple tasks would have to be performed by most people in the same or similar way. The difference in cognitive styles can, in our opinion, contribute to the variability of individual rhythmic EEG patterns, but cannot explain the huge difference we observed.

3. The mental procedures performed by the subjects while solving tasks are similar to each other, but these procedures are supported by different rhythmic processes. In other words, the rhythms that ensure the performance of the same mental functions can differ (for some reason not clear to us) in different people. These rhythmic mechanisms are formed in the process of individual intellectual and emotional development, but, having formed, remain unchanged for many years.

It should be noted that the brain activation patterns obtained in fMRI experiments show significantly less inter-person variability, and its explanation fits into the above assumptions 1 and 2.

When searching for the answer to the second question about the neurophysiological processes, we made assumptions and tested them. Thus, the central rhythm at a frequency of ~11 Hz (often arising with the performance of spatial tasks) can be interpreted as a mu rhythm reflecting the suppression of movement that unintentionally arises in spatial imagination. We tested this hypothesis by running a specially designed experiment [47]. Two different tasks were offered to the subjects: 1) solve a spatial problem; 2) initiate or suppress the hand movement shown on the screen (the Go-Nogo paradigm). In most subjects, this 11 Hz rhythm appeared in both tasks: spatial thinking and suppression of movement. This rhythm was identified by the independent component analysis in each person tested, and then its dipole source was found. The position of the dipole was identical under the two experimental conditions, which confirmed the initial hypothesis.

The frontal theta rhythm detectable in the spectra of many subjects (see, for example, Figure 6) performing visually presented cognitive tasks can be attributed to the reflex of orientation or the attention focused on the stimulus or the activation of short-term memory [48, 49].

For the central (mostly, right-sided) 8 Hz rhythm that often occurs in verbal and logical tasks we have no unambiguous explanation. Speculations on this issue are discussed in [23].

In some individual cases, we cannot explain part of clearly detectable stable rhythms.

Conclusion

From our studies conducted over the reviewed period, we can conclude that:

1. In the process of mental activity in humans, characteristic EEG rhythm patterns appear; those are unequivocally compatible with the nature of cognitive operations performed. These rhythm patterns are individually specific and persist over time. They form a stable electroencephalographic “portrait” of the individual.

2. The rhythmic patterns are inter-related with each other in a manner that can be identified by introducing the metrics on their space. Then, by analyzing the rhythms, one can create an EEG map of a person’s cognitive space.

3. A similar map can be constructed for the sensory-emotional space.

4. The cognitive patterns of brain rhythms become unstable in some mental disorders, especially in negative form of schizophrenia.

5. EEG maps reflecting the emotional space of an individual can be distorted by oppression of consciousness; then, they would reflect the physical qualities of the stimuli rather than their emotional content.

The obtained results can be used for the development of novel techniques, similar to the above mentioned cognitive BCI (which facilitates training for some types of activities) or the cognovisor — a device for visualizing the current cognitive state of a person.

Acknowledgements. The group of scientists who participated in this work included: the physicists M.S Atanov, I.V. Tarotin, and R.A. Naumov, the psychophysiologist A.O. Roik, the neurologist and neuropsychologist G.V. Portnova, the psychiatrist M.E. Baklushev, the software engineer O.D. Kashevarova. The author expresses his deep appreciation of their contribution.

Research funding. This study was partially supported by grants from the Russian Foundation for Basic Research No.15-04-04449 A, No.19-013-00925 A, as well as the program of the Presidium of the Russian Academy of Sciences 1.26P.

Conflict of interest. The authors confirm the absence of conflicts of interest that have to be reported.