Development of a Neurally-Controlled Vehicle — Neuro-Mobile — for Driving by Individuals with Motor Deficiency
The aims of the study were to develop a vehicle with elements of neural-control for people with limited mobility and to test algorithms for transforming human bioelectric activity into the driving commands.
Results. A running model of the first urban electric vehicle (neuro-mobile) with elements of neural-control and neuro-assistance intended for people with motor deficits and the elderly, has been created. The neuro-mobile is an original combination of the car body and asynchronous motor-wheels drives, able to accommodate the pilot on a wheelchair and a second person (passenger). It is equipped with a pilot assistance gear — a technical vision system, which reliably assesses the traffic situation by integrating the sensor data and recommends the correct driving path. Thus, the pilot is advised on possible driving scenarios (e.g., “changing lanes”, “staying in this lane”, “turning”, etc.) presented by the audiovisual information tools. Another component of the neuro-mobile is the neural-control system able to analyze various bio-signals, including the electroencephalogram (brain–computer interface) and electromyogram (neuromuscular interface). The signals reflect the pilot’s intention to choose one of the proposed movement options. A combined solution of the assisting electronic systems and the neuro-interfaces is transmitted to the drive control system (motor-wheel, steering wheel, brakes, etc.) to carry out the selected command. Thus, the proposed neuro-mobile may substantially increase the number of individuals with musculoskeletal deficiency, capable of moving with the road traffic.
The real robot pertains to the performance of the Pioneer 3DX robot; virtual object — performance of the virtual object; classification error — the value of classification error for all subjects (the dots represent the mean values, bars — standard errors); the results of each test were normalized to the first day of training
Introduction
The limited mobility resulted from diseases, road accidents, or surgery is an important medical and social problem that concerns millions of people. According to the Federal Registry of Disabled Persons, as of January 1, 2018, 12 million 111 thousand disabled people are officially registered in Russia. Among the typical persistent dysfunctions, disorders of neuromuscular, skeletal and movement-related functions occur most often. Only in 2017, the number of individuals for the first time diagnosed with disability caused by these disorders was 167,618.
According to the accepted classification of persistent disorders (directive No.1024n of the Ministry of Labor and Social Protection of the Russian Federation dated December 17, 2015), clinical signs of static-dynamic dysfunctions can result from diseases of 15 major types. The most substantial contribution to the overall statistics is made by disorders of the central nervous system caused by spinal injuries, tumors or vascular accidents of the spinal cord, neoplasms, vascular brain disorders, and demyelinating diseases. In most cases, spinal injuries result in disability, and every year the number of such individuals is rising.
In accordance with the International Classification of Functioning, Disability and Health (ICF, 2001), mobility is recognized a major component of person’s daily life. The concept of “mobility” is multifaceted, it includes 4 main categories: “changing and maintaining the body position”, “transferring, moving and manipulating the objects”, “walking and moving”, “moving by transportation”. Globally, people with limited mobility show a lower health status, lower educational attainment, lower economic activity, and higher poverty rates as compared with people with unrestricted mobility. This is due to a limited access to services that many of us are accustomed to, such as health care, education, employment, and transportation.
Therefore, social protection of individuals with disabilities and their rehabilitation are among the most important problems facing the society.
At the moment, the mobility of people with static-dynamic dysfunction is assisted by wheelchairs. There are many types of wheelchairs that allow moving at the expense of muscle strength, electrical energy, or alternative sources, but their capabilities are limited.
The technical means of rehabilitation existing in the Russian market provide the disabled person with means of limited movement within personal apartments or other residential facilities, medical institutions (hospitals, sanatoriums, etc.), educational organizations, etc. Such technical tools do not provide an ultimate solution to the problem of mobility in these people, since their movement in the street remains difficult.
Social integration of the disabled and a fundamental improvement in their quality of life requires a comprehensive approach that allows for a long-continued movement.
We sought a solution to this problem by developing an individual vehicle to provide for the movement of disabled people through the streets of a city and for long-time trips.
Materials and Methods
The present study is part of a bigger project run at the National Research Lobachevsky State University of Nizhny Novgorod. Here, a neurally-controlled vehicle for low-mobile individuals with dysfunctions of the extremities is under development. Although visually similar to regular cars, this vehicle is equipped with a series of unique scientific and technical innovations. Those include an original design, a neuro-mobile control system, and active assistance in driving. Notably, the neuro-mobile under development is a platform solution, which allows for subsequent individual selection of the most comfortable neural interface for a specific driver, while considering his/her personal needs and type of motor dysfunction.
The proposed control system assumes a part of human functions and analyzes the situation on the road; based on this analysis, the system issues recommendations for the pilot (assistance). Then the pilot creates a control action that is read by a multimodal recorder of brain signals and then transformed to commands (movement direction, speed etc.) received by the neuro-mobile actuators (motor-wheel drive, steering control, light signaling, doors, windows, multimedia, etc.).
Hardware and drive components of the neuro-mobile. In the automotive industry (including electric drive vehicles), the traditional platform design is based on a set of power units (including transmission, gearboxes), and constructive variations for different transport models.
In the development of this neuro-mobile, we are using the modular layout (Figure 1). The vehicle is designed as a capsule body (power cage with synthetic panels), mounted on a universal transmission-free platform. The key component of the platform is the electric asynchronous motor-wheels (AMW) unit, which controls the movement of the vehicle. For that purpose, the AMW performs control actions, including a direct control over the driver.
|
Figure 1. The modular design of the electric vehicle:
1 — capsule; 2 — platform; 3 — asynchronous motor-wheel |
Another integral component of the neuro-mobile is a set of sensors for monitoring the traffic situation in combination with an on-board computer that assimilates the high-level vehicle control commands with the hardware and software components of the power drive, braking systems, lighting, and climate control.
The layout of the body and the transmission-free platform provide, despite the small dimensions of the neuro-mobile, a capacious space that allows for accommodating not only the pilot on a wheelchair and the accompanying person, but also a powerful computer unit that activates and maintains the driver assistance system (with elements of autopilot) and the neural control systems.
Assistance to the neuro-mobile pilot. To assist the driver in monitoring the traffic situation, a set of critical sensors has been applied. These include:
TV cameras to provide information about the vehicle location on the road and details of the traffic situation;
a thermal imaging camera to detect pedestrians and other people in the dark, in rainy weather, etc;
laser range finders to accurately determine the distance to the objects and their speed, and also to obtain information about the free zone near the vehicle;
ultrasonic sensors to prevent collisions in low-speed maneuvering under jammed conditions (reversing, driving through narrow streets, parking).
To integrate data from different types of sensors, the neuro-mobile control system uses the following algorithm (Figure 2) that identifies the road objects and search for the best movement options.
|
Figure 2. Structured algorithm for collecting and processing information about road conditions:
IR — frontal infrared camera; C — set of |
To detect the objects, algorithms based on deep neural networks [1] are applied, and for object tracking, universal algorithms [2] and optical flow-based methods are used.
Neural-control over the vehicle. The system of neural-control — part of the neuro-mobile prototype under development — is trained to analyze bio-electrical signals. The signals are represented by brain electroencephalogram (EEG), the brain–computer interface (BCI), and electromyogram (EMG), i.e., the neuromuscular interface. As already noted, this list is not closed and can be modified according to the needs of individual drivers, considering their specific motor dysfunctions.
The brain–computer interface. The functioning of the brain is associated with the generation of signals, which can be detected by various sensors and are collectively used to build a BCI. The most convenient signal is the EEG recording, since it requires neither intervention into the brain (as in the case of microelectrodes implanted into the brain) nor a bulky expensive equipment (as for recording of a magnetoencephalogram or magnetic tomogram) [3, 4]. In the current model of the neuro-mobile, we used the brain EEG signals because of their feasibility and accessibility [5].
The brain–computer interface used in the circuit of the neural-control is based on the motor-imagination technology and includes the following major units: an EEG signal recording unit, an EEG signal classifier, and the software for transmitting the commands to the vehicle.
A certified NVX52 amplifier (Medical Computer Systems, Russia) is used to record EEG signals via seven leads (C5, C3, C1, Cz, C2, C4, C6) placed in accordance with the 10-10 international scheme. The reference electrode is located on the left ear lobe and the grounding electrode — on the forehead. The digitizing frequency is 500 Hz. The resistance under the electrodes does not exceed 10 kΩ. The signal is filtered in the range from 8 to 15 Hz with a 50-Hz Notch filter.
The studies involved 10 healthy subjects (6 men and 4 women) aged 18 to 23 years who gave written consent to participate in the experiment. The study was approved by the Bioethics Committee of the National Research Lobachevsky State University of Nizhny Novgorod.
The BCI-assisted driving included two successive sessions: a training session and a “driving” session. The training session was used to setup the BCI classifier. During the training session, the operator performs one of three instructions: to relax when a “cross” image appears on the monitor, to imagine the movement of the left or right hand when a “left” or “right” sign appears on the monitor. The operator is invited to choose any imaginary hand movement that is comfortable in his/her perception. For example, fingering or rotating the hand about the wrist joint. After receiving the “relax” instructions, the operator should sit quietly, concentrating on breathing. For each instruction, 10 attempts of 4 s each were allowed. The interval between the attempts was is 3 s (empty screen). The stimuli were presented in a random order. The classifier training session lasted 4 min. During the stimulus presentation, EEG signals were recorded using the NeoRec program (Medical Computer Systems, Russia) and then transmitted according to the LSL protocol. The transmitted signal was read by a script written in Python; the script controlled the synchronization between the stimulus presentation and the EEG signal.
Neuromuscular interface. Another modality used in the neurally-controlled vehicle is the electromyography (EMG) that reflects neuromuscular activity.
Due to the high information capacity of EMG signals, we paid a lot of attention to this modality and its role in the concept of the neural-control system. In earlier studies [6], we showed that the main factors limiting the use of the EMG interface were anatomical and physiological characteristics of the users. Therefore, a key task in the present study was to create and improve the skill of using an EMG interface for driving a mobile object.
The Pioneer 3DX robot (Adept MobileRobots, USA) and a virtual car in the arcade computer game were used as moving objects (Figure 3). The participants controlled the mobile object using movement of their hands.
|
Figure 3. Scheme of a training experiment using the EMG interface for driving control: (a) with the Pioneer 3DX mobile robot; (b) the wrist flexion was used to simulate the turn “left” command; the red spots show the muscles generating the highest EMG signal during this hand movement; (c) controlling the mobile virtual object in a computer game |
The EMG signals were recorded using a MYO bracelet (Thalmic Labs, Canada) with 8 electrodes that encircled the forearm (Figure 3 (b)). A wireless Bluetooth interface facilitated the connection between the bracelet and the computer where the EMG signals were processed. To quantify the muscular force, the mean absolute value (MAV) of the EMG signal was used [7]. Classification of EMG patterns was performed using an artificial neural network represented by a multilayer perceptron. For the input signal, the root mean square (RMS) value of the EMG signal calculated per channel was used. The averaging interval included 100 time points. The EMG control over the virtual object in the computer game was implemented with the MyoCursor application [8].
The study involved 18 participants aged 20 to 40 years. All subjects gave their written consents. They were divided into two groups, one of which was in control over the real robot, and the other — over the virtual object. The training sessions were carried out every other day for two weeks. The subjects performed a series of major and intermediate hand movements; simultaneously, EMG signals were recorded to adjust the classifier and assess its accuracy. Each movement was performed for 3 s and alternated with a 3 s relaxation period. The series of movements was repeated 4 times. After the classifier training procedure, the subjects performed the “driving” task within a predetermined time interval. The obtained data (time and classification error) for each training day were normalized to the first day for each user. Then the normalized data were averaged for all users.
Results
Hardware and drive components. We consider the development of the AMW unit achieved in this project as an important step in designing the basic hardware components of the transmission-free neuro-mobile.
The AMW was designed considering the operational characteristics of the future car. For example, special attention was paid to the search for a technical solution that would provide accessible service, e.g., wheel replacement in the event of a flat tire.
In Figure 4, the final version of the AMW design is depicted as a 3D model. To comply with the safety requirements, the design provides a duplication of the braking system. In other words, the braking system incorporates a dual-circuit: electromagnetic and hydraulic. When braking starts, AMW enters the recuperation mode, i.e. braking occurs. The controller, reading data from the AMW sensors (from the encoder disk and incremental sensor), regulates the braking mode and the power supply charging. In emergency braking or the need for braking at low speeds, the system switches from the recuperative braking regime to the opposite braking conditions, and the hydraulic braking mechanism is activated. If the electric part of the motor-wheel braking system fails and the driver cannot activate braking by a magnetic field, the system operates using the hydraulic brakes.
 Figure 4. The drive-based components of the AMW in the running model of neuro-mobile
|
Assistance to the neuro-mobile pilot. The proposed algorithm (see Figure 2) was tested in a number of experiments using the real data related to traffic signs. For this purpose, the following experiments with increasing number of signs (classes) were conducted:
1) one class: “pedestrian crossing” (index 5.19.1 in accordance with the driving regulations);
2) two classes: “pedestrian crossing” and “main road” (5.19.1; 2.1 in accordance with the driving regulations);
3) three classes: “pedestrian crossing”, “main road”, and “sign” (5.19.1; 2.1); the term “sign” denoted all other classes in the sample;
4) one class: “sign”.
The results are presented in Table 1.
|
Table 1. Summary of the experimental results |
As can be seen from the table, the average accuracy for “narrow” classes 5.19.1 (“pedestrian crossing”) and 2.1 (“main road”) is quite high, while in the superclass “sign” it does not exceed 85%. In experiment 3 with a sample containing both “narrow” classes and the superclass, the average accuracy for all classes is lower than in experiment 2 with no superclass — due to the influence of the superclass accuracy value on the overall result. At the same time, the accuracy of the “narrow” classes did not change significantly.
Vehicle neural-control system
Brain–computer interface. The EEG signals were processed using a classifier based on the linear discriminant analysis, where the classification characteristics were selected by pairs using the spatial filtration method (CSP) [9] for all types of commands.
After a classifier training session, the operator could start “driving” the neuro-mobile. The operator was asked to successively choose one of the three commands, at his/her discretion (e.g., imagine a movement of the left/right hand or a rest). Every 2 s, the classifier analyzed the EEG record, made a conclusion and transmitted the command selected by the operator to the external device.
Based on the results of the operator’s work, the average accuracy of his/her driving control (assisted by this BCI) was calculated and expressed as the ratio of the number of correctly entered commands to the total number of attempts. In each session, the operator tried 10 attempts for each command. In 10 volunteers, the average accuracy of the BCI-assisted control for the three commands was 72%. Notably, when testing the classifier, the individual results on accuracy were well predictive of the subsequent estimates of driving accuracy (correlation coefficient = 0.8). We also noted that about a third of the participants confidently achieved a fairly high accuracy (90% and higher).
Neuromuscular interface. Figure 5 presents the results of two-week workouts on EMG-controlled “driving” in 18 subjects. Both with the physical robot and with the virtual object in the computer game, we observed a gradual decrease in the time required to complete the test. On average, by the end of the training period, the time of passing the “snake” test by the Pioneer 3DX robot was reduced by 60%, and with the game test — by 50%. In both cases, the performance improved with each next training day and reflected the learning effect in a linear manner. The effect of training could not be explained by changes in the “quality” of the EMG patterns, which was also confirmed by the classification error data. On the first training day, the error mean values were slightly but insignificantly higher than those on the next day. Therefore, the observed decrease in the test time wasn’t due to changes in the hand movements used to learn the classifier, but was due to another factor.
|
Figure 5. The EMG-controlled driving at different days of training The real robot pertains to the performance of the Pioneer 3DX robot; virtual object — performance of the virtual object; classification error — the value of classification error for all subjects (the dots represent the mean values, bars — standard errors); the results of each test were normalized to the first day of training |
We got the clue on the nature of this factor by comparing the paths of motion of the controlled virtual object on the first and last training days (Figure 6). In a typical example (Figure 6 (a)), the first day had a zigzag path, meaning that the EMG patterns and hand movements were incorrect in the context of the current task. Over the time of training, the number of these incorrect movements decreased, and the driving paths became optimized. Figure 6 (b) shows the portions of optimal vs incorrect movements averaged over all subjects. After training, the portion of the incorrect movements decreased, and for the movements following the “down” and “left” commands, the decrease was statistically significant. Accordingly, the portion of the optimal movements increased, and it was statistically significant for movements following the “left” and “right” commands.
|
Figure 6. Driving paths of the controlled virtual object:
(a) the driving paths on the first and last day of training; user — paths of the object controlled by the participant; best — the optimal direction computed by the Simple Rule algorithm; target — the path of the moving target; (b) the portions of incorrect and optimal movements averaged over all subjects on the first and last day of training; asterisks indicate statistically significant differences |
The results suggest that short-term training can improve the “correctness” of the movements that control a mobile object in dynamically changing conditions.
Discussion
We consider the developed unique body structure and transmission-free AMW-based drive to be a significant step towards creating a city car for the disabled. Firstly, within its small dimensions, the neuro-mobile has an impressive capacity for accommodating the driver in a wheelchair, the accompanying person, the batteries, and the powerful computer center. Secondly, the system controlling the actuators (including the motor-wheels) is fully compatible with the active assistance and neural control. The developed neurally-controlled piloting system allows driving in a slow (urban) mode. We demonstrated an effective control of the mobile object by using signals of different modalities (either EEG or EMG signals). In further research, we plan to study the feasibility of using these algorithms for simultaneous control by various modalities.
In this study, a neurally-controlled training system for a virtual neuro-mobile simulator has been developed. The training takes several months and can be modified in accordance with the type of motor dysfunction in the given individual.
The technical characteristics of the proposed vehicle are similar to those reported elsewhere (Table 2). The materialization of such a neuro-mobile will allow social rehabilitation of the disabled and people with movement dysfunctions. These developments will provide them with an independent comfortable vehicle and thus improve their quality of life.
|
Table 2. Main technical characteristics of this neuro-mobile in comparison with those reported by others |
Conclusion
The development of a self-used vehicle (neuro-mobile) for people with motor dysfunctions (wheelchair users, the elderly) is described. The car incorporates active driver assistance based on a technical vision system and a multimodal neurointerface, including a basic brain–computer interface and a neuromuscular interface.
Research funding. This work was supported by the Ministry of Education and Science of the Russian Federation within the Federal Program “Priority Research and Development Areas for the Development of the Scientific and Technological Complex in Russia for 2014–2020”. Grant approval No.14.581.21.0022 of 03.10.2017 (the unique project identifier — RFMEFI58117X0022).
Conflict of interest. The authors have no conflicts of interest.
References
- Orlova S.R., Bakhshiev A.V. Raspoznavanie dorozhnykh znakov na baze glubokikh neyronnykh setey. V kn.: Ekstremal’naya robototekhnika i konversionnye tendentsii [Road sign recognition using deep neural networks. In: Extreme robotics and conversion tendencies]. Saint Petersburg; 2018; p. 451–459.
- Nebehay G. Robust object tracking based on tracking-learning-detection [dissertation]. TU Wien; 2012.
- Lebedev M.A., Nicolelis M.A. Brain-machine interfaces: past, present and future. Trends Neurosci 2006; 29(9): 536–546, https://doi.org/10.1016/j.tins.2006.07.004 .
- McFarland D.J., Wolpaw J.R. Brain-computer interfaces for communication and control. Commun ACM 2011; 54(5): 60–66, https://doi.org/10.1145/1941487.1941506.
- Gordleeva S.Yu., Lukoyanov M.V., Mineev S.A., Khoruzhko M.A., Mironov V.I., Kaplan A.Ya., Kazantsev V.B. Exoskeleton control system based on motor-imaginary brain–computer interface. Sovremennye tehnologii v medicine 2017; 9(3): 31–38, https://doi.org/10.17691/stm2017.9.3.04.
- Lobov S., Krilova N., Kastalskiy I., Kazantsev V., Makarov V.A. Latent factors limiting the performance of sEMG-interfaces. Sensors 2018; 18(4): 1122, https://doi.org/10. 3 390/s18041122.
- Lobov S.А., Mironov V.I., Kastalskiy I.А., Kazantsev V.B. Combined use of command-proportional control of external robotic devices based on electromyography signals. Sovremennye tehnologii v medicine 2015; 7(4): 30–38, https://doi.org/10.17691/stm2015.7.4.04.
- Lobov S., Krilova N., Kastalskiy, I., Kazantsev V., Makarov V. A human-computer interface based on electromyography command-proportional control. In: Proceedings of the 4th International Congress on Neurotechnology, Electronics and Informatics — Volume 1: NEUROTECHNIX. Portugal; 2016; p. 57–64, https://doi.org/10.5220/0006033300570064.
- Koles Z.J., Lazar M.S., Zhou S.Z. Spatial patterns underlying population differences in the background EEG. Brain Topogr 1990; 2(4): 275–284, https://doi.org/10.1007/bf01129656.
