|Figure 1. Brain-Computer Interface Device|
|Diodato, B. (2008) Brain-computer Interface Device. [Photo]
Retrieved March 28, 2013 from http://ricelander.wordpress.com/
Synesthesia is a neurological phenomenon where an individual experiences simultaneous stimulation of two different sensory pathways by only one stimulus. Today, scientists have used synesthetic concepts to develop new technologies that converts one sensory input into a different sensory output. One of the main purposes for synesthetic technology is sensory prosthesis where a severely damaged sense is replaced by a functional one. An example would be reading Braille, which is a form of writing language using patterns of raised dots, so that the blind can “read” the words with their fingers instead of their eyes. There are also synesthesia technology that enhances certain sensory functions such as detection of toxins in chemical workplaces, gaming control, and medical diagnosis of certain diseases. A major research subfield is the Brain Computer Interface (BCI) technology that links the human brain to a computerized device. An example of such a device is the BrainPort System developed by the Wicab Lab in Wisconsin, USA. It uses a camera to pick up images of one’s surroundings and an Intra-Oral Device (IOD) to convert it for transfer to the brain via the stimulation of the tongue. Another synesthetic technology is the Optoelectronic Nose that a lab from the chemistry department at University of Illinois created. The Optoelectronic Nose transfigures olfactory input to visual output using a colorimetric sensor array with chemically responsive dyes that changes color with changes in their chemical environment.
Table of Contents
1. Brain-Computer Interface System
A brain-computer interface (BCI) is a system that combines software and hardware communication devices so that cerebral activity alone can utilize them to control external gadgets such as computers. The main goal of BCIs is to allow for communication functions in disabled people who could not express themselves in a normal fashion due to paralysis, spinal cord injuries, or other neuromuscular disorders. A BCI recognizes a set of brain signal patterns using five stages: signal acquisition, preprocessing (or signal enhancement), feature extraction, classification, and control interface.
|Figure 2. Brain-Computer Interface Process|
|(2006) Multi-stage procedure for on-line BCI. [Digital]
Retrieved March 28, 2013 from http://www.brain.riken.jp/bsi-news/bsinews34/no34/research1e.html
Signal acquisition stage receives input from brain signals and may additionally execute artifact processing and noise reduction. The preprocessing stage converts the signals into a more suitable format for further processing. The feature extraction stage extracts target information from the recorded signals where it is then mapped onto a vector with distinct characteristics from the observed signals. At the classification stage, the characteristics of the vectors are used to classify the signals, thus allowing for pattern recognition so that the BCI machine can interpret the user’s objective. The control interface stage is where the classified signals transformed into directions for the apparatus, e.g. a wheelchair.
1.2.1. Signal Acquisition
To receive the user’s intentions via brain signals, CBIs collects measured brain activity via tractable electrical signals during the recording stage. Brain activity is usually determined either electrophysiologically or hemodynamically.
In the electrophysiological monitoring, brain activity is measured by amount of ionic current flow changes between neurons during the info exchange process by electrochemical transmitters. This can be done with electrical signal acquisition procedures such as electroencephalography (EEG) or magnetoencephalography(MEG). In hemodynamic monitoring, the hemodynamic response is exploited, where glucose is released more often to areas with active neurons than those with inactive neurons. The discernible difference in oxyhemoglobin to deoxyhemoglobin ratio are calculated by neuroimaging methods like functional magnetic resonance and near infrared spectroscopy (NIRS). Other possible neuroimaging techniques that can be used for this stage are listed in Table 1 with their functional properties.
1.2.2. Preprocessing/Signal Enhancement
Brain signal patterns have been translated in relation to cognitive tasks. Thus, using these translations, BCI systems can decipher the user’s goals. By manipulating these brain signals, people can express their intent to the BCI system. These brain signals can act as control signals in BCI units. Today, a variety of brain signal groups are studied to investigate their potential as a control signal in BCI systems. Table 2 lists some currently employed control signals in BCI systems.
1.2.3. Feature Extraction
BCIs recognizes and classifies different brain signaling patterns into a unique group based on its traits. Brain signals are evaluated at multiple channels and undergo dimension reduction techniques to remove extraneous info, thereby reducing computational costs as well. Table 3 lists some feature extraction methods and their properties.
|Table 4. Feature Extraction Methods - Part 2|
|Adapted from “Brain Computer Interfaces, a Review,” by L.F. Nicolas-Alonso and J. Gomez-Gil, 2012, Sensors, 12, pp. 1211-1279.
AR: AutoRegressive Components; MF: Matched Filtering; WT: Wavelet Transform
The neurological phenomenon of brain activity can be affected by artifacts (unwanted signals that influence brain activity), which could attenuate the BCI system’s performance. Two major types are physiological and non-physiological or technical artifacts. Physiological artifacts may come from other body organ activities such as the heart [electrocardiography (ECG) artifacts], bodily muscles [electromusculography (EMG) artifacts] and the eye [electrooculography (EOG) artifacts]. To avoid such artifacts, patients are asked to refrain from blinking or any body movements , or signal samples where ocular or muscular activity is detected can be removed via observation of EOG and EMG signals. A more effective method of artifact handling would be using artifact removal algorithms in EEG such as linear filtering, linear combination and regression, which finds artifacts and excises them, keeping the brain activity intact instead of discarding samples. Technical artifacts are usually due to power-line noises or electrode disruptions, but can be solved with good filtering or shielding.
This is the stage where the BCI attempts to learn what the user wants by the characteristic vector that distinguishes different brain activity. Two types of algorithms are currently in use: regression and classification. Regression algorithms directly use EEG signal features as independent variables in calculating the user’s objective. In classification algorithms, the EEG signal features are employed to outline the perimeters between two targets in the target space.
BCI allows for new methods of control and communication sans use of peripheral muscles and nerves. By restoring lost function in those with severe motor deficits, the BCI system indirectly helps alleviate financial burden of caregivers and other medical costs. Generally, BCI target users have three populations: i) Complete Lock-In State (CLIS) users who have no motor control, ii) Locked-In State (LIS) users who are partially paralyzed with residual control over movement, and iii) healthy users with good motor control. In figure 3, Kübler and Birbaumer (2008) showed a strong negative correlation between BCI performance and the degree physical deficiency. I.e., the more physically impaired the user is, the less uses the user has for the BCI.
Scientists have been able to develop communication techniques using BCI technology, such as the virtual keyboard where the user modulates his/her EEG through mental imagery of hand and leg movement to “type” out words on the screen with a cursor.
Not only can it allow for communication, BCIs can also return motor function, allowing for restoration of normal daily function and may even eliminate psychological and social distress. An example of such a BCI would be the functional electrical stimulation (FES) that stimulation artificial muscle contractions by electrical currents that generate action potentials in periperhal motor nerves in the target muscle.
Currently, researchers have also been interested in the positive applications of BCIs where they act to enrich senses instead of recovering lost functions. BCI can produce a new interaction medium in video games (which can in turn potentially enhance cognitive functions) or other interactive programs, permitting a new set of experiences and challenges. Also, researchers can use data from BCI to retrieve uncontaminated responses of test subjects' behaviours during trials.
2. BrainPort System
To be more specific, the Brainport System (BPS) should be called a Computer-Brain Interface (CBI) since it transfers information from the computer to the brain, unlike the usual BCI direction of information flow from brain to computer. An example of a CBI device would be the cochlear implant. The BPS project was initiated by Paul Bach-y-Rita and its development was continued by the Wicab, Inc. company. It was originally made to help people with balance disorders such as bilateral vestibular dysfunction, but now the BPS has gained more purposes. Being a platform technology, the BPS can act as the basis for many new applications involving other systems, technologies, and devices. Useful info is delivered to the brain by the tongue using electrotactile stimulation conducted by the BPS. The BPS’ goals are to provide absent sensory signals and to reduce the probability of sensory overload.
2.2. The BrainPort Mechanism
|Figure 4. Schematic Outline of the BrainPort System|
|Adapted from “Brainport: an alternative input to the brain,” by Danilov, Y. & Tyler, M., 2005, Journal of Integrative Neuroscience, 4, 537-550.|
With respect to the BrainPort Balance Device, the system includes an Intra-Oral Device (IOD) and a controller that contains a microprocessor with drive electronics, user controls, safety circuits, and battery power supply. The IOD consists a Micro-Electro-Mechanical Systems (MEMS) accelerometer to sense motion and head tilt. A wireless radio frequency (RF) link is used to conduct information transfer to external devices.
The IOD with the proper circuitry transfigures the signals into monophasic, pulsed stimuli onto a round tongue display via electrodes. The tongue display is a polytetrafluoroethylene circuit board that acts as a palate for tongue array. Surface electrode array sends qualitative and quantitative info on the tongue by electrically stimulation that forms an “electrotactile screen” which represents an image in real time with different degrees of complexity. The tongue projects a “tactile image” to the brain that can decode this info by its spatial, temporal, intensive and qualitative features. This translation can help settle an urgent need and resolve signal detection and recognition problems. Although not completely understood, scientists have suggested neuroplasticity as the reason to how one sense can be interpreted as another in the brain. Some theories on such neuroplasticity (changes in neuronal functional representation) have been discussed in Why Synesthesia Occurs neurowiki page.
|Video 1. BrainPort Vision System|
|(2011, June 12). Electro-Medicine : Biological Physics - Cure for Blindness : BrainPort Vision Through Tongue
[Video file] Retrieved from http://www.youtube.com/watch?v=MegGOFkO-Sg
2.3. Electrotactile Stimulation in the Human Brain
Many experiments since 1970 have shown that electrotactile stimulation are effective sensory substitution systems and have hypothesized that cross-modal plasticity is what causes this synesthesia-like effect. For example, when the blind uses the Braille reading method, their primary and secondary visual cortical areas are activated. After BrainPort training, the primary visual cortex that was dormant before was shown to be strongly active, suggesting that the training aroused the more analytical and complex components of the visual cortex for mechanosensitive tactile information analysis. With positron emission tomography, PET, it was demonstrated how the congenitally blind can train the tongue to stimulate the visual cortex using somatosensory information. Figure 5 shows how much more active the occipital cortex were in blind subjects (row A) compared to control (blindfolded individuals with normal visual acuity).
|Figure 6. The four classes of BrainPort Applications|
|Adapted from “Brainport: an alternative input to the brain,” by Danilov, Y. & Tyler, M., 2005, Journal of Integrative Neuroscience, 4, 537-550.|
Using different resolution grids, the BPS can noninvasively transfer a variety of qualitative and quantitative data. By combining low or high resolution with qualitative or quantitative info, four possible application classes have been proposed, each with a unique platform for further advancement into other possible functions. It is valuable to note that as the resolution increases of a BPS, it’s sensitivity decreases.
2.4.1. First Class: Qualitative Info, Low Resolution
In this class, the BrainPort detects specific environmental changes (e.g. temperature, chemical), and reports using a “Yes/No” paradigm to alert the operator of any toxins for example. Research has already implemented it's expansion into a source orientation system, which would allow for identification of a target and permit response to simple situations, like catching a ball.
2.4.2. Second Class: Qualitative Info, High Resolution
With higher resolution, a more precise, clear image of surroundings can be delivered to the operator, thus allowing for locomotion since one can amend the trajectory to their destination in the case of obstacles. Signal feedback has also been incorporated to help with navigation using the BPS.
2.4.3. Third Class: Quantitative Info, Low Resolution
An example of a third class BPS would be the BrainPort Balance Device with a MEMS accelerometer that informs the brain of any deviations of the head position from a reference point so that the brain would know when and how to readjust the body posture to not lose balance. This balance device has already shown positive results in treating bilateral vestibular dysfunction.
2.4.4. Fourth Class: Quantitative Info, High Resolution
The BrainPort Vision system is a good representation of a fourth class BPS. Signals captured from a video camera are transfigured onto a microprocessor where it then gets sent to the Intra-Oral Unit and subsequently the tongue as a real-time electrotactile image. This quantitative, high resolution processing can be used to perform visual detection and recognition tasks, like picking up a newspaper. Besides restoring eyesight, this class can also be used as an enhancement tool where the BrainPort Vision system can be adapted for night vision using infrared light or for ultraviolet vision.
3. Related Links
Cognitive Benefits of Video Games
Subcortical Visual Prosthetics
Why Synesthesia Occurs