Auditory Visual Synesthesia

Introduction

Auditory Visual Synesthesia
Image Unavailable
Auditory visual synesthesia allows synesthetes to experience
anomalous sensation in which sounds, chords, or pitch could elicit
different visual experiences [23] (image adapted from
http://lofalexandria.blogspot.ca/2012/12/synesthesia.html)

Auditory visual synesthesia is a neurological phenomenon in which a sound stimulus can elicit visual experiences [23]. Unlike auditory-induced visual image experienced by non-synesthetic population, auditory visual synesthesia is involuntary and consistent[7]. Auditory visual synesthesia is further categorized by the type of audio stimuli paired with specific visual responses such as sound colour synesthesia and coloured music synesthesia[7]. Previous studies stated that auditory visual synesthesia could be the consequence of excessive multimodal integration [16]. Recent studies involving the double flash illusion, however, have observed reduced multimodal integration [22]. Recent researches also discovered the anomalous activation of the cortical areas such as V4 and parietal lobe that could lead to potential auditory visual synesthetic experiences [23]. The study of synesthesia is important as it is a rare neurological condition, and understanding of synesthesia can provide insight regarding to the mechanisms of perception, cognition and communication between cortical regions [7].

This video illustrates the synesthetic experiences by a synesthete, Jane Mackay

1. Cortical Activities on Auditory Visual Synesthesia

Many research papers focused on one type of synesthesia called grapheme- colour synesthesia. In those articles, they found evidences of the involvement of parietal lobe and V4 in grapheme-colour synesthesia[4][19]. It is possible that similar cortical activities could occur in auditory visual synesthesia because both types of synesthesia are involved with involuntary control of visual synesthetic experiences [7]. The following pivotal points will give further details of the involvement of parietal lobe and V4 region in auditory visual synesthesia.

1.1 Parietal and Auditory Hub observed via EEG

Parietal lobe was suggested to be essential in creating synesthetic experiences [15] [26]. Therefore, it is crucial to explore this region for its role in auditory visual synestheisa. Different imaging techniques have been used to observe the cortical activities such as electroencephalographic (EEG) technique. Langer and Jake (2011) applied this technique on 12 auditory visual synesthetes at a resting state. Four different frequency bands (alpha 1, alpha 2, beta, and theta) were employed in observing the cortical activities [18]. Resting EEG activity manifests stronger hubs (strong connection between regions) occurring in the left parietal lobe (especially in the angular gyrus and the region that is anterior to the postcentral gryus) and the right side of auditory cortex in auditory visual synesthetes when compared to non-synthestes[18]. This result is consistent with other research [23].

The strong parietal lobe activities observed in auditory visual synesthetes leads to the implication that parietal lobe could be essential in generating synesthetic perceptions [18]. The hyperactivity observed at rest, strongly suggests that there could be a potential differences in functional brain network that is specific to auditory visual synesthestes [18].

A stronger alpha2 frequency band was observed in relation to the resting activity in the right auditory cortex [18]. The right auditory cortex is essential in processing pitch information [13]; therefore, this could be the potential cause in the lateralization observed in the auditory cortex [18]. The strong auditory hub observed could be playing a pivotal role in supporting auditory visual synesthesia [18].

1.2 Functional MRI investigation: Inferior Parietal Cortex

Functional magnetic resonance imaging (fMRI) was used to trace the cortical activities in auditory visual synesthetes under a non-linguistic sound test (chords and tones)[23]. Furthermore, this study shares similar result acquired from the resting EEG experiment, mentioned previously. Both results pointed to the potential involvement of parietal cortex in auditory visual synaesthesia [18] [23]. Nonetheless, a more precise result was obtained from the fMRI study under an active brain. The result indicates that there was hyperactivity that occurred in the left inferior parietal cortex (IPC) compared to non synesthetes [23].

It is suggested that functions of parietal lobe include multimodal integration (such as integration between auditory and visual) and spatio-dynamic processing [5]. Therefore, the hyperactivity on the left side of inferior parietal cortex could be the consequence of excessive multimodal integration (hyperbinding) [18]. This excessive integration between the two modalities: inducer region (such as auditory) and concurrent region (such as visual) in the parietal lobe through disinhibited feedback is thought as the underlying mechanism in synesthesia[25] [11]. Nonetheless, opposing results are presented by double flash illusion and McGurk illusion experiments [22] [27]. This will be further discussed in the next section.

Figure 1. Cortical Activation in Auditory Visual Synesthesia.
Image Unavailable
(A) This image indicates a region with significant activation in the IPC upon auditory stimulations
in the brain of synesthetes. (B) Moreover, two regions at the left and right temporal lobe showed increase
in activation in synesthetes during auditory stimulations. (Image adapted from Neufeld, et al., 2012 [23])

1.3 V4 in Fusiform Gyrus

The V4 region is involved in processing information such as colour and object properties, making it the best candidate in causing synesthetic perceptions experienced by auditory visual or grapheme- colour synesthetes [3] [10][12]. Moreover, many neuro-imaging studies provide evidences on the anomalous activities in colour area V4 regions for grapheme- colour synesthesisa. These evidences support the involvement of V4 in synesthetic perceptions [4][28].  However, such results were not observed in the resting EEG and fMRI studies presented earlier [18] [23]. The V4 region in auditory visual synesthetes showed no significant activation at rest (in a resting EEG study) nor did it show any anomalous activities in an active brain (under non-linguistic sound fMRI study) [18] [23].

The absence of V4 activation in those two studies, however, cannot be used against in its possible involvement in auditory visual synesthesia [14]. It is possible that different forms of synesthesia might require different involvements of visual regions [23].  Also, other associative visual regions such as lingual gyrus [24] and inferior temporal cortex [1] have been observed with significant in activation in synesthetes. These associative regions could be the potential recruitment in the production of synesthetic experiences [23]. 

2. Controversy on Level of Auditory and Visual Integration in Auditory Visual Synesthesia

Many recent publications stated that the occurrence of synesthesia could be the consequence of enhancement in non-synthesthetic multimodal integration (such as auditory and visual integration) [9] [21]. This hypothesis is supported by the hyperactivity observed in the parietal lobe presented in early sections [18] [23]. Reduction in multimodal integration was reported from recent experiments on auditory visual synesthesia, yet research done on music-colour synesthetes stated otherwise [23][27][29]. Therefore, the auditory and visual integration mechanism regarding to auditory visual synesthesia remains to be controversial.

2.1 Double Flash Illusion Experiment

Figure 2: Data Information Regarding to Double Flash Illusion Experiment
Image Unavailable
The dotted line represents the non-synethestes
in which illusionary flash was reported at the SOA (stimulus onset asynchrony) of -150 to +150ms
while a narrower SOA of -100 to +100ms illustrated by
a solid line represents the illusionary flash experienced by synesthetes. This figure also
demonstrates the reduction in reported illusonary flash perceived by synesthetes
compared to non-synesthetes.(image adopted from Neufeld, et al., 2012 [21])

Double flash illusion is an illusion elicited by a single light flash coupled with multiple beep sounds, thus multiple flashes could be perceived by individuals [22]. Hence, if the increment of illusions perceived by the auditory visual synesthetes occurs, this will allude to the general enhancement of auditory and visual integration [22]. Moreover, the variance in separating the auditory and visual inducers presented in the double flash illusion test could be used to test the temporal binding window of auditory visual integration [22]. It was hypothesized that synesthetes would perceive more illusionary flashes with widened temporal binding window of auditory-visual integration, if higher general multimodal integration mechanism occurs in synesthetes [22]. However, opposing result was obtained from the double flash illusion test. It was indicated that synesthetes perceived less illusions with a narrow temporal binding window compared to non-synesthetes[22].

These results suggested that synesthetes have less general auditory and visual integration compared to non-synesthetes [22]. Thus, this opposes to the hypothesis which states the excessive multimodal integration could contribute to synesthesia [9] [21]. One of the postulations for the decrease in illusions perceived in synesthetes is their distractor-filtering ability [22]. Synesthetes are better trained in focusing on relevant inducers in order to compensate for the distractive synesthestic perceptions; therefore, the production of multisensory perceptions along with reduction in general multimodal integration was observed [22].

2.2 McGurk Illusion and Speech Comprehension Test

Professor Lawrence Rosenblum from university of California explains and demonstrates the effect of McGurk Illusion.

Apart from the double flash illusion, McGurk illusion and comprehension test were employed in exploring the excessive multimodal binding in auditory visual synesthetes. The McGurk illusion was stimulated when formation of new perception was caused by the fusion of different auditory and visual information [27]. McGurk illusion was used in determining the strength in auditory and visual integration, it also predicted that synesthetes would be more susceptible to this test (this is because more fusion would occur from auditory and visual information, if there was excessive integration between the two modalities) [27]. As for the speech comprehension test, its objective was to observe rather or not synesthetes could profit more in speech perception while observing articulatory movements in a noisy environment [27]. Analogous to the double flash illusion experiment, synesthetes showed reduced in number of illusions and profited less in speech perception during the speech comprehension test compared to non-synesthetes [27].

Results acquired from double flash illusion, McGurk illusion, and speech comprehension test all pointed to the general reduction in auditory and visual integration [22] [27]. Thus, it is possible that the synesthetic experiences are the result from the reduced in multimodal integration [27]. This statement could be supported by the way how children learn to write and read. As children are gaining the skills of reading and writing, the transmission of information form auditory to visual region becomes essential [27]. Therefore, it may be necessary to offset such multimodal integration insufficiency by generating the multisensory perceptions experienced by synesthetes [27]. Moreover, the enhanced distracter-filtering ability in synesthetes enables synesthetes to perceive less illusions in McGurk and double flash illusion and to profit less from the speech comprehension test [22] [27].

2.3 Coloured Music: Enhanced Structural Activity

Figure 3: Different Images of The IFOF Tracts
Image Unavailable
This shows the IFOF tracts in the brain of
synesthetes (top) and non-synesthetes (bottom) in three different
views (sagittal, coronal, and axial) (image adapted from zamm et al., 2013 [29])

Coloured music synesthesia is a specific type of auditory visual synesthesia in which musical sounds like chords or tones induce synesthetic perceptions [8]. Aside from the double flash and McGurk illusion experiments, a neuroanatomical approach was used in investigating the multimodal integration in coloured music synesthesia. Hence, a different mechanism could be found between auditory and visual association regions in the brain of coloured music synesthetes [29]. Diffusion Tensor Imaging (DTI) was employed in focusing on the white matter tracts between the occipital and temporal domain [29]. The main white matter pathways found between the temporal and occipital lobe were the inferior fronto-occipital fasciculus (IFOF) [17]and inferior longitudinal fasciculus (ILF) [6]. IFOF is the white matter tract that links the occipital cortical regions with the frontal lobe via the temporal areas [6] [17] while ILF is the pathway that joins the temporal lobe with the occipital lobe [6]. The fractional anisotropy (FA) (is a measure that indicates the connectivity of white matter) showed that the IFOF on the right hemisphere was greater in connectivity in coloured music synesthetes compared to non-synesthetes [29]. However, such difference was not found in the ILF between synesthetes and non-synesthetes [29]. These observations were also consistent with the results acquired from the behavioural tests done in coloured music synesthetes [29].

Figure 4: Activity of Right IFOF
Image Unavailable
The colour blue indicates the right IFOF in the brain of coloured music synesthetes.
It was suggested that this region greatly associated with the score on the behavioural tests
done on coloured music synesthetes. (image adapted from zamm et al., 2013 [29])

The high FA value of IFOF acquired was critical in supporting the excessive multimodal integration as a contributor to synestheisa [29]. This is because IFOF is involved with multimodal integration and the process of visual and speech information [2] [20]. Therefore, such enhancement could be associated with the multisensory perceptions observed in synesthetes [29]. This is further supported by these behavioural tests (sound to colour mapping and Synesthesia Battery) in which synesthetes with higher consistency in this behavioural tests also showed greater white matter integrity in the right IFOF [29]. Furthermore, the rightward asymmetry was also observed in the resting EEG study, in which the right side of the auditory cortex showed enhanced activity [18]. The actual underlying cause in auditory visual synesthesia remains to be ambiguous as many researches are contrary to each other [22] [27] [29].

Bibliography
1. Aleman, A., et al. Activation of striate cortex in the absence of visual stimulation: An fMRI study of synesthesia. Neuroreport. (2001) 12: 2827–2830.
2. Baggio, H.C., et al. Structural correlates of facial emotion recognition deficits in Parkinson's disease patients. Neuropsychologia. (2012) 50: 2121–2128.
3. Bartels, A., and Zeki, S.The architecture of the colour centre in the human visual brain: New results and a review. European Journal of Neuroscience.(2000) 12: 172–193.
4. Brang, D et al. Magnetoencephalography reveals early activation of V4 in grapheme-color synesthesia. NeuroImage .(2010) 53: 268–274.
5. Calvert, G. A..Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex. (2001) 11: 1110–1123.
6. Catani, M., et al. Virtual in vivo interactive dissection of white matter fasciculi in the human brain. Neuroimage. (2002)17: 77–94.
7. Chiou R, et al. Beyond colour perception: Auditory- visual synaesthesia induces experiences of geometric objects in specific locations, Cortex. (2012), doi:10.1016/j.cortex.2012.04.006
8. Cytowic, R.E., Eagleman, D.M.. Wednesday is Indigo Blue: Discovering the Brain 611of Synesthesia. MIT Press, Cambridge. (2009)
9. Esterman, M., et al. Coming unbound: Disrupting automatic integration of synesthetic color and graphemes by transcranial magnetic stimulation of the right parietal lobe. Journal of Cognitive Neuroscience. (2006) 18:1570–1576.
10. Gallant, J. L., et al. A human extrastriate area functionally homologous to macaque V4. Neuron. (2000). 27: 227–235.
11. Grossenbacher, P. G.and Lovelace, C. T. Mechanisms of synesthesia: Cognitive and physiological constraints. Trends in Cognitive Sciences. (2001) 5:36–41.
12. Gustavsen, K. and Gallant, J. L. Shape perception: Complex contour representation in visual area V4. Current Biology (2003). 13: 234–R235.
13. Hickok,G., et al. The cortical organization of speech processing. Nature Reviews Neuroscience. (2007) 8 (5): 393-402.
14. Hubbard, E. M.. Neurophysiology of synesthesia. Current Psychiatry Reports. (2007) 9: 193–199.
15. Hubbard, E. M., et al. Neurocognitive mechanisms of synesthesia. Neuron. (2005) 48(3), 509–520.
16. Hubbard, E.M., et al. The cross activation theory at 10. Journal of Neuropsychology. (2011) 5(2): 152-177.
17. Kier, E.L., et al. MR imaging of the temporal stem: anatomic dissection tractography of the uncinate fasciculus, inferior occipitofrontal fasciculus, and Meyer's loop of the optic radiation. AJNR Am. J. Neuroradiol. (2004) 25 (5) : 677–691
18. Langer, N., Jancke, L. A strong parietal hub in the small-world network of coloured-hearing synaesthetes during resting state EEG. Journal of Neuropsycholog. (2011) 5: 178-202
19. 19. Van Leeuwen, T.M., et al. Synaesthetic Colour in the Brain: Beyond Colour Areas. A Functional Magnetic Resonance Imaging Study of Synaesthetes and Matched Controls. Plos One. (2010) 5: e12074.
20. Martino, J., et al. Anatomic dissection of the inferior fronto-occipital fasciculus in the lights of brain stimulation data. Cortex. (2010) 46: 691–699.
21. Mulvenna, C.M. and Walsh, V.. Synaesthesia: supernormal integration? Trends Cognitive Science. (2006) 10: 350–352.
22. Neufeld, J., et al. Reduced audio-visual integration in synaesthetes indicated by the double-flash illusion. Brain Research 1473. (2012) 78-86.
23. Neufeld, J., et al. The neural correlates of coloured music: a functional MRI investigation of auditory- visual synaesthesia. Neuropsychologia. (2012) 50:85-89
24. Rich, A. N., et al. Neural correlates of imagined and synaesthetic colours. Neuropsychologia. (2006) 44: 2918–2925.
25. Robertson, L. C..Binding, spatial attention and perceptual awareness. Nature Reviews Neuroscience. (2003) 4: 93–102.
26. Rouw, R., et al. Brain areas involved in synaesthesia: a review. J. Neuropsychol. (2011) 5: 214–242.
27. Sinke, C., et al. Reduced audiovisual integration in synestheisa- evidence from bimodal speech perception. (2012) doi: 10.1111/jnp.12006
28. Steven, M. S., et al.. Activation of color-selective areas of the visual cortex in a blind synesthete. Cortex. (2006) 42: 304–308.
29. Zamm, A., et al., Pathways to seeing music: Enhanced structural connectivity in colored-music synesthesia, NeuroImage. (2013), http://dx.doi.org/10.1016/j.neuroimage.2013.02.024
Add a New Comment
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License