Auditory Visual Synesthesia |
![]() |
Auditory visual synesthesia allows synesthetes to experience anomalous sensation in which sounds, chords, or pitch could elicit different visual experiences [23] (image adapted from http://lofalexandria.blogspot.ca/2012/12/synesthesia.html) |
Auditory visual synesthesia is a neurological phenomenon in which a sound stimulus can elicit visual experiences [23]. Unlike auditory-induced visual image experienced by non-synesthetic population, auditory visual synesthesia is involuntary and consistent[7]. Auditory visual synesthesia is further categorized by the type of audio stimuli paired with specific visual responses such as sound colour synesthesia and coloured music synesthesia[7]. Previous studies stated that auditory visual synesthesia could be the consequence of excessive multimodal integration [16]. Recent studies involving the double flash illusion, however, have observed reduced multimodal integration [22]. Recent researches also discovered the anomalous activation of the cortical areas such as V4 and parietal lobe that could lead to potential auditory visual synesthetic experiences [23]. The study of synesthesia is important as it is a rare neurological condition, and understanding of synesthesia can provide insight regarding to the mechanisms of perception, cognition and communication between cortical regions [7].
This video illustrates the synesthetic experiences by a synesthete, Jane Mackay |
Table of Contents
|
1. Cortical Activities on Auditory Visual Synesthesia
Many research papers focused on one type of synesthesia called grapheme- colour synesthesia. In those articles, they found evidences of the involvement of parietal lobe and V4 in grapheme-colour synesthesia[4][19]. It is possible that similar cortical activities could occur in auditory visual synesthesia because both types of synesthesia are involved with involuntary control of visual synesthetic experiences [7]. The following pivotal points will give further details of the involvement of parietal lobe and V4 region in auditory visual synesthesia.
1.1 Parietal and Auditory Hub observed via EEG
Parietal lobe was suggested to be essential in creating synesthetic experiences [15] [26]. Therefore, it is crucial to explore this region for its role in auditory visual synestheisa. Different imaging techniques have been used to observe the cortical activities such as electroencephalographic (EEG) technique. Langer and Jake (2011) applied this technique on 12 auditory visual synesthetes at a resting state. Four different frequency bands (alpha 1, alpha 2, beta, and theta) were employed in observing the cortical activities [18]. Resting EEG activity manifests stronger hubs (strong connection between regions) occurring in the left parietal lobe (especially in the angular gyrus and the region that is anterior to the postcentral gryus) and the right side of auditory cortex in auditory visual synesthetes when compared to non-synthestes[18]. This result is consistent with other research [23].
The strong parietal lobe activities observed in auditory visual synesthetes leads to the implication that parietal lobe could be essential in generating synesthetic perceptions [18]. The hyperactivity observed at rest, strongly suggests that there could be a potential differences in functional brain network that is specific to auditory visual synesthestes [18].
A stronger alpha2 frequency band was observed in relation to the resting activity in the right auditory cortex [18]. The right auditory cortex is essential in processing pitch information [13]; therefore, this could be the potential cause in the lateralization observed in the auditory cortex [18]. The strong auditory hub observed could be playing a pivotal role in supporting auditory visual synesthesia [18].
1.2 Functional MRI investigation: Inferior Parietal Cortex
Functional magnetic resonance imaging (fMRI) was used to trace the cortical activities in auditory visual synesthetes under a non-linguistic sound test (chords and tones)[23]. Furthermore, this study shares similar result acquired from the resting EEG experiment, mentioned previously. Both results pointed to the potential involvement of parietal cortex in auditory visual synaesthesia [18] [23]. Nonetheless, a more precise result was obtained from the fMRI study under an active brain. The result indicates that there was hyperactivity that occurred in the left inferior parietal cortex (IPC) compared to non synesthetes [23].
It is suggested that functions of parietal lobe include multimodal integration (such as integration between auditory and visual) and spatio-dynamic processing [5]. Therefore, the hyperactivity on the left side of inferior parietal cortex could be the consequence of excessive multimodal integration (hyperbinding) [18]. This excessive integration between the two modalities: inducer region (such as auditory) and concurrent region (such as visual) in the parietal lobe through disinhibited feedback is thought as the underlying mechanism in synesthesia[25] [11]. Nonetheless, opposing results are presented by double flash illusion and McGurk illusion experiments [22] [27]. This will be further discussed in the next section.
Figure 1. Cortical Activation in Auditory Visual Synesthesia. |
![]() |
(A) This image indicates a region with significant activation in the IPC upon auditory stimulations in the brain of synesthetes. (B) Moreover, two regions at the left and right temporal lobe showed increase in activation in synesthetes during auditory stimulations. (Image adapted from Neufeld, et al., 2012 [23]) |
1.3 V4 in Fusiform Gyrus
The V4 region is involved in processing information such as colour and object properties, making it the best candidate in causing synesthetic perceptions experienced by auditory visual or grapheme- colour synesthetes [3] [10][12]. Moreover, many neuro-imaging studies provide evidences on the anomalous activities in colour area V4 regions for grapheme- colour synesthesisa. These evidences support the involvement of V4 in synesthetic perceptions [4][28]. However, such results were not observed in the resting EEG and fMRI studies presented earlier [18] [23]. The V4 region in auditory visual synesthetes showed no significant activation at rest (in a resting EEG study) nor did it show any anomalous activities in an active brain (under non-linguistic sound fMRI study) [18] [23].
The absence of V4 activation in those two studies, however, cannot be used against in its possible involvement in auditory visual synesthesia [14]. It is possible that different forms of synesthesia might require different involvements of visual regions [23]. Also, other associative visual regions such as lingual gyrus [24] and inferior temporal cortex [1] have been observed with significant in activation in synesthetes. These associative regions could be the potential recruitment in the production of synesthetic experiences [23].
2. Controversy on Level of Auditory and Visual Integration in Auditory Visual Synesthesia
Many recent publications stated that the occurrence of synesthesia could be the consequence of enhancement in non-synthesthetic multimodal integration (such as auditory and visual integration) [9] [21]. This hypothesis is supported by the hyperactivity observed in the parietal lobe presented in early sections [18] [23]. Reduction in multimodal integration was reported from recent experiments on auditory visual synesthesia, yet research done on music-colour synesthetes stated otherwise [23][27][29]. Therefore, the auditory and visual integration mechanism regarding to auditory visual synesthesia remains to be controversial.
2.1 Double Flash Illusion Experiment
Figure 2: Data Information Regarding to Double Flash Illusion Experiment |
![]() |
The dotted line represents the non-synethestes in which illusionary flash was reported at the SOA (stimulus onset asynchrony) of -150 to +150ms while a narrower SOA of -100 to +100ms illustrated by a solid line represents the illusionary flash experienced by synesthetes. This figure also demonstrates the reduction in reported illusonary flash perceived by synesthetes compared to non-synesthetes.(image adopted from Neufeld, et al., 2012 [21]) |
Double flash illusion is an illusion elicited by a single light flash coupled with multiple beep sounds, thus multiple flashes could be perceived by individuals [22]. Hence, if the increment of illusions perceived by the auditory visual synesthetes occurs, this will allude to the general enhancement of auditory and visual integration [22]. Moreover, the variance in separating the auditory and visual inducers presented in the double flash illusion test could be used to test the temporal binding window of auditory visual integration [22]. It was hypothesized that synesthetes would perceive more illusionary flashes with widened temporal binding window of auditory-visual integration, if higher general multimodal integration mechanism occurs in synesthetes [22]. However, opposing result was obtained from the double flash illusion test. It was indicated that synesthetes perceived less illusions with a narrow temporal binding window compared to non-synesthetes[22].
These results suggested that synesthetes have less general auditory and visual integration compared to non-synesthetes [22]. Thus, this opposes to the hypothesis which states the excessive multimodal integration could contribute to synesthesia [9] [21]. One of the postulations for the decrease in illusions perceived in synesthetes is their distractor-filtering ability [22]. Synesthetes are better trained in focusing on relevant inducers in order to compensate for the distractive synesthestic perceptions; therefore, the production of multisensory perceptions along with reduction in general multimodal integration was observed [22].
2.2 McGurk Illusion and Speech Comprehension Test
Professor Lawrence Rosenblum from university of California explains and demonstrates the effect of McGurk Illusion. |
Apart from the double flash illusion, McGurk illusion and comprehension test were employed in exploring the excessive multimodal binding in auditory visual synesthetes. The McGurk illusion was stimulated when formation of new perception was caused by the fusion of different auditory and visual information [27]. McGurk illusion was used in determining the strength in auditory and visual integration, it also predicted that synesthetes would be more susceptible to this test (this is because more fusion would occur from auditory and visual information, if there was excessive integration between the two modalities) [27]. As for the speech comprehension test, its objective was to observe rather or not synesthetes could profit more in speech perception while observing articulatory movements in a noisy environment [27]. Analogous to the double flash illusion experiment, synesthetes showed reduced in number of illusions and profited less in speech perception during the speech comprehension test compared to non-synesthetes [27].
Results acquired from double flash illusion, McGurk illusion, and speech comprehension test all pointed to the general reduction in auditory and visual integration [22] [27]. Thus, it is possible that the synesthetic experiences are the result from the reduced in multimodal integration [27]. This statement could be supported by the way how children learn to write and read. As children are gaining the skills of reading and writing, the transmission of information form auditory to visual region becomes essential [27]. Therefore, it may be necessary to offset such multimodal integration insufficiency by generating the multisensory perceptions experienced by synesthetes [27]. Moreover, the enhanced distracter-filtering ability in synesthetes enables synesthetes to perceive less illusions in McGurk and double flash illusion and to profit less from the speech comprehension test [22] [27].
2.3 Coloured Music: Enhanced Structural Activity
Figure 3: Different Images of The IFOF Tracts |
![]() |
This shows the IFOF tracts in the brain of synesthetes (top) and non-synesthetes (bottom) in three different views (sagittal, coronal, and axial) (image adapted from zamm et al., 2013 [29]) |
Coloured music synesthesia is a specific type of auditory visual synesthesia in which musical sounds like chords or tones induce synesthetic perceptions [8]. Aside from the double flash and McGurk illusion experiments, a neuroanatomical approach was used in investigating the multimodal integration in coloured music synesthesia. Hence, a different mechanism could be found between auditory and visual association regions in the brain of coloured music synesthetes [29]. Diffusion Tensor Imaging (DTI) was employed in focusing on the white matter tracts between the occipital and temporal domain [29]. The main white matter pathways found between the temporal and occipital lobe were the inferior fronto-occipital fasciculus (IFOF) [17]and inferior longitudinal fasciculus (ILF) [6]. IFOF is the white matter tract that links the occipital cortical regions with the frontal lobe via the temporal areas [6] [17] while ILF is the pathway that joins the temporal lobe with the occipital lobe [6]. The fractional anisotropy (FA) (is a measure that indicates the connectivity of white matter) showed that the IFOF on the right hemisphere was greater in connectivity in coloured music synesthetes compared to non-synesthetes [29]. However, such difference was not found in the ILF between synesthetes and non-synesthetes [29]. These observations were also consistent with the results acquired from the behavioural tests done in coloured music synesthetes [29].
Figure 4: Activity of Right IFOF |
![]() |
The colour blue indicates the right IFOF in the brain of coloured music synesthetes. It was suggested that this region greatly associated with the score on the behavioural tests done on coloured music synesthetes. (image adapted from zamm et al., 2013 [29]) |
The high FA value of IFOF acquired was critical in supporting the excessive multimodal integration as a contributor to synestheisa [29]. This is because IFOF is involved with multimodal integration and the process of visual and speech information [2] [20]. Therefore, such enhancement could be associated with the multisensory perceptions observed in synesthetes [29]. This is further supported by these behavioural tests (sound to colour mapping and Synesthesia Battery) in which synesthetes with higher consistency in this behavioural tests also showed greater white matter integrity in the right IFOF [29]. Furthermore, the rightward asymmetry was also observed in the resting EEG study, in which the right side of the auditory cortex showed enhanced activity [18]. The actual underlying cause in auditory visual synesthesia remains to be ambiguous as many researches are contrary to each other [22] [27] [29].
Solid intro. It lets the reader know the clear structure to your wiki and I liked how you made smooth transitions from one section
to the other. I dont know if it's necessary but when you're alluding to a section further down, you could always link that word or phrase to a particular subsection in your wiki via: individidual: #toc _. (Sorry if that's not right but I'm pretty sure it is!)
Also, don't forget the code the reference for the second sentence of your intro!
^^Thanks Yunjoo and good job on the first few sections of the wiki!
Hey awesome topic and information! Not sure sure if you know but a few of your images are unavailable. Captions and titles for them would be awesome! Great work!
Hey,
The page looks great, with valuable information posted. I just came across a video on google named "What is audio visual synesthesia?"
Maybe this link can be useful for your page :)
(I can not send you the link it self , because of the low karma points, sorry)
Nice job, and great pictures! I see you've included the captions for the videos outside of the embedded frame - since you've embedded as an html. You can use this code (which I believe is on the main page) to have the captions inside the border:
[[include inc:video |
videourl=http://www.youtube.com/embed/qM4WvF_PikI?list=PLF0F34B7DAD8B6A2C |
heading= Heading text |
caption= Caption text |
float=right |
height= 300 |
width= 550 |
]]
I've included your first video into the code.
Thank you so much for your help!
Great job so far! It seems like your bib cites aren't linking to your bibliography section or is that just me? If you want to double check that that'd be great.
And some of your images are unavailable :(
Great job! Love the topic. I dont know why the youtube video isnt showing up for me. :S
It's absolutely fascinating to know how much processing actually goes into every sensation that you perceive. Great integration of figures, images, and videos. Seems comprehensive and very well done.
P.S. I actually didn't only hear ba with a b. It seems I have a well developed auditory cortex. :-)