Multimodal processing in simultaneous interpreting with text: Interpreters focus more on the visual than the auditory modality
Agnieszka Chmiel,Przemysław Janikowski and Agnieszka Lijewska
Adam Mickiewicz University in Poznań | University of Silesia in Katowice
The present study focuses on (in)congruence of input between the visual and the auditory modality in simultaneous
interpreting with text. We asked twenty-four professional conference interpreters to simultaneously interpret an aurally and
visually presented text with controlled incongruences in three categories (numbers, names and control words), while measuring interpreting
accuracy and eye movements. The results provide evidence for the dominance of the visual modality, which goes against the
professional standard of following the auditory modality in the case of incongruence. Numbers enjoyed the greatest accuracy across
conditions possibly due to simple cross-language semantic mappings. We found no evidence for a facilitation effect for congruent
items, and identified an impeding effect of the presence of the visual text for incongruent items. These results might be
interpreted either as evidence for the Colavita effect (in which visual stimuli take precedence over auditory ones) or as
strategic behaviour applied by professional interpreters to avoid risk.
Simultaneous interpreting is sometimes likened to the job of an air traffic controller working at a very busy airport (Zeier 1997; Leeson 2005). Just like the air traffic controller has to monitor the position, speed and altitude of numerous aircraft in the assigned airspace and communicate with pilots, the interpreter has to juggle multiple tasks, such as listening to the source language text, expressing the same meaning in the target language and monitoring their own output. Interpreters’ performance oftentimes sparks awe among their listeners who admire bilingual processing under extreme temporal constraints. Although simultaneous interpreting seems a very difficult task in itself due to the cross-linguistic multitasking involved, professional interpreters are successful even when yet more multitasking is required, as is the case in simultaneous interpreting with text. In this type of interpreting, interpreters have access to the text of the speech to be read out by the speaker. In this case, the interpreter has to control three different channels – the speech produced by the speaker in the source language in the auditory channel, the self-produced interpreting in the target language in the auditory channel, and the source language text provided in the visual channel. Since the speaker may depart from the written text while delivering the to-be-interpreted speech, the interpreter has to constantly monitor both input channels. Additionally, self-monitoring gains importance in this type of interpreting because interpreters have to avoid source language interference not only from the auditory, but also from the visual channel (i.e., the written text).
Braun, Susanne, and Andrea Clarici
1996 “Inaccuracy for Numerals in Simultaneous Interpretation: Neurolinguistic and Neuropsychological Perspectives.” Interpreting 7: 85–102.
Cammoun, Rawdha, Catherine Davies, Konstantin Ivanov, and Boris Naimushin
2009Simultaneous Interpretation with Text. Is the Text ‘Friend’ or ‘Foe’? Laying Foundations for a Teaching Module.” MA diss. University of Geneva.
2015 “Przetwarzanie w tłumaczeniu symultanicznym.” In Dydaktyka tłumaczenia ustnego, edited by Agnieszka Chmiel and Przemysław Janikowski, 227–247. Katowice: Stowarzyszenie Inicjatyw Wydawniczych przy współpracy UŚ i UAM.
2007 “Cultural Recycling of Cortical Maps.” Neuron 56 (2): 384–398.
Desmet, Bart, Mieke Vandierendonck, and Bart Defrancq
2018 “Simultaneous Interpretation of Numbers and the Impact of Technological Support.” In Interpreting and Technology, edited by Claudio Fantinuoli, 13–27. Berlin: Language Science Press.
Deutsch, Avital, and Shlomo Bentin
2001 “Syntactic and Semantic Factors in Processing Gender Agreement in Hebrew: Evidence from ERPs and Eye Movements.” Journal of Memory and Language 45 (2): 200–224.
Doehrmann, Oliver, and Marcus Naumer
2008 “Semantics and the Multisensory Brain: How Meaning Modulates Processes of Audio-Visual Integration.” Brain Research 1242 (November): 136–150.
1997 “Conference Interpreting as a Cognitive Management Problem.” In Cognitive Processes in Translation and Interpreting, edited by Joseph Danks, Gregory Shreve, Stephen Fountain, and Michael McBeath, 196–214. Thousand Oaks, CA: Sage.
2009Basic Concepts and Models for Interpreter and Translator Training. Revised edition. Amsterdam: John Benjamins.
2018 “Vision Dominates Audition in Adults but Not Children: A Meta-Analysis of the Colavita Effect.” Neuroscience & Biobehavioral Reviews 94 (November): 286–301.
Iodice, Rosario, Juan José García Meilán, Juan Carro Ramos, and Jeff A. Small
2018 “Sentence Context and Word-Picture Cued-Recall Paired-Associate Learning Procedure Boosts Recall in Normal and Mild Alzheimer’s Disease Patients.” Behavioural Neurology.
Ivanov, Konstantin, Kate Davies, and Boris Naimushin
2014 “Teaching Simultaneous Interpretation with Text.” In Взаимопонимание в Многоязычном Мире: Культура, Язык, Перевод. Сборник Статей в Честь Профессора и.с. Алексеевой / Fighting the Fog in Multilingualism. A Festschrift in Honour of Irina S. Alekseeva, edited by Sergey Goncharov and Angelique Antonova, 38–48. St. Petersburg: Herzen University Press.
Janikowski, Przemysław, Agnieszka Chmiel, and Agnieszka Lijewska
In preparation). “Coping with Sensory Overload in Simultaneous Interpreting with Text: Examining the Timecourse with Eye-Ear, Ear-Voice, and Eye-Voice Span Measures.”
Jiménez Ivars, Maria Amparo
1999La traducción a la vista: Un análisis descriptivo. PhD diss. Universitat Jaume I.
Koppen, Camille, Agnès Alsius, and Charles Spence
2008 “Semantic Congruency and the Colavita Visual Dominance Effect.” Experimental Brain Research 184 (4): 533–546.
2015 “The Visual or the Aural: Which Modality Is Dominant in Simultaneous Interpreting?” Conference paper presented at the ICEAL (International Conference on Eyetracking and Applied Linguistics), Warszawa, September 22.
Korpal, Paweł, and Katarzyna Stachowiak
2017 “A Closer Look at Numbers in Simultaneous Interpreting: An Eye-Tracking Study.” Poster presented at the 19th European Conference on Eye Movements, Wuppertal, August 20.
Lamberger-Felber, Heike, and Julia Schneider
2009 “Linguistic Interference in Simultaneous Interpreting with Text: A Case Study.” In Efforts and Models in Interpreting and Translation Research. A Tribute to Daniel Gile (Benjamins Translation Library 80), edited by Gyde Hansen, Andrew Chesterman, and Heidrun Gerzymisch-Arbogast, 215–236. Amsterdam: John Benjamins.
2004 “Shared Attention during Sight Translation, Sight Interpretation and Simultaneous Interpretation.” Meta: Journal des traducteurs / Meta: Translators’ Journal 49 (2): 294–306.
Laurienti, Paul, Robert Kraft, Joseph Maldjian, Jonathan Burdette, and Mark Wallace
2004 “Semantic Congruence Is a Critical Factor in Multisensory Behavioral Performance.” Experimental Brain Research 158 (4): 405–414.
2005 “Making the Effort in Simultaneous Interpreting: Some Considerations for Signed Language Interpreters.” In Topics in Signed Language Interpreting: Theory and Practice, edited by Terry Janzen, 51–68. Amsterdam: John Benjamins.
Lefcheck, Jonathan, and Robert Freckleton
2016 “PIECEWISESEM: Piecewise Structural Equation Modelling in R for Ecology, Evolution, and Systematics.” Methods in Ecology and Evolution 7 (5): 573–579.
Lemhöfer, Kristin, and Mirjam Broersma
2012 “Introducing LexTALE: A Quick and Valid Lexical Test for Advanced Learners of English.” Behavior Research Methods 44 (2): 325–43.
Matusz, Pawel, Hannah Broadbent, Jessica Ferrari, Benjamin Forrest, Rebecca Merkley, and Gaia Scerif
2004 “Text and Risk in Translation.” In Choice and Difference in Translation: The Specifics of Transfer, edited by Maria Sidiropoulou and Anastasia Papaconstantinou, 27–42. Athens: University of Athens.
Schweda Nicholson, Nancy
1989 “Documentation and Text Preparation for Simultaneous Interpretation.” In Coming of Age: Proceedings of the 30th Annual Conference of the American Translators Association, Oct. 11–15, 1989, Washington, D.C., edited by Deanna Hammond, 163–182. Medford, N.J.: Learned Information Inc.
Seeber, Kilian G.
2016 “Modeling Multimodal Processing in Simultaneous Interpreting.” Poster presented at the 75th anniversary of FTI, Genève, September 30.
2017 “Multimodal Processing in Simultaneous Interpreting.” In The Handbook of Translation and Cognition, edited by John W. Schwieter and Aline Ferreira, 461–475. Hoboken, NJ: Wiley Blackwell.
Setton, Robin, and Andrew Dawrant
2016Conference Interpreting: A Complete Course. Amsterdam: John Benjamins.
2003 “Simultaneous Interpreting with Text: A Suggestion for Teaching Methodology.” In The Third Riga Symposium on Pragmatic Aspects of Translation: Proceedings, edited by Andrejs Veisbergs, 153–162. Riga: University of Latvia, Aarhus School of Business.
Sinnett, Scott, Charles Spence, and Salvador Soto-Faraco
2007 “Visual Dominance and Attention: The Colavita Effect Revisited.” Perception & Psychophysics 69 (5): 673–686.
Sinnett, Scott, Salvador Soto-Faraco, and Charles Spence
2008 “The Co-occurrence of Multisensory Competition and Facilitation.” Acta Psychologica 128 (1): 153–161.
2009 “Explaining the Colavita Visual Dominance Effect.” In Attention (Progress in Brain Research 176), edited by Narayanan Srinivasan, 245–258. Amsterdam: Elsevier.