Multimodal processing in simultaneous interpreting with text: Interpreters focus more on the visual than the auditory modality

Agnieszka Chmiel, Przemysław Janikowski and Agnieszka Lijewska
Adam Mickiewicz University in Poznań | University of Silesia in Katowice

Abstract

The present study focuses on (in)congruence of input between the visual and the auditory modality in simultaneous interpreting with text. We asked twenty-four professional conference interpreters to simultaneously interpret an aurally and visually presented text with controlled incongruences in three categories (numbers, names and control words), while measuring interpreting accuracy and eye movements. The results provide evidence for the dominance of the visual modality, which goes against the professional standard of following the auditory modality in the case of incongruence. Numbers enjoyed the greatest accuracy across conditions possibly due to simple cross-language semantic mappings. We found no evidence for a facilitation effect for congruent items, and identified an impeding effect of the presence of the visual text for incongruent items. These results might be interpreted either as evidence for the Colavita effect (in which visual stimuli take precedence over auditory ones) or as strategic behaviour applied by professional interpreters to avoid risk.

Keywords:
Publication history

Date received: 19 December 2018

Date accepted: 18 December 2019

Published online: 21 January 2020

Table of contents

Simultaneous interpreting is sometimes likened to the job of an air traffic controller working at a very busy airport (Zeier 1997; Leeson 2005). Just like the air traffic controller has to monitor the position, speed and altitude of numerous aircraft in the assigned airspace and communicate with pilots, the interpreter has to juggle multiple tasks, such as listening to the source language text, expressing the same meaning in the target language and monitoring their own output. Interpreters’ performance oftentimes sparks awe among their listeners who admire bilingual processing under extreme temporal constraints. Although simultaneous interpreting seems a very difficult task in itself due to the cross-linguistic multitasking involved, professional interpreters are successful even when yet more multitasking is required, as is the case in simultaneous interpreting with text. In this type of interpreting, interpreters have access to the text of the speech to be read out by the speaker. In this case, the interpreter has to control three different channels – the speech produced by the speaker in the source language in the auditory channel, the self-produced interpreting in the target language in the auditory channel, and the source language text provided in the visual channel. Since the speaker may depart from the written text while delivering the to-be-interpreted speech, the interpreter has to constantly monitor both input channels. Additionally, self-monitoring gains importance in this type of interpreting because interpreters have to avoid source language interference not only from the auditory, but also from the visual channel (i.e., the written text).

Full-text access is restricted to subscribers. Log in to obtain additional credentials. For subscription information see Subscription & Price. Direct PDF access to this article can be purchased through our e-platform.

References

Braun, Susanne, and Andrea Clarici
1996 “Inaccuracy for Numerals in Simultaneous Interpretation: Neurolinguistic and Neuropsychological Perspectives.” Interpreting 7: 85–102.Google Scholar
Cammoun, Rawdha, Catherine Davies, Konstantin Ivanov, and Boris Naimushin
2009 Simultaneous Interpretation with Text. Is the Text ‘Friend’ or ‘Foe’? Laying Foundations for a Teaching Module.” MA diss. University of Geneva.Google Scholar
Chmiel, Agnieszka
2015 “Przetwarzanie w tłumaczeniu symultanicznym.” In Dydaktyka tłumaczenia ustnego, edited by Agnieszka Chmiel and Przemysław Janikowski, 227–247. Katowice: Stowarzyszenie Inicjatyw Wydawniczych przy współpracy UŚ i UAM.Google Scholar
Colavita, Francis
1974 “Human Sensory Dominance.” Perception & Psychophysics 16 (2): 409–412. DOI logoGoogle Scholar
Dehaene, Stanislas, and Laurent Cohen
2007 “Cultural Recycling of Cortical Maps.” Neuron 56 (2): 384–398. DOI logoGoogle Scholar
Desmet, Bart, Mieke Vandierendonck, and Bart Defrancq
2018 “Simultaneous Interpretation of Numbers and the Impact of Technological Support.” In Interpreting and Technology, edited by Claudio Fantinuoli, 13–27. Berlin: Language Science Press.Google Scholar
Deutsch, Avital, and Shlomo Bentin
2001 “Syntactic and Semantic Factors in Processing Gender Agreement in Hebrew: Evidence from ERPs and Eye Movements.” Journal of Memory and Language 45 (2): 200–224. DOI logoGoogle Scholar
Doehrmann, Oliver, and Marcus Naumer
2008 “Semantics and the Multisensory Brain: How Meaning Modulates Processes of Audio-Visual Integration.” Brain Research 1242 (November): 136–150. DOI logoGoogle Scholar
Gile, Daniel
1997 “Conference Interpreting as a Cognitive Management Problem.” In Cognitive Processes in Translation and Interpreting, edited by Joseph Danks, Gregory Shreve, Stephen Fountain, and Michael McBeath, 196–214. Thousand Oaks, CA: Sage.Google Scholar
2009 Basic Concepts and Models for Interpreter and Translator Training. Revised edition. Amsterdam: John Benjamins. DOI logoGoogle Scholar
2018 “The Effort Models and Gravitational Model Clarifications and Update.” June 1. http://​www​.cirinandgile​.com​/powerpoint​/The​-Effort​-Models​-and​-Gravitational​-Model​-Clarifications​-and​-update​.pdf
Hirst, Rebecca, Lucy Cragg, and Harriet Allen
2018 “Vision Dominates Audition in Adults but Not Children: A Meta-Analysis of the Colavita Effect.” Neuroscience & Biobehavioral Reviews 94 (November): 286–301. DOI logoGoogle Scholar
Iodice, Rosario, Juan José García Meilán, Juan Carro Ramos, and Jeff A. Small
2018 “Sentence Context and Word-Picture Cued-Recall Paired-Associate Learning Procedure Boosts Recall in Normal and Mild Alzheimer’s Disease Patients.” Behavioural Neurology. DOI logoGoogle Scholar
Ivanov, Konstantin, Kate Davies, and Boris Naimushin
2014 “Teaching Simultaneous Interpretation with Text.” In Взаимопонимание в Многоязычном Мире: Культура, Язык, Перевод. Сборник Статей в Честь Профессора и.с. Алексеевой / Fighting the Fog in Multilingualism. A Festschrift in Honour of Irina S. Alekseeva, edited by Sergey Goncharov and Angelique Antonova, 38–48. St. Petersburg: Herzen University Press.Google Scholar
Janikowski, Przemysław, Agnieszka Chmiel, and Agnieszka Lijewska
In preparation). “Coping with Sensory Overload in Simultaneous Interpreting with Text: Examining the Timecourse with Eye-Ear, Ear-Voice, and Eye-Voice Span Measures.”
Jiménez Ivars, Maria Amparo
1999 La traducción a la vista: Un análisis descriptivo. PhD diss. Universitat Jaume I.Google Scholar
Koppen, Camille, Agnès Alsius, and Charles Spence
2008 “Semantic Congruency and the Colavita Visual Dominance Effect.” Experimental Brain Research 184 (4): 533–546. DOI logoGoogle Scholar
Koppen, Camille, and Charles Spence
2007 “Spatial Coincidence Modulates the Colavita Visual Dominance Effect.” Neuroscience Letters 417 (2): 107–111. DOI logoGoogle Scholar
Korpal, Paweł, and Katarzyna Stachowiak
2015 “The Visual or the Aural: Which Modality Is Dominant in Simultaneous Interpreting?” Conference paper presented at the ICEAL (International Conference on Eyetracking and Applied Linguistics), Warszawa, September 22.
2017 “A Closer Look at Numbers in Simultaneous Interpreting: An Eye-Tracking Study.” Poster presented at the 19th European Conference on Eye Movements, Wuppertal, August 20.
Lamberger-Felber, Heike, and Julia Schneider
2009 “Linguistic Interference in Simultaneous Interpreting with Text: A Case Study.” In Efforts and Models in Interpreting and Translation Research. A Tribute to Daniel Gile (Benjamins Translation Library 80), edited by Gyde Hansen, Andrew Chesterman, and Heidrun Gerzymisch-Arbogast, 215–236. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Lambert, Sylvie
2004 “Shared Attention during Sight Translation, Sight Interpretation and Simultaneous Interpretation.” Meta: Journal des traducteurs / Meta: Translators’ Journal 49 (2): 294–306. DOI logoGoogle Scholar
Laurienti, Paul, Robert Kraft, Joseph Maldjian, Jonathan Burdette, and Mark Wallace
2004 “Semantic Congruence Is a Critical Factor in Multisensory Behavioral Performance.” Experimental Brain Research 158 (4): 405–414. DOI logoGoogle Scholar
Leeson, Loraine
2005 “Making the Effort in Simultaneous Interpreting: Some Considerations for Signed Language Interpreters.” In Topics in Signed Language Interpreting: Theory and Practice, edited by Terry Janzen, 51–68. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Lefcheck, Jonathan, and Robert Freckleton
2016 “PIECEWISESEM: Piecewise Structural Equation Modelling in R for Ecology, Evolution, and Systematics.” Methods in Ecology and Evolution 7 (5): 573–579. DOI logoGoogle Scholar
Lemhöfer, Kristin, and Mirjam Broersma
2012 “Introducing LexTALE: A Quick and Valid Lexical Test for Advanced Learners of English.” Behavior Research Methods 44 (2): 325–43. DOI logoGoogle Scholar
Matusz, Pawel, Hannah Broadbent, Jessica Ferrari, Benjamin Forrest, Rebecca Merkley, and Gaia Scerif
2015 “Multi-Modal Distraction: Insights from Children’s Limited Attention.” Cognition 136 (March): 156–165. DOI logoGoogle Scholar
Messina, Alessandro
1998 “The Reading Aloud of English Language Texts in Simultaneously Interpreted Conferences.” Interpreting 3 (2): 147–161. DOI logoGoogle Scholar
Nakagawa, Shinichi, and Holger Schielzeth
2013 “A General and Simple Method for Obtaining R2 from Generalized Linear Mixed-Effects Models.” Methods in Ecology and Evolution 4 (2): 133–142. DOI logoGoogle Scholar
Pinochi, Diletta
2009 “Simultaneous Interpretation of Numbers: Comparing German and English to Italian. An Experimental Study.” The Interpreters’ Newsletter 14: 33–57.Google Scholar
Prior, Anat, Michal Katz, Islam Mahajna, and Orly Rubinsten
2015 “Number Word Structure in First and Second Language Influences Arithmetic Skills.” Frontiers in Psychology 6 (March). DOI logoGoogle Scholar
Pöchhacker, Franz
2004 Introducing Interpreting Studies. London: Routledge. DOI logoGoogle Scholar
Pym, Anthony
2004 “Text and Risk in Translation.” In Choice and Difference in Translation: The Specifics of Transfer, edited by Maria Sidiropoulou and Anastasia Papaconstantinou, 27–42. Athens: University of Athens.Google Scholar
Schweda Nicholson, Nancy
1989 “Documentation and Text Preparation for Simultaneous Interpretation.” In Coming of Age: Proceedings of the 30th Annual Conference of the American Translators Association, Oct. 11–15, 1989, Washington, D.C., edited by Deanna Hammond, 163–182. Medford, N.J.: Learned Information Inc.Google Scholar
Seeber, Kilian G.
2016 “Modeling Multimodal Processing in Simultaneous Interpreting.” Poster presented at the 75th anniversary of FTI, Genève, September 30.
Seeber, Kilian
2017 “Multimodal Processing in Simultaneous Interpreting.” In The Handbook of Translation and Cognition, edited by John W. Schwieter and Aline Ferreira, 461–475. Hoboken, NJ: Wiley Blackwell. DOI logoGoogle Scholar
Setton, Robin, and Andrew Dawrant
2016 Conference Interpreting: A Complete Course. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Sile, Dina
2003 “Simultaneous Interpreting with Text: A Suggestion for Teaching Methodology.” In The Third Riga Symposium on Pragmatic Aspects of Translation: Proceedings, edited by Andrejs Veisbergs, 153–162. Riga: University of Latvia, Aarhus School of Business.Google Scholar
Sinnett, Scott, Charles Spence, and Salvador Soto-Faraco
2007 “Visual Dominance and Attention: The Colavita Effect Revisited.” Perception & Psychophysics 69 (5): 673–686. DOI logoGoogle Scholar
Sinnett, Scott, Salvador Soto-Faraco, and Charles Spence
2008 “The Co-occurrence of Multisensory Competition and Facilitation.” Acta Psychologica 128 (1): 153–161. DOI logoGoogle Scholar
Spence, Charles
2009 “Explaining the Colavita Visual Dominance Effect.” In Attention (Progress in Brain Research 176), edited by Narayanan Srinivasan, 245–258. Amsterdam: Elsevier. DOI logoGoogle Scholar
Spence, Charles, Cesare Parise, and Yi-Chuan Chen
2012 “The Colavita Visual Dominance Effect.” In The Neural Bases of Multisensory Processes, edited by Micah M. Murray and Mark T. Wallace, 529–566. Boca Raton, FL: CRC Press. http://​www​.ncbi​.nlm​.nih​.gov​/books​/NBK92851/
Spychała, Joanna Maria
2015 The Impact of Source Text Availability on Simultaneous Interpreting Performance. MA diss. Uniwersytet im. Adama Mickiewicza w Poznaniu.Google Scholar
Stubblefield, Alexandra, Lauryn Jacobs, Yongju Kim, and Paula Goolkasian
2013 “Colavita Dominance Effect Revisited: The Effect of Semantic Congruity.” Attention, Perception, & Psychophysics 75 (8): 1827–1839. DOI logoGoogle Scholar
Sullivan, Brian, Harm Brouwer, Alba Rodriguez, Matthew Crocker, and Pia Knoeferle
2016 “Eye Movements during Linguistic-Visual Conflicts.” Poster presented at The Attentive Listener in the Visual World. 3rd AttLis workshop, Potsdam, March 10.
Whiting, Caroline, Yury Shtyrov, and William Marslen-Wilson
2015 “Real-Time Functional Architecture of Visual Word Recognition.” Journal of Cognitive Neuroscience 27 (2): 246–265. DOI logoGoogle Scholar
Wille, Claudia, and Mirjam Ebersbach
2016 “Semantic Congruency and the (Reversed) Colavita Effect in Children and Adults.” Journal of Experimental Child Psychology 141 (January): 23–33. DOI logoGoogle Scholar
Zeier, Hans
1997 “Psychophysiological Stress Research.” Interpreting 2 (1–2): 231–249. DOI logoGoogle Scholar