Cover not available
Article published In:
Target
Vol. 32:1 (2020) ► pp.37–58

Multimodal processing in simultaneous interpreting with text

Interpreters focus more on the visual than the auditory modality

ORCID logo | Adam Mickiewicz University in Poznań
| University of Silesia in Katowice
| Adam Mickiewicz University in Poznań
The present study focuses on (in)congruence of input between the visual and the auditory modality in simultaneous interpreting with text. We asked twenty-four professional conference interpreters to simultaneously interpret an aurally and visually presented text with controlled incongruences in three categories (numbers, names and control words), while measuring interpreting accuracy and eye movements. The results provide evidence for the dominance of the visual modality, which goes against the professional standard of following the auditory modality in the case of incongruence. Numbers enjoyed the greatest accuracy across conditions possibly due to simple cross-language semantic mappings. We found no evidence for a facilitation effect for congruent items, and identified an impeding effect of the presence of the visual text for incongruent items. These results might be interpreted either as evidence for the Colavita effect (in which visual stimuli take precedence over auditory ones) or as strategic behaviour applied by professional interpreters to avoid risk.
Keywords: simultaneous interpreting, multimodal processing, multisensory processing, incongruence, visual modality
Article outline
  • 1.Introduction
  • 2.Existing research on multimodal processing
    • 2.1Multimodal processing in simultaneous interpreting with text
    • 2.2Cognitive aspects of multimodal processing
  • 3.The present study
    • 3.1Participants
    • 3.2Materials
    • 3.3Procedure
  • 4.Results
  • 5.Discussion
  • 6.Conclusions
  • References
Published online: 21 January 2020
DOI logo
https://doi.org/10.1075/target.18157.chm
References (48)
References
Braun, Susanne, and Andrea Clarici. 1996. “Inaccuracy for Numerals in Simultaneous Interpretation: Neurolinguistic and Neuropsychological Perspectives.” Interpreting 71: 85–102.Google Scholar
Cammoun, Rawdha, Catherine Davies, Konstantin Ivanov, and Boris Naimushin. 2009. Simultaneous Interpretation with Text. Is the Text ‘Friend’ or ‘Foe’? Laying Foundations for a Teaching Module.” MA diss. University of Geneva.Google Scholar
Chmiel, Agnieszka. 2015. “Przetwarzanie w tłumaczeniu symultanicznym.” In Dydaktyka tłumaczenia ustnego, edited by Agnieszka Chmiel and Przemysław Janikowski, 227–247. Katowice: Stowarzyszenie Inicjatyw Wydawniczych przy współpracy UŚ i UAM.Google Scholar
Colavita, Francis. 1974. “Human Sensory Dominance.” Perception & Psychophysics 16 (2): 409–412. DOI logoGoogle Scholar
Dehaene, Stanislas, and Laurent Cohen. 2007. “Cultural Recycling of Cortical Maps.” Neuron 56 (2): 384–398. DOI logoGoogle Scholar
Desmet, Bart, Mieke Vandierendonck, and Bart Defrancq. 2018. “Simultaneous Interpretation of Numbers and the Impact of Technological Support.” In Interpreting and Technology, edited by Claudio Fantinuoli, 13–27. Berlin: Language Science Press.Google Scholar
Deutsch, Avital, and Shlomo Bentin. 2001. “Syntactic and Semantic Factors in Processing Gender Agreement in Hebrew: Evidence from ERPs and Eye Movements.” Journal of Memory and Language 45 (2): 200–224. DOI logoGoogle Scholar
Doehrmann, Oliver, and Marcus Naumer. 2008. “Semantics and the Multisensory Brain: How Meaning Modulates Processes of Audio-Visual Integration.” Brain Research 12421 (November): 136–150. DOI logoGoogle Scholar
Gile, Daniel. 1997. “Conference Interpreting as a Cognitive Management Problem.” In Cognitive Processes in Translation and Interpreting, edited by Joseph Danks, Gregory Shreve, Stephen Fountain, and Michael McBeath, 196–214. Thousand Oaks, CA: Sage.Google Scholar
. 2009. Basic Concepts and Models for Interpreter and Translator Training. Revised edition. Amsterdam: John Benjamins. DOI logoGoogle Scholar
. 2018. “The Effort Models and Gravitational Model Clarifications and Update.” June 1. [URL]
Hirst, Rebecca, Lucy Cragg, and Harriet Allen. 2018. “Vision Dominates Audition in Adults but Not Children: A Meta-Analysis of the Colavita Effect.” Neuroscience & Biobehavioral Reviews 941 (November): 286–301. DOI logoGoogle Scholar
Iodice, Rosario, Juan José García Meilán, Juan Carro Ramos, and Jeff A. Small. 2018. “Sentence Context and Word-Picture Cued-Recall Paired-Associate Learning Procedure Boosts Recall in Normal and Mild Alzheimer’s Disease Patients.” Behavioural Neurology. DOI logoGoogle Scholar
Ivanov, Konstantin, Kate Davies, and Boris Naimushin. 2014. “Teaching Simultaneous Interpretation with Text.” In Взаимопонимание в Многоязычном Мире: Культура, Язык, Перевод. Сборник Статей в Честь Профессора и.с. Алексеевой / Fighting the Fog in Multilingualism. A Festschrift in Honour of Irina S. Alekseeva, edited by Sergey Goncharov and Angelique Antonova, 38–48. St. Petersburg: Herzen University Press.Google Scholar
Janikowski, Przemysław, Agnieszka Chmiel, and Agnieszka Lijewska. (In preparation). “Coping with Sensory Overload in Simultaneous Interpreting with Text: Examining the Timecourse with Eye-Ear, Ear-Voice, and Eye-Voice Span Measures.”
Jiménez Ivars, Maria Amparo. 1999. La traducción a la vista: Un análisis descriptivo. PhD diss. Universitat Jaume I.Google Scholar
Koppen, Camille, Agnès Alsius, and Charles Spence. 2008. “Semantic Congruency and the Colavita Visual Dominance Effect.” Experimental Brain Research 184 (4): 533–546. DOI logoGoogle Scholar
Koppen, Camille, and Charles Spence. 2007. “Spatial Coincidence Modulates the Colavita Visual Dominance Effect.” Neuroscience Letters 417 (2): 107–111. DOI logoGoogle Scholar
Korpal, Paweł, and Katarzyna Stachowiak. 2015. “The Visual or the Aural: Which Modality Is Dominant in Simultaneous Interpreting?” Conference paper presented at the ICEAL (International Conference on Eyetracking and Applied Linguistics), Warszawa, September 22.
. 2017. “A Closer Look at Numbers in Simultaneous Interpreting: An Eye-Tracking Study.” Poster presented at the 19th European Conference on Eye Movements, Wuppertal, August 20.
Lamberger-Felber, Heike, and Julia Schneider. 2009. “Linguistic Interference in Simultaneous Interpreting with Text: A Case Study.” In Efforts and Models in Interpreting and Translation Research. A Tribute to Daniel Gile (Benjamins Translation Library 80), edited by Gyde Hansen, Andrew Chesterman, and Heidrun Gerzymisch-Arbogast, 215–236. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Lambert, Sylvie. 2004. “Shared Attention during Sight Translation, Sight Interpretation and Simultaneous Interpretation.” Meta: Journal des traducteurs / Meta: Translators’ Journal 49 (2): 294–306. DOI logoGoogle Scholar
Laurienti, Paul, Robert Kraft, Joseph Maldjian, Jonathan Burdette, and Mark Wallace. 2004. “Semantic Congruence Is a Critical Factor in Multisensory Behavioral Performance.” Experimental Brain Research 158 (4): 405–414. DOI logoGoogle Scholar
Leeson, Loraine. 2005. “Making the Effort in Simultaneous Interpreting: Some Considerations for Signed Language Interpreters.” In Topics in Signed Language Interpreting: Theory and Practice, edited by Terry Janzen, 51–68. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Lefcheck, Jonathan, and Robert Freckleton. 2016. “PIECEWISESEM: Piecewise Structural Equation Modelling in R for Ecology, Evolution, and Systematics.” Methods in Ecology and Evolution 7 (5): 573–579. DOI logoGoogle Scholar
Lemhöfer, Kristin, and Mirjam Broersma. 2012. “Introducing LexTALE: A Quick and Valid Lexical Test for Advanced Learners of English.” Behavior Research Methods 44 (2): 325–43. DOI logoGoogle Scholar
Matusz, Pawel, Hannah Broadbent, Jessica Ferrari, Benjamin Forrest, Rebecca Merkley, and Gaia Scerif. 2015. “Multi-Modal Distraction: Insights from Children’s Limited Attention.” Cognition 1361 (March): 156–165. DOI logoGoogle Scholar
Messina, Alessandro. 1998. “The Reading Aloud of English Language Texts in Simultaneously Interpreted Conferences.” Interpreting 3 (2): 147–161. DOI logoGoogle Scholar
Nakagawa, Shinichi, and Holger Schielzeth. 2013. “A General and Simple Method for Obtaining R2 from Generalized Linear Mixed-Effects Models.” Methods in Ecology and Evolution 4 (2): 133–142. DOI logoGoogle Scholar
Pinochi, Diletta. 2009. “Simultaneous Interpretation of Numbers: Comparing German and English to Italian. An Experimental Study.” The Interpreters’ Newsletter 141: 33–57.Google Scholar
Prior, Anat, Michal Katz, Islam Mahajna, and Orly Rubinsten. 2015. “Number Word Structure in First and Second Language Influences Arithmetic Skills.” Frontiers in Psychology 61 (March). DOI logoGoogle Scholar
Pöchhacker, Franz. 2004. Introducing Interpreting Studies. London: Routledge. DOI logoGoogle Scholar
Pym, Anthony. 2004. “Text and Risk in Translation.” In Choice and Difference in Translation: The Specifics of Transfer, edited by Maria Sidiropoulou and Anastasia Papaconstantinou, 27–42. Athens: University of Athens.Google Scholar
Schweda Nicholson, Nancy. 1989. “Documentation and Text Preparation for Simultaneous Interpretation.” In Coming of Age: Proceedings of the 30th Annual Conference of the American Translators Association, Oct. 11–15, 1989, Washington, D.C., edited by Deanna Hammond, 163–182. Medford, N.J.: Learned Information Inc.Google Scholar
Seeber, Kilian G. 2016. “Modeling Multimodal Processing in Simultaneous Interpreting.” Poster presented at the 75th anniversary of FTI, Genève, September 30.
Seeber, Kilian. 2017. “Multimodal Processing in Simultaneous Interpreting.” In The Handbook of Translation and Cognition, edited by John W. Schwieter and Aline Ferreira, 461–475. Hoboken, NJ: Wiley Blackwell. DOI logoGoogle Scholar
Setton, Robin, and Andrew Dawrant. 2016. Conference Interpreting: A Complete Course. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Sile, Dina. 2003. “Simultaneous Interpreting with Text: A Suggestion for Teaching Methodology.” In The Third Riga Symposium on Pragmatic Aspects of Translation: Proceedings, edited by Andrejs Veisbergs, 153–162. Riga: University of Latvia, Aarhus School of Business.Google Scholar
Sinnett, Scott, Charles Spence, and Salvador Soto-Faraco. 2007. “Visual Dominance and Attention: The Colavita Effect Revisited.” Perception & Psychophysics 69 (5): 673–686. DOI logoGoogle Scholar
Sinnett, Scott, Salvador Soto-Faraco, and Charles Spence. 2008. “The Co-occurrence of Multisensory Competition and Facilitation.” Acta Psychologica 128 (1): 153–161. DOI logoGoogle Scholar
Spence, Charles. 2009. “Explaining the Colavita Visual Dominance Effect.” In Attention (Progress in Brain Research 176), edited by Narayanan Srinivasan, 245–258. Amsterdam: Elsevier. DOI logoGoogle Scholar
Spence, Charles, Cesare Parise, and Yi-Chuan Chen. 2012. “The Colavita Visual Dominance Effect.” In The Neural Bases of Multisensory Processes, edited by Micah M. Murray and Mark T. Wallace, 529–566. Boca Raton, FL: CRC Press. [URL]
Spychała, Joanna Maria. 2015. The Impact of Source Text Availability on Simultaneous Interpreting Performance. MA diss. Uniwersytet im. Adama Mickiewicza w Poznaniu.Google Scholar
Stubblefield, Alexandra, Lauryn Jacobs, Yongju Kim, and Paula Goolkasian. 2013. “Colavita Dominance Effect Revisited: The Effect of Semantic Congruity.” Attention, Perception, & Psychophysics 75 (8): 1827–1839. DOI logoGoogle Scholar
Sullivan, Brian, Harm Brouwer, Alba Rodriguez, Matthew Crocker, and Pia Knoeferle. 2016. “Eye Movements during Linguistic-Visual Conflicts.” Poster presented at The Attentive Listener in the Visual World. 3rd AttLis workshop, Potsdam, March 10.
Whiting, Caroline, Yury Shtyrov, and William Marslen-Wilson. 2015. “Real-Time Functional Architecture of Visual Word Recognition.” Journal of Cognitive Neuroscience 27 (2): 246–265. DOI logoGoogle Scholar
Wille, Claudia, and Mirjam Ebersbach. 2016. “Semantic Congruency and the (Reversed) Colavita Effect in Children and Adults.” Journal of Experimental Child Psychology 1411 (January): 23–33. DOI logoGoogle Scholar
Zeier, Hans. 1997. “Psychophysiological Stress Research.” Interpreting 2 (1–2): 231–249. DOI logoGoogle Scholar
Cited by (15)

Cited by 15 other publications

Li, Tianyun & Agnieszka Chmiel
2024. Automatic subtitles increase accuracy and decrease cognitive load in simultaneous interpreting. Interpreting. International Journal of Research and Practice in Interpreting DOI logo
Santamaría Urbieta, Alexandra, Elena Alcalde Peñalver & Peter Bannister
2024. Estrategias metacomunicativas y de multimodalidad en inglés y en español de los medinfluencers en Instagram. Vivat Academia ► pp. 1 ff. DOI logo
Chekin, Leonid S.
2023. First Secretary Gierek, President Carter, and the president’s Polish interpreter. Babel. Revue internationale de la traduction / International Journal of Translation / Revista Internacional de Traducción 69:6 ► pp. 725 ff. DOI logo
Gao, Fei
2023. Getting the message in ‘Sound’ across at conference interpreting: a case study on rendering prosodic emphasis. Perspectives 31:6 ► pp. 1033 ff. DOI logo
Yuan, Lu & Binhua Wang
2023. Cognitive processing of the extra visual layer of live captioning in simultaneous interpreting. Triangulation of eye-tracked process and performance data. Ampersand 11 ► pp. 100131 ff. DOI logo
Yuan, Lu & Binhua Wang
2024. Eye-tracking the processing of visual input of live transcripts in remote simultaneous interpreting. FORUM. Revue internationale d’interprétation et de traduction / International Journal of Interpretation and Translation 22:1 ► pp. 118 ff. DOI logo
Zhao, Nan, Zhenguang G. Cai, Yanping Dong & Daniel Mirman
2023. Speech errors in consecutive interpreting: Effects of language proficiency, working memory, and anxiety. PLOS ONE 18:10 ► pp. e0292718 ff. DOI logo
Cheung, Andrew K. F.
2022. Remote Simultaneous Interpreting from Home or Hub: Accuracy of Numbers from English into Mandarin Chinese. In Translation and Interpreting in the Age of COVID-19 [Corpora and Intercultural Studies, 9], ► pp. 113 ff. DOI logo
Cheung, Andrew K. F.
2024. Cognitive load in remote simultaneous interpreting: place name translation in two Mandarin variants. Humanities and Social Sciences Communications 11:1 DOI logo
Chmiel, Agnieszka & Nicoletta Spinolo
2022. Testing the impact of remote interpreting settings on interpreter experience and performance. Translation, Cognition & Behavior 5:2 ► pp. 250 ff. DOI logo
Zhu, Xuelian & Vahid Aryadoust
2022. A Synthetic Review of Cognitive Load in Distance Interpreting: Toward an Explanatory Model. Frontiers in Psychology 13 DOI logo
Kruger, Jan-Louis
2021. Eye tracking. In Handbook of Translation Studies [Handbook of Translation Studies, 5], ► pp. 80 ff. DOI logo
Pöchhacker, Franz
2021. Multimodality in interpreting. In Handbook of Translation Studies [Handbook of Translation Studies, 5], ► pp. 152 ff. DOI logo
Seeber, Kilian G., Laura Keller & Alexis Hervais-Adelman
2020. When the ear leads the eye – the use of text during simultaneous interpretation. Language, Cognition and Neuroscience 35:10 ► pp. 1480 ff. DOI logo
Su, Wenchao
2020. Issues and Approaches to CTIS. In Eye-Tracking Processes and Styles in Sight Translation [New Frontiers in Translation Studies, ], ► pp. 9 ff. DOI logo

This list is based on CrossRef data as of 27 september 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.