Cover not available
Article published In:
Translation, Cognition & Behavior
Vol. 4:1 (2021) ► pp.1–25

Does it help to see the speaker’s lip movements?

An investigation of cognitive load and mental effort in simultaneous interpreting

ORCID logo | Zurich University for Applied Sciences (ZHAW)

Simultaneous interpreting combines auditory and visual information. Within a multitude of visual inputs that interpreters receive, the one from the speaker seems to be particularly important (Bühler 1985; Seubert 2019). One reason might be that lip movements enhance speech perception and might thus reduce cognitive load in simultaneous interpreting and hence, induce lower mental effort. This effect may be even more pronounced when noise is added to the source speech. This study was conducted to investigate cognitive load and mental effort during simultaneous interpreting (a) with and without the ability to see speaker’s lip movements, and (b) with and without interfering noise. A group of listeners was included to control for task-related effects. Mental effort and cognitive load were measured using pupillometry, interpreting accuracy measures, and subjective reports. The facilitation hypothesis for lip movements was not confirmed. However, the pupillometric data suggests that lip movements may increase arousal.

Keywords: simultaneous interpreting, cognitive load, arousal, pupillometry, visual input
Article outline
  • 1.Introduction
    • 1.1Lip movements as visual input in simultaneous interpreting
    • 1.2Tapping into cognitive load and mental effort
  • 2.Materials and methods
    • 2.1Participants
    • 2.2Experimental stimuli
    • 2.3Apparatus
    • 2.4Procedure
  • 3.Analysis and results
    • 3.1Subjective reports
    • 3.2Interpreting accuracy
    • 3.3Pupillometry
  • 4.Discussion and conclusions
    • 4.1What is the effect of lip movements on interpreting and what do pupil sizes tell us?
    • 4.2Limitations
    • 4.3Conclusion
  • Acknowledgments
  • Notes
  • References
Published online: 19 May 2021
DOI logo
https://doi.org/10.1075/tcb.00049.gie
References
Anderson, Linda
1994 “Simultaneous Interpretation: Contextual and Translation Aspects.” In Benjamins Translation Library: Vol. 31. Bridging the Gap: Empirical Research in Simultaneous Interpretation. Edited by S. Lambert and B. Moser-Mercer, 101–248. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Bates, Douglas, Martin Mächler, Ben Bolker & Steve Walker
2015 “Fitting Linear Mixed-Effects Models Using lme4.” Journal of Statistical Software 67 (1). DOI logoGoogle Scholar
Beatty, Jack
1982 “Phasic Not Tonic Pupillary Responses Vary With Auditory Vigilance Performance.” Psychophysiology, 19(2): 167–172. DOI logoGoogle Scholar
Bernstein, Lynne E., Edward T. Auer, Sumik Takayanagi
2004 “Auditory Speech Detection in Noise Enhanced by Lipreading.” Speech Communication, 44(1–4): 5–18. DOI logoGoogle Scholar
Brancazio, Lawrence, Catherine T. Best and Carol A. Fowler
2006 “Visual Influences on Perception of Speech and Nonspeech Vocal-Tract Events.” Language and Speech, 49(1): 21–53. DOI logoGoogle Scholar
Brown, Robert and Howard. E. Page
1939 “Pupil Dilatation and Dark Adaptation.” Journal of Experimental Psychology, 25(4): 347–360. DOI logoGoogle Scholar
Bühler, Hildegund
1985 “Conference Interpreting: A Multichannel Communication Phenomenon.” Meta: Journal des traducteurs, 30(1): 49–54. DOI logoGoogle Scholar
Chapman, C. Richard, Shunichi Oka, David H. Bradshaw, Robert C. Jacobson and Gary W. Donaldson
1999 “Phasic Pupil Dilation Response to Noxious Stimulation in Normal Volunteers: Relationship to Brain Evoked Potentials and Pain Report.” Psychophysiology, 36(1): 44–52. DOI logoGoogle Scholar
Christensen, Rune H. B.
2015 Ordinal – Regression Models for Ordinal Data. Version 2015.1–21. [URL] (accessed 01 December 2015).
Courtney, Christopher G., Michael E. Dawson, Anne M. Schell, Arvind Iyer, Thomas D. Parsons
2010 “Better than the Real Thing: Eliciting fear with Moving and Static Computer-generated Stimuli.” International Journal of Psychophysiology, 78(2): 107–114. DOI logoGoogle Scholar
Davies, M.
2008 Word frequency data. Retrieved from The Corpus of Contemporary American English (COCA): [URL] (accessed 19 March 2021).
Díaz-Galaz, Stephanie, Presentacion Padilla Benítez, and M. Teresa Bajo Molina
2015 “The Role of Advance Preparation in Simultaneous Interpreting: A Comparison of Professional Interpreters and Interpreting Students”. Interpreting 17(1): 1–25. DOI logoGoogle Scholar
Dillinger, Mike
1990 “Comprehension during Interpreting: What Do Interpreters Know that Bilinguals Don’t?” The Interpreters’ Newsletter, 31: 41–58. Accessed March 03 17, 2021. [URL]
Ehrensberger-Dow, Maureen, Michaela Albl-Mikasa, Katrin Andermatt, Andrea Hunziker Heeb and Caroline Lehr
2020 “Cognitive Load in Processing ELF: Translators, Interpreters, and Other Multilinguals.” Journal of English as a Lingua Franca 9(2): 217–238. DOI logoGoogle Scholar
Engelhardt, Paul E., Fernanda Ferreira and Elana G. Patsenko
2010 “Pupillometry Reveals Processing Load during Spoken Language Comprehension.” Quarterly Journal of Experimental Psychology, 63(4): 639–645. DOI logoGoogle Scholar
European Commission
2009a “United Airlines rewards fittest people.” Retrieved from Speech Repository: [URL] (accessed 4 February 2021).
European Commission
2009b “Disenchantment at work.” Retrieved from Speech Repository: [URL] (accessed 4 February 2021)
2012a “Demographic shift in Europe.” Retrieved from Speech Repository: [URL] (accessed 4 February 2021).
European Commission
2012b “Greece in the doldrums.” Retrieved from Speech Repository: [URL] (accessed 4 February 2021).
Fisk, Angus S., Shu K. E. Tam, L. A. Brown, Vladyslav V. Vyazovskiy, David M. Bannerman and Stuart N. Peirson
2018 “Light and Cognition: Roles for Circadian Rhythms, Sleep, and Arousal.” Frontiers in Neurology, 91: 56. DOI logoGoogle Scholar
Gerver, David
2002 “The Effects of Source Language Presentation Rate on the Performance of Simultaneous Conference Interpreters.” The Interpreting Studies. Reader. Edited by F. Pöchhacker and M. Shesinger, 52–66. London: Routledge.Google Scholar
1974 “The Effects of Noise on the Performance of Simultaneous Interpreters: Accuracy of Performance.” Acta Psychologica, 38(3): 159–167. DOI logoGoogle Scholar
Gieshoff, Anne Catherine
2021 “The impact of visible lip movements on silent pauses in simultaneous interpreting.” Interpreting. Advanced online publication. DOI logoGoogle Scholar
2018 “The Impact of Visual Input on Cognitive Load in Simultaneous Interpreting”. Mainz: Johannes Gutenberg Universität.Google Scholar
Granholm, Eric, Robert F. Asarnow, Andrew J. Sarkin and Karen, L.
1996 “Pupillary Responses Index Cognitive Resource Limitations.” Psychophysiology, 331: 457–461. DOI logoGoogle Scholar
Halverson, Sandra
2017 “Multimethod approaches.” The Handbook of Translation and Cognition. Edited by J. Schwieter and A. Ferreira. 195–212. Hoboken: Wiley Blackwell. DOI logoGoogle Scholar
Hild, Adelina
2015 “Discourse Comprehension in Simultaneous Interpreting: The Role of Expertise and Redundancy.” Psycholinguistic and Cognitive Inquiries into Translation and Interpreting. Edited by A. Ferreira and J. W. Schwieter. 67–100. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Hyönä, Jukka, Jorma Tommola, Anna-Maria Alaja
1995 “Pupil Dilation as a Measure of Processing Load in Simultaneous Interpretation and Other Language Tasks.” The Quarterly Journal of Experimental Psychology Section A, 48(3): 598–612. DOI logoGoogle Scholar
Ivars, Amparo J., Daniel P. Calatayud
2001 “ ‘I Failed because I Got Nervous’. Anxiety and Performance in Interpreter Trainees: An Empirical Study.” The Interpreter’s Newsletter, 111: 105–120. Accessed March 17, 2021. [URL]
Jesse, Alexandra, Nick Vrignaud, Michael M. Cohen and Dominic W. Massaro
2000 “The Processing of Information from Multiple Sources in Simultaneous Interpreting.” Interpreting, 5(2): 95–115. DOI logoGoogle Scholar
Jordan, Timothy R. and Paul Sergeant
2000 “Effects of Distance on Visual and Audiovisual Speech Recognition.” Language and Speech, 43(1): 107–124. DOI logoGoogle Scholar
Kahneman, Daniel
1973 Attention and Effort. Englewood Cliffs, N.J.: Prentice-Hall.Google Scholar
Korpal, Paweł
2016 “Interpreting as a Stressful Activity: Physiological Measures of Stress in Simultaneous Interpreting.” Poznan Studies in Contemporary Linguistics, 52(2): 297–316. DOI logoGoogle Scholar
Kramer, Sophia E., Theo S. Kapteyn, Josst M. Festen, and Dirk J. Kuik
1997 “Assessing Aspects of Auditory Handicap by Means of Pupil Dilation.” Audiology, 361: 155–164. DOI logoGoogle Scholar
Krejtz, Krzysztof, Andrew T. Duchowski, Anna Niedzielska, Cezary Biele, and Izabela Krejtz
2018 “Eye Tracking Cognitive Load Using Pupil Diameter and Microsaccades with Fixed Gaze”. PLOS ONE 13(9): e0203629. DOI logoGoogle Scholar
Kuchinsky, Stefanie E., Jayne B. Ahlstrom, Kenneth I. Vaden, Stephanie L. Cute, Larry E. Humes, Judy R. Dubno and Mark A. Eckert
2013 “Pupil Size Varies with Word Listening and Response Selection Difficulty in Older Adults with Hearing Loss: Pupil Size in Older Adults.” Psychophysiology, 50(1): 23–34. DOI logoGoogle Scholar
Lecumberri, Maria L. G., Martin Cooke and Anne Cutler
2010 “Non-native Speech Perception in Adverse Conditions: A Review.” Speech Communication, 52(1): 864–886. DOI logoGoogle Scholar
Lee, Jieun
2008 “Rating Scales for Interpreting Performance Assessment.” The Interpreter and Translator Trainer, 2(2): 165–184. DOI logoGoogle Scholar
Liu, Minhua, Diane L. Schallert, and Patrick J. Carroll
2004 ‘Working Memory and Expertise in Simultaneous Interpreting’. Interpreting 6(1): 19–42. DOI logoGoogle Scholar
Lowenstein, Otto, Richard Feinberg and Irene Loewenfeld
1963 “Pupillary Movements During Acute and Chronic Fatigue A New Test for the Objective Evaluation of Tiredness.” Investigative Ophthalmology, 2(2): 138–158.Google Scholar
Massaro, Dominic W. And Michael M. Cohen
1999 “Speech Perception in Perceivers with Hearing Loss: Synergy of Multiple Modalities.” Journal of Speech, Language, and Hearing Research, 42(1), 21–41. DOI logoGoogle Scholar
Mirman, Daniel
2014 Growth Curve Analysis and Visualization Using R. Boca Raton, FL: CRC Press.Google Scholar
Moser-Mercer, Barbara, Alexandra Künzli and Marina Korac
1998 “Prolonged Turns in Interpreting: Effects on Quality, Physiological and Psychological Stress (Pilot study).” Interpreting, 3(1): 47–64. DOI logoGoogle Scholar
Oliva, Manuel and Andrey Anikin
2018 “Pupil Dilation Reflects the Time Course of Emotion Recognition in Human Vocalizations.” Scientific Reports, 81, 4871. DOI logoGoogle Scholar
Peelle, Jonathan E. and Mitchell S. Sommers
2015 “Prediction and Constraint in Audiovisual Speech Perception.” Cortex, 681: 169–181. DOI logoGoogle Scholar
Peirce, Jonathan, Jeremy R. Gray, Sol Simpson, Michael MacAskill, Richard Höchenberger, Hiroyuki Sogo, Eric Kastman and Jonas Kristoffer Lindeløv
2019 “PsychoPy2: Experiments in Behavior made Easy.” Behavior Research Methods, 51(1): 195–203. DOI logoGoogle Scholar
Peirce, Jonathan W.
2007 “PsychoPy – Psychophysics Software in Python.” Journal of Neuroscience Methods, 162(1–2), 8–13. DOI logoGoogle Scholar
R Core Team
2016 “R: A language and environment for statistical computing.” Version 3.2.4. Retrieved from R Foundation for Statistical Computing: [URL] (accessed 1 March 2016).
Rennert, Sylvie
2008 “Visual Input in Simultaneous Interpreting.” Meta: journal des traducteurs, 53(1): 204–217. DOI logoGoogle Scholar
Roziner, Ilan & Miriam Shlesinger
2010 “Much Ado about Something Remote: Stress and Performance in Remote Interpreting.” Interpreting, 12(2): 214–247. DOI logoGoogle Scholar
Seeber, Kilian G.
2017 “Multimodal Processing in Simultaneous Interpreting.” The Handbook of Translation and Cognition. Edited by J. Schwieter and A. Ferreira. 461–475. Hoboken: Wiley Blackwell. DOI logoGoogle Scholar
2015 “Cognitive Load in Simultaneous Interpreting: Measures and Methods.” Interdisciplinarity in translation and interpreting process research. Edited by M. Ehrensberger-Dow, S. Göpferich & S. O’Brien (Eds.). 18–33. DOI logoGoogle Scholar
Seeber, Kilian G. and Dirk Kerzel
2012 “Cognitive Load in Simultaneous Interpreting: Model Meets Data.” International Journal of Bilingualism, 16(2): 228–242. DOI logoGoogle Scholar
Seeber, Kilian G.
2011 “Cognitive Load in Simultaneous Interpreting.” Interpreting, 13(2): 176–204. DOI logoGoogle Scholar
Setton, Robin
1999 Simultaneous Interpretation: A Cognitive-Pragmatic Analysis. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Seubert, Sabine
2019 Visuelle Informationen beim Simultandolmetschen: Eine Eyetracking-Studie. Berlin: Frank & Timme GmbH.Google Scholar
Stachowiak-Szymczak, Katarzyna
2019 Eye Movements and Gestures in Simultaneous and Consecutive Interpreting. DOI logoGoogle Scholar
Sumby, William H. and Irwin Pollack
1954 “Visual Contribution to Speech Intelligibility in Noise.” Journal of the Acoustical Society of America, (261): 212–215. DOI logoGoogle Scholar
Tabri, Dollen, Kim M. S. A. Chacra & Tim Pring
2010 “Speech Perception in Noise by Monolingual, Bilingual and Trilingual Listeners.” International Journal of Language & Communication Disorders, 46(4): 411–422. DOI logoGoogle Scholar
Thomas, Sharon M., Timothy R. Jordan
2004 “Contributions of Oral and Extraoral Facial Movement to Visual and Audiovisual Speech Perception.” Journal of Experimental Psychology: Human Perception and Performance, 30(5): 873–888. DOI logoGoogle Scholar
Tobii support
2016, July 20. Personal communication.Google Scholar
Tobii Technology
2010 Tobii TX300 Eye Tracker. User Manual. Accessed March 17, 2021. [URL]
Tommola, Jorma and Johan Lindholm
1995 “Experimental Research on Interpreting: Which Dependent Variable?” Topics in interpreting research. Edited by J. Tommola. 121–133. Turku: University of Turku.Google Scholar
Tye-Murray, Nancy, Mitchell Sommers and Brent Spehar
2007 “Auditory and Visual Lexical Neighborhoods in Audiovisual Speech Perception.” Trends in Amplification, 11(4): 233–241. DOI logoGoogle Scholar
van der Wel, Pauline and Henk van Steenbergen
2018 “Pupil Dilation as an Index of Effort in Cognitive Control Tasks: A Review.” Psychonomic Bulletin & Review, 25(6): 2005–2015. DOI logoGoogle Scholar
Vatikiotis-Bateson, Eric, Inge-Marie Eigsti, Sumio Yano and Kevin G. Munhall
1998 “Eye Movement of Perceivers during Audiovisual Speech Perception.” Perception & Psychophysics, 60(6): 926–940. DOI logoGoogle Scholar
von Kriegstein, Katharina, Özgür Dogan, Martina Grüter, Anne-Lise Giraud, Christian A. Kell, Thomas Grüter, Andreas Kleinschmidt and Stefan A. Kiebel
2008 “Simulation of Talking Faces in the Human Brain Improves Auditory Speech Recognition.” Proceedings of the National Academy of Sciences, 105(18): 6747–6752. DOI logoGoogle Scholar
Wickham, H.
2009 ggplot2: Elegant Graphics for Data Analysis. New York: Springer. DOI logoGoogle Scholar
Wong, Hoe K. and Julien Epps
2016 “Pupillary Transient Responses to Within-Task Cognitive Load Variation.” Computer Methods and Programs in Biomedicine, 1371: 47–63. DOI logoGoogle Scholar
Yoon, Jinny J. and Helen V. Danesh-Meyer
2019 “Caffeine and the Eye.” Survey of Ophthalmology, 64(3): 334–344. DOI logoGoogle Scholar
Zekveld, Adriana A., Dirk J. Heslenfeld, Ingrid S. Johnsrude, Niek J. Versfeld and Sophia E. Kramer
2014 “The Eye as a Window to the Listening Brain: Neural Correlates of Pupil Size as a Measure of Cognitive Listening Load.” NeuroImage, 1011: 76–86. DOI logoGoogle Scholar
Zwischenberger, Cornelia
2010 “Quality Criteria in Simultaneous Interpreting: An International vs. A National View.” The Interpreters’ Newsletter, 151: 127–142. Accessed 17 March 2021. [URL]