Article In:
Interpreting: Online-First Articles

Visual processing during computer-assisted consecutive interpreting

Evidence from eye movements

ORCID logo | Guangdong University of Foreign Studies
ORCID logo | Macquarie University

This study investigates the visual processing patterns during computer-assisted consecutive interpreting (CACI). In phase I of the proposed CACI workflow, the interpreter listens to the source speech and respeaks it into speech recognition (SR) software. In phase II, the interpreter produces target speech supported by the SR text and its machine translation (MT) output. A group of students performed CACI with their eye movements tracked. In phase I, the participants devoted the majority of their attention to listening and respeaking, with very limited attention distributed to the SR text. However, a positive correlation was found between the percentage of dwell time on the SR text and the quality of respeaking, which suggests that active monitoring could be important. In phase II, the participants devoted more visual attention to the MT text than to the SR text and engaged in deeper and more effortful processing when reading the MT text. We identified a positive correlation between the percentage of dwell time on the MT text and interpreting quality in the L2–L1 direction but not in the L1–L2 direction. These results contribute to our understanding of computer-assisted interpreting and can provide insights for future research and training in this area.

Keywords: computer-assisted interpreting, consecutive interpreting, eye tracking, speech recognition, machine translation
Article outline
  • 1.Introduction
  • 2.Literature review
    • 2.1Eye movements in reading
    • 2.2Eye movements in respeaking, sight translation and post-editing
  • 3.The present study
    • 3.1Participants
    • 3.2Apparatus
    • 3.3Procedure
    • 3.4Data and analysis
  • 4.Results
    • 4.1Visual processing in the respeaking phase
    • 4.2Visual processing in the production phase
  • 5.Discussion
  • 6.Conclusion
  • References
This content is being prepared for publication; it may be subject to changes.
References
Barr, D. J., Levy, R., Scheepers, C. & Tily, H. J.
(2013) Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language 68 (3), 255–278. DOI logoGoogle Scholar
Bates, D., Mächler, M., Bolker, B. & Walker, S.
(2015) Fitting linear mixed-effects models using lme4. Journal of Statistical Software 67 1, 1–48. DOI logoGoogle Scholar
Carl, M., Dragsted, B., Elming, J., Hardt, D. & Jakobsen, A. L.
(2011) The process of post-editing: A pilot study. In A. L. Jakobsen, M. Carl, B. Sharp & M. Zock (Eds.), Proceedings of the 8th International NLPCS Workshop. Frederiksberg: Samfundslitteratur, 131–142.Google Scholar
Chafe, W.
(1985) Linguistic differences produced by differences between speaking and writing. In D. R. Olson, A. Hildyard & N. Torrance (Eds.), Literacy, language, and learning: The nature and consequences of reading and writing. Cambridge: Cambridge University Press, 105–122.Google Scholar
Chen, S.
(2017) Note-taking in consecutive interpreting: New data from pen recording. Translation and Interpreting 9 (1), 4–23. DOI logoGoogle Scholar
(2020a) The impact of directionality on the process and product in consecutive interpreting between Chinese and English: Evidence from pen recording and eye tracking. The Journal of Specialised Translation 34 1, 100–117.Google Scholar
Chen, S. & Kruger, J.-L.
(2023) The effectiveness of computer-assisted interpreting: A preliminary study based on English–Chinese consecutive interpreting. Translation and Interpreting Studies 18 (3), 399–420. DOI logoGoogle Scholar
under review). A computer-assisted consecutive interpreting mode: Training and evaluation.
Chmiel, A., Janikowski, P. & Cieślewicz, A.
(2020) The eye or the ear? Source language interference in sight translation and simultaneous interpreting. Interpreting 22 (2), 187–210. DOI logoGoogle Scholar
(2022) Reading patterns, reformulation and eye-voice span (IVS) in sight translation. Translation and Interpreting Studies 18 (2), 213–234. DOI logoGoogle Scholar
Cop, U., Drieghe, D. & Duyck, W.
(2015) Eye movement patterns in natural reading: A comparison of monolingual and bilingual reading of a novel. PLoS ONE 10 (8), e0134008. DOI logoGoogle Scholar
Dam, H. V. & Engberg, J.
(2006) Assessing accuracy in consecutive interpreting: a comparison of semantic network analyses and intuitive assessments. In C. Heine, K. Schubert & H. Gerzymisch-Arbogast (Eds.), Text and translation: Theory and methodology of translation. Tübingen: Narr Francke Attempto Verlag, 215–234.Google Scholar
Davitti, E. & Sandrelli, A.
(2020) Embracing the complexity: A pilot study on interlingual respeaking. Journal of Audiovisual Translation 3 (2), 103–139. DOI logoGoogle Scholar
Demareva, V. & Edeleva, Y.
(2020) Eye-tracking based L2 detection: Universal and specific eye movement patterns in L1 and L2 reading. Procedia Computer Science 169 1, 673–676. DOI logoGoogle Scholar
Downie, J.
(2023) Where is it all going? Technology, economic pressures and the future of interpreting. In G. Corpas Pastor & B. Defrancq (Eds.), Interpreting technologies – current and future trends. Amsterdam: John Benjamins, 277–301. DOI logoGoogle Scholar
Dragsted, B. & Hansen, I. G.
(2009) Exploring translation and interpreting hybrids. The case of sight translation. Meta 54 (3), 588–604. DOI logoGoogle Scholar
Eskenazi, M. A.
(2023) Best practices for cleaning eye movement data in reading research. Behavior Research Methods. DOI logoGoogle Scholar
Fantinuoli, C.
(2023) Towards AI-enhanced computer-assisted interpreting. In G. Corpas Pastor & B. Defrancq (Eds), Interpreting technologies – current and future trends. Amsterdam: John Benjamins, 46–71. DOI logoGoogle Scholar
Gerver, D.
(1969/2002) The effects of source language presentation rate on the performance of simultaneous conference interpreters. In F. Pöchhacker & M. Shlesinger (Eds.), The interpreting studies reader. London and New York: Routledge, 53–66.Google Scholar
Han, C.
(2018) Latent trait modelling of rater accuracy of formative peer assessment of English-Chinese consecutive interpreting. Assessment & Evaluation in Higher Education 43(6), 979–994.Google Scholar
Ho, C-E.
(2017) An integrated eye-tracking study into the cognitive process of English–Chinese sight translation: Impacts of training and experience. PhD dissertation, National Taiwan Normal University.
Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H. & van de Weijer, J.
(2011) Eye tracking: A comprehensive guide to methods and measures. Oxford: Oxford University Press.Google Scholar
Huang, C-C.
(2011) Tracking eye movements in sight translation. MA thesis, National Taiwan Normal University.
Koglin, A.
(2015) An empirical investigation of cognitive effort required to post-edit machine translated metaphors compared to the translation of metaphors. Translation and Interpreting 7 (1), 126–141.Google Scholar
Kuperman, V., Siegelman, N., Schroeder, S., Acartürk, C., Alexeeva, S., Amenta, S. … Usal, K. A.
(2022) Text reading in English as a second language: Evidence from the Multilingual Eye-Movements Corpus. Studies in Second Language Acquisition 45 (1), 3–37. DOI logoGoogle Scholar
Lijewska, A., Chmiel, A. & Inhoff, A. W.
(2022) Stages of sight translation: Evidence from eye movements. Applied Psycholinguistics 43 1, 997–1018. DOI logoGoogle Scholar
Moorkens, J., O’Brien, S., da Silva, I. A. L., Fonseca, N. B. d. L. & Alves, F.
(2015) Correlations of perceived post-editing effort with measurements of actual effort. Machine Translation 29 1, 267–284. DOI logoGoogle Scholar
O’Brien, S.
(2011) Towards predicting post-editing productivity. Machine Translation 25 (3), 197–215. DOI logoGoogle Scholar
Pan, J., Yan, M. & Laubrock, J.
(2017) Perceptual span in oral reading: The case of Chinese. Scientific Studies of Reading 21 (3), 254–263. DOI logoGoogle Scholar
Parshina, O., Laurinavichyute, A. K. & Sekerina, I. A.
(2021) Eye-movement benchmarks in Heritage Language reading. Bilingualism: Language and Cognition 24 (1), 69–82. DOI logoGoogle Scholar
Pisani, E. & Fantinuoli, C.
(2021) Measuring the impact of automatic speech recognition on number rendition in simultaneous interpreting. In C. Wang & B. Zheng (Eds.), Empirical studies of translation and interpreting. New York: Routledge, 181–197. DOI logoGoogle Scholar
R Core Team
(2023) R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing.Google Scholar
Rayner, K.
(1998) Eye movements in reading and information processing: 20 years of research. Psychological Bulletin 124 (3), 372–422. DOI logoGoogle Scholar
(2009) Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology 62 (8), 1457–1506. DOI logoGoogle Scholar
Rinne, J. O., Tommola, J., Laine, M., Krause, B. J., Schmidt, D., Kaasinen, V., Teräs, M., Sipilä, H. & Sunnari, M.
(2000) The translating brain: Cerebral activation patterns during simultaneous interpreting. Neuroscience Letters 294 (2), 85–88. DOI logoGoogle Scholar
Romero-Fresco, P.
(2011) Subtitling through speech recognition: Respeaking. Manchester: Routledge.Google Scholar
(2015) Respeaking. In F. Pöchhacker (Ed.), Routledge encyclopedia of interpreting studies. Abingdon: Routledge, 350–351.Google Scholar
Romero-Fresco, P. & Martínez, J.
(2015) Accuracy rate in live subtitling: The NER model. In R. Baños & J. Díaz-Cintas (Eds.), Audiovisual translation in a global context: Mapping an ever-changing landscape. London: Palgrave Macmillan, 28–50. DOI logoGoogle Scholar
Scott, S. D., Sharpe, H., O’Leary, K., Dehaeck, U., Hindmarsh, K., Moore, J. G. & Osmond, M. H.
(2009) Court reporters: A viable solution for the challenges of focus group data collection? Qualitative Health Research 19 (1), 140–146. DOI logoGoogle Scholar
Shreve, G. M., Lacruz, I. & Angelone, E.
(2010) Cognitive effort, syntactic disruption, and visual interference in a sight translation task. In G. M. Shreve & E. Angelone (Eds.). Translation and cognition. Amsterdam and Philadelphia: John Benjamins, 63–84. DOI logoGoogle Scholar
Sun, F.
(1993) Eye movements in reading Chinese: Paragraphs, single characters and pinyin. In S. F. Wright & R. Groner (Eds.), Facets of dyslexia and its remediation. Amsterdam: North-Holland, 245–255. DOI logoGoogle Scholar
Sun, F., Morita, M. & Stark, L. W.
(1985) Comparative patterns of reading eye movement in Chinese and English. Perception and Psychophysics 37 1, 502–506. DOI logoGoogle Scholar
Szarkowska, A., Dutka, Ł., Szychowska, A. & Pilipczuk, O.
(2018) Visual attention distribution in intralingual respeaking: An eye-tracking study. In C. Walker & F. M. Federici (Eds.), Eye tracking and multidisciplinary studies on translation. Amsterdam: John Benjamins, 185–201. DOI logoGoogle Scholar
Toral, A., Wieling, M. & Way, A.
(2018) Post-editing effort of a novel with statistical and neural machine translation. Frontiers in Digital Humanities 5 1, 1–11. DOI logoGoogle Scholar
White, S. J., Warrington, K. L., McGowan, V. A. & Paterson, K. B.
(2015) Eye movements during reading and topic scanning: Effects of word frequency. Journal of Experimental Psychology: Human Perception and Performance 41 (1), 233–248.Google Scholar