Cover not available
Article published In:
FORUM
Vol. 21:1 (2023) ► pp.139–162

Cognitive effort in human translation and machine translation post-editing processes

A holistic and phased view

| Faculty of Languages and Linguistics, Universiti Malaya
ORCID logo | Faculty of Languages and Linguistics, Universiti Malaya

Due to the inaccuracies of Machine Translation (MT), Post-Editing (PE) is inevitable. This poses questions over whether human effort to polish an MT is worthwhile or whether it would be more efficient to translate manually. However, to date, fewer attempts have been made to compare the cognitive effort in the PE process and the sub-phases (orientation, drafting, and revision) of PE with that of Human Translation (HT). To fill this gap, the current study aims to investigate and compare cognitive effort in HT and PE processes in translation from Chinese to English. Data were collected via eye-tracking and keyboard-logging approaches from 25 participants recruited to fulfil three HT and three PE tasks respectively. The comparison of cognitive effort was made from the processes of HT and PE, and their different sub-phases. The study reveals a significant difference in cognitive effort, orientation duration, and drafting duration between HT and PE.

Keywords: machine translation, post-editing, cognitive effort, translation process
Article outline
  • 1.Introduction
  • 2.Related research
    • 2.1Cognitive effort in the human translation process
    • 2.2PE effort
    • 2.3Pause
  • 3.Method
    • 3.1Research design
    • 3.2Participants
    • 3.3Materials
    • 3.4Data quality
    • 3.5Data analysis
  • 4.Findings
    • 4.1Cognitive effort in HT and PE: Whole process
      • 4.1.1Total task time
      • 4.1.2Total fixation duration
      • 4.1.3Fixation count
      • 4.1.4Pauses
      • 4.1.5Keyboard activities
    • 4.2Cognitive effort in HT and PE: Phased view
      • 4.2.1Orientation
      • 4.2.2Drafting
      • 4.2.3Revision
  • 5.Discussion and conclusion
  • Acknowledgements
  • Notes
  • References
Published online: 6 July 2023
DOI logo
https://doi.org/10.1075/forum.22009.wan
References
Alves, Fabio, Arlene Koglin, Bartolomé Mesa-Lao, Mercedes García Martínez, Norma B. de Lima Fonseca, Arthur de Melo Sá, José Luiz Gonçalves, Karina Sarto Szpak, Kyoko Sekino, and Marceli Aquino
2016 “Analysing the Impact of Interactive Machine Translation on Post-Editing Effort.” In New Directions in Empirical Translation Process Research, edited by Michael Carl, Srinivas Bangalore, and Moritz Schaeffer, 1st ed., 77–94. Cham: Springer. DOI logoGoogle Scholar
Bradley, James V.
1958 “Complete Counterbalancing of Immediate Sequential Effects in a Latin Square Design.” Journal of the American Statistical Association 53 (282): 525–28. DOI logoGoogle Scholar
Carl, Michael
2012 “The CRITT TPR-DB 1.0: A Database for Empirical Human Translation Process Research.” In Workshop on Post-Editing Technology and Practice, 9–18. San Diego, California, USA: Association for Machine Translation in the Americas. [URL]
Carl, Michael, Barbara Dragsted, Jakob Elming, Daniel Hardt, and Arnt Lykke Jakobsen
2011 “The Process of Post-Editing: A Pilot Study.” Copenhagen Studies in Language 411: 131–42.Google Scholar
Carl, Michael, Barbara Dragsted, and Arnt Lykke Jakobsen
2011 “On the Systematicity of Human Translation Processes.” In Tralogy 2011. Translation Careers and Technologies: Convergence Points for the Future. Paris, France. [URL]
Carl, Michael, and Martin Kay
2011 “Gazing and Typing Activities during Translation: A Comparative Study of Translation Units of Professional and Student Translators.” Meta 56 (4): 952–75. DOI logoGoogle Scholar
Conklin, Kathy, Ana Pellicer-Sánchez, and Gareth Carrol
2018 Eye-Tracking: A Guide for Applied Linguistics Research. Cambridge, UK: Cambridge University Press. DOI logoGoogle Scholar
Cumbreno, Cristina, and Nora Aranberri
2019 “Comparison of Temporal, Technical and Cognitive Dimension Measurements for Post-Editing Effort.” In Second MEMENTO Workshop on Modelling Parameters of Cognitive Effort in Translation Production, 5–6. Dublin, Ireland.Google Scholar
Daems, Joke, Sonia Vandepitte, Robert J. Hartsuiker, and Lieve Macken
2017 “Translation Methods and Experience: A Comparative Analysis of Human Translation and Post-Editing with Students and Professional Translators.” Meta 62 (2): 245–70. DOI logoGoogle Scholar
Dragsted, Barbara
2010 “Coordination of Reading and Writing Processes in Translation.” In Translation and Cognition, edited by Gregory M. Shreve and Erik Angelone, 41–61. Philadelphia, USA: John Benjamins Publishing. DOI logoGoogle Scholar
Dragsted, Barbara, and Michael Carl
2013 “Towards a Classification of Translation Styles Based on Eye-Tracking and Keylogging Data.” Journal of Writing Research 5 (1): 133–58. DOI logoGoogle Scholar
Durban, Chris
2011 “Translation – Getting It Right. A Guide to Buying Translation.” American Translators Association. Accessed April 22, 2022. [URL]
Ferreira, Aline, John Wayne Schwieter, Alexandra Gottardo, and Jefferey Jones
2016 “Cognitive Effort in Direct and Inverse Translation Performance: Insight from Eye-Tracking Technology.” Cadernos de Tradução 36 (3): 60–80. DOI logoGoogle Scholar
Green, Spence, Jeffrey Heer, and Christopher D. Manning
2013 “The Efficacy of Human Post-Editing for Language Translation.” In SIGCHI Conference on Human Factors in Computing Systems, 439–48. Paris, France. DOI logoGoogle Scholar
Guerberof-Arenas, Ana
2012 Productivity and Quality in the Post-Editing of Outputs from Translation Memories and Machine Translation. PhD diss. Universitat Rovira I Virgili.
2014 “The Role of Professional Experience in Post-Editing from a Quality and Productivity Perspective.” In Post-Editing of Machine Translation: Processes and Applications, edited by Laura Winther Balling, Michael Carl, Sharon O’Brien, Michel Simard, and Specia Lucia, 51–77. Newcastle-upon-Tyne, UK: Cambridge Scholars Publishing.Google Scholar
Hvelplund, Kristian Tangsgaard
2011 Allocation of Cognitive Resources in Translation. An Eye-Tracking and Key-Logging Study.Google Scholar
2014 “Eye Tracking and the Translation Process: Reflections on the Analysis and Interpretation of Eye-Tracking Data.” MonTI Special Issue – Minding Translation, 201–23. DOI logoGoogle Scholar
Jakobsen, Arnt Lykke
2002 “Translation Drafting by Professional Translators and by Translation Students.” Copenhagen Studies in Language 271: 191–204.Google Scholar
Jia, Yanfang, Michael Carl, and Xiangling Wang
2019 “Post-Editing Neural Machine Translation versus Phrase-Based Machine Translation for English–Chinese.” Machine Translation 33 (1): 9–29. DOI logoGoogle Scholar
Just, M., and P. Carpenter
1976 “Eye Fixations and Cognitive Processes.” Cognitive Psychology, 8 (4): 441–80. DOI logoGoogle Scholar
Just, Marcel Adam, and Patricia A. Carpenter
1980 “A Theory of Reading: From Eye Fixations to Comprehension.” Psychological Review 871: 329–54. DOI logoGoogle Scholar
Koglin, Arlene
2015 “An Empirical Investigation of Cognitive Effort Required to Post-Edit Machine Translated Metaphors Compared to the Translation of Metaphors.” Translation & Interpreting 7 (1): 126–41. DOI logoGoogle Scholar
Koglin, Arlene, and Rossana Cunha
2019 “Investigating the Post-Editing Effort Associated with Machine-Translated Metaphors: A Process-Driven Analysis.” Journal of Specialised Translation 311: 38–59.Google Scholar
Koponen, Maarit
2012 “Comparing Human Perceptions of Post-Editing Effort with Post-Editing Operations.” In Seventh Workshop on Statistical Machine Translation, 181–90. Montreal, Canada: Association for Computational Linguistics. [URL]
2016 “Is Machine Translation Post-Editing Worth the Effort? A Survey of Research into Post-Editing and Effort.” Journal of Specialised Translation 251: 131–48.Google Scholar
Koponen, Maarit, Wilker Aziz, Luciana Ramos, and Lucia Specia
2012 “Post-editing time as a measure of cognitive effort.“ In AMTA 2012 Workshop on Post-editing Technology and Practice (WPTP). San Diego, California: Association for Machine Translation in the Americas. [URL]
Krings, Hans P.
2001 Repairing Texts: Empirical Investigations of Machine Translation Post-Editing Processes. Kent, Ohio, USA: Kent State University Press.Google Scholar
Lacruz, Isabel, Michael Carl, Masaru Yamada, and Akiko Aizawa
2016 “Pause Metrics and Machine Translation Utility.” In The 22nd Annual Meeting of the Association for Natural Language Processing, NLP 2016, 1213–16. Sendai, Japan: The Association for Natural Language Processing.Google Scholar
Lacruz, Isabel, Michael Denkowski, and Alon Lavie
2014 “Cognitive Demand and Cognitive Effort in Post-Editing.” In Proceedings of the 11th Conference of the Association for Machine Translation in the Americas, 73–84. Vancouver, Canada: Association for Machine Translation in the Americas. [URL]
Lacruz, Isabel, and Gregory M. Shreve
2014 “Pauses and Cognitive Effort in Post-Editing.” In Post-Editing of Machine Translation: Processes and Applications, edited by Laura Winther Balling, Michael Carl, and Sharon O’Brien, 246–73. Newcastle upon Tyne, UK: Cambridge Scholars Publishing.Google Scholar
Lacruz, Isabel, Gregory M. Shreve, and Erik Angelone
2012 “Average Pause Ratio as an Indicator of Cognitive Effort in Post-Editing: A Case Study.” In Workshop on Post-Editing Technology and Practice. San Diego, California, USA: Association for Machine Translation in the Americas. [URL]
Läubli, Samuel, Chantal Amrhein, Patrick Düggelin, Beatriz Gonzalez, Alena Zwahlen, and Martin Volk
2019 “Post-Editing Productivity with Neural Machine Translation: An Empirical Assessment of Speed and Quality in the Banking and Finance Domain.” ArXiv Preprint. arxiv.org/abs/1906.01685.Google Scholar
Martín, Ricardo Muñoz
2010 “On Paradigms and Cognitive Translatology.” In Translation and Cognition, edited by Gregory M. Shreve and Erik Angelone, 169–87. Philadelphia, Pa, USA: John Benjamins Publishing. DOI logoGoogle Scholar
Mesa-Lao, Bartolomé
2014 “Gaze Behaviour on Source Texts: An Exploratory Study Comparing Translation and Post-Editing.” In Post-Editing of Machine Translation: Processes and Applications, 219–45. Cambridge, UK: Cambridge Scholars Publishing.Google Scholar
Nitzke, Jean, and Katharina Oster
2016 “Comparing Translation and Post-Editing: An Annotation Schema for Activity Units.” In New Directions in Empirical Translation Process Research, edited by Michael Carl, Srinivas Bangalore, and Moritz Schaeffer, 293–308. DOI logoGoogle Scholar
O’Brien, Sharon
2006 “Pauses as Indicators of Cognitive Effort in Post-Editing Machine Translation Output.” Across Languages and Cultures 7 (1): 1–21. DOI logoGoogle Scholar
2007 “An Empirical Investigation of Temporal and Technical Post-Editing Effort.” Translation and Interpreting Studies 2 (1): 83–136. DOI logoGoogle Scholar
Pellicer-Sánchez, Ana
2016 “Incidental L2 Vocabulary Acquisition from and While Reading: An Eye-Tracking Study.” Studies in Second Language Acquisition 381: 97–130. DOI logoGoogle Scholar
Plitt, Mirko, and François Masselot
2010 “A Productivity Test of Statistical Machine Translation Post-Editing in a Typical Localisation Context.” The Prague Bulletin of Mathematical Linguistics 931: 7–16. DOI logoGoogle Scholar
R Core Team
2021 “R: A Language and Environment for Statistical Computing.” Vienna, Austria: R Foundation for Statistical Computing. [URL]
Richardson, John T. E.
2018 “The Use of Latin-Square Designs in Educational and Psychological Research.” Educational Research Review 241: 84–97. DOI logoGoogle Scholar
Sanchez-Torron, Marina, and Philipp Koehn
2016 “Machine Translation Quality and Post-Editor Productivity.” In AMTA 2016 – The Twelfth Conference of the Association for Machine Translation in the Americas, edited by Spence Green and Lane Schwartz, 11:16–26. Austin, Texas, USA: Association for Machine Translation in the Americas. [URL]
Schaeffer, Moritz, Michael Carl, Isabel Lacruz, and Akiko Aizawa
2016 “Measuring Cognitive Translation Effort with Activity Units.” Baltic Journal of Modern Computing 4 (2): 331–45.Google Scholar
Screen, Benjamin
2017 “Productivity and Quality When Editing Machine Translation and Translation Memory Outputs: An Empirical Analysis of English to Welsh Translation.” Studia Celtica Posnaniensia 2 (1): 119–42. DOI logoGoogle Scholar
Sjørup, Annette Camilla
2013 Cognitive Effort in Metaphor Translation: An Eye-Tracking and Key-Logging Study. PhD diss. Copenhagen Business School.
Sousa, Sheila C. M. de, Wilker Aziz, and Lucia Specia
2011 “Assessing the Post-Editing Effort for Automatic and Semi-Automatic Translations of DVD Subtitles.” In Recent Advances in Natural Language Processing, edited by Ruslan Mitkov and Galia Angelova, 97–103. Hissar, Bulgaria.Google Scholar
Stasimioti, Maria, and Vilelmini Sosoni
2020 “Translation vs Post-Editing of NMT Output: Measuring Effort in the English-Greek Language Pair.” In 14th Conference of the Association for Machine Translation in the Americas,1st Workshop on Post-Editing in Modern-Day Translation, 109–24.Google Scholar
2021 “Investigating Post-Editing: A Mixed-Methods Study with Experienced and Novice Translators in the English-Greek Language Pair.” In Translation, Interpreting, Cognition: The Way out of the Box, edited by Tra&Co Group, 79–104. Berlin, Germany: Language Science Press. DOI logoGoogle Scholar
Sun, Sanjun, and Gregory M. Shreve
2014 “Measuring Translation Difficulty.” Target 26 (1): 98–127. DOI logoGoogle Scholar
Sung, Yao-Ting, Tao-Hsing Chang, Wei-Chun Lin, Kuan-Sheng Hsieh, and Kuo-En Chang
2016 “CRIE: An Automated Analyzer for Chinese Texts.” Behavior Research Methods 48 (4): 1238–51. DOI logoGoogle Scholar
Vanroy, Bram, Orphée De Clercq, and Lieve Macken
2019 “Correlating Process and Product Data to Get an Insight into Translation Difficulty.” Perspectives: Studies in Translation Theory and Practice 27 (6): 924–41. DOI logoGoogle Scholar
Vieira, Lucas Nunes
2014 “Indices of Cognitive Effort in Machine Translation Post-Editing.” Machine Translation 281: 187–216. DOI logoGoogle Scholar
Whyatt, Bogusława
2019 “In Search of Directionality Effects in the Translation Process and in the End Product.” Translation, Cognition & Behavior 2 (1): 79–100. DOI logoGoogle Scholar
Whyatt, Bogusława, Katarzyna Stachowiak, and Marta Kajzer-Wietrzny
2016 “Similar and Different: Cognitive Rhythm and Effort in Translation and Paraphrasing.” Poznan Studies in Contemporary Linguistics 52 (2): 175–208. DOI logoGoogle Scholar
Whyatt, Bogusława, Olga Witczak, and Ewa Tomczak
2021 “Information Behaviour in Bidirectional Translators: Focus on Online Resources.” Interpreter and Translator Trainer 15 (2): 154–71. DOI logoGoogle Scholar
Zhechev, Ventsislav
2012 “Machine Translation Infrastructure and Post-Editing Performance at Autodesk.” In Workshop on Post-Editing Technology and Practice. San Diego, California: Association for Machine Translation in the Americas. [URL]