Cover not available
Part of
Translation in Transition: Human and machine intelligence
Edited by Isabel Lacruz
[American Translators Association Scholarly Monograph Series XX] 2023
► pp. 57–82

Chapter 4
An eye-tracking study of productivity and effort in Chinese-to-English translation and post-editing

| Shandong University of Finance and Economics
| Guangdong University of Foreign Studies
| Kent State University
| Guangzhou University of Chinese Medicine
| Beijing Foreign Studies University
| Guangdong University of Finance | Guangdong University of Foreign Studies
| Guangzhou Municipal Foreign Affairs Office

For several language pairs, an emerging consensus finds that post-editing of machine translations is faster and less cognitively effortful than from-scratch human translation, resulting in increased translator productivity and decreased translator fatigue. These benefits have yet to be robustly established in some language pairs that are linguistically and culturally remote with very different writing systems. We carry out a systematic Chinese-to-English study using keystroke logger timing measures and eye-tracking measures of cognitive effort, taking into account translator education levels, different source text domains, and quality of the translation product. We observe significant post-editing productivity gains for more highly educated participants and for more straightforward and less technical texts. Measures of cognitive effort show significantly reduced cognitive effort in post-editing.

Keywords: cognitive effort, post-editing, human translation, eye-tracking, key-logging, Chinese-to-English
Article outline
  • 1.Introduction
  • 2.Effort in post-editing
    • 2.1General considerations on post-editing
    • 2.2Definition and measurement of effort in post-editing and translation
  • 3.Methodology
    • 3.1Overview
    • 3.2Participants
    • 3.3Materials
    • 3.4Apparatus
    • 3.5Procedure
      • 3.5.1Environment
      • 3.5.2Task execution
  • 4.Results and discussion
    • 4.1Data quality
    • 4.2Analyses
      • 4.2.1Average processing time
        • Discussion
      • 4.2.2Quality as a potential confounding variable
      • 4.2.3Pupil dilation
        • Discussion
      • 4.2.4Average fixation duration
        • Average fixation duration in source and target text combined
        • Discussion
        • Average fixation duration in source text only
        • Average fixation duration in target text only
        • Discussion (ST only and TT only)
  • 5.Conclusion
  • Acknowledgements
  • References
Published online: 26 July 2023
DOI logo
https://doi.org/10.1075/ata.xx.04sun
References
Albrecht, Joshua S., Rebecca Hwa, and G. Elisabeta Marai
2009 “Correcting Automatic Translations through Collaborations between MT and Monolingual Target-language Users.” In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, 60–68. Athens, Greece. DOI logoGoogle Scholar
Balling, Laura W., and Michael Carl
2014 “Production Time across Languages and Tasks: A Large-scale Analysis Using the CRITT Translation Process Database.” In The Development of Translation Competence: Theories and Methodologies from Psycholinguistics and Cognitive Science, ed. by John W. Schwieter, and Aline Ferreira, 239–268. Newcastle: Cambridge Scholar Publishers.Google Scholar
Blatz, John, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueng
2004 “Confidence Estimation for Machine Translation.” Mental Imagery 33: 9–40. DOI logoGoogle Scholar
Callison-Burch, Chris, Philipp Koehn, Christof Monz, Kay Peterson, Mark A. Przybocki, and Omar F. Zaidan
2010 “Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation.” In Joint Fifth Workshop on Statistical Machine Translation and Metrics MATR, ed. by Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, and Omar Zaidan, 17–53. Association for Computational Linguistics.Google Scholar
Callison-Burch, Chris, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia
2012 “Findings of the 2012 Joint Workshop on Statistical Machine Translation.” In Proceedings of the Seventh Workshop on Statistical Machine Translation, ed. by Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia, 10–51. Association for Computational Linguistics.Google Scholar
Carl, Michael, Srinivas Bangalore, and Moritz Schaeffer
2016 New Directions in Empirical Translation Process Research: Exploring the CRITT TPRDB. Heidelberg: Springer. DOI logoGoogle Scholar
Carl, Michael, Barbara Dragsted, Jakob Elming, Daniel Hardt, and Arnd Lykke Jakobsen
2011 “The Process of Post-editing: A Pilot Study.” In Proceedings of the 8th International NLPSC Workshop. Special Theme: Human-machine Interaction in Translation, ed. by Bernadette Sharp, Michael Zock, Michael Carl, and Arnt Lykke Jakobsen, 131–142. Frederiksberg: Samfundslitteratur.Google Scholar
Carl, Michael, Silke Gutermuth, and Silvia Hansen-Schirra
2015 “Post-editing machine translation: A usability test for professional translation settings..” In Psycholinguistic and Cognitive Inquiries into Translation and Interpreting, ed. by Aline Ferreira, and John W. Schwieter, 145–174. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Da Silva, Igor Antônio Lourenço, Marcia Schmaltz, Fabio Alves, Adriana Pagano, Wong, Lidia Chao, Ana Luisa Varani Leal, Paulo Quaresma, and Ciao Garcia
2015 “Translating and Post-editing in the Chinese-Portuguese Language Pair: Insights from an Exploratory Study of Key Logging and Eye Tracking.” Translation Spaces 4 (1): 145–169.Google Scholar
Daems, Joke, Sonia Vandepitte, Robert J. Hartsuiker, and Lieve Macken
2017 “Translation Methods and Experience: A Comparative Analysis of Human Translation and Post-editing with Students and Professional Translators.” Meta 62 (2): 245–270. DOI logoGoogle Scholar
Doherty, Stephen
2012 Investigating the Effects of Controlled Language on the Reading and Comprehension of Machine Translated Texts: A Mixed-methods Approach. PhD Dissertation, Dublin City University.
Feng, Quangong, and Qiliang Cui
2016 “Research Focuses and Trends in Post-editing of Machine Translation.” Shanghai Journal of Translators 6: 67–89.Google Scholar
Fiederer, Rebecca, and Sharon O’Brien
2009 “Quality and Machine Translation: A Realistic Objective?” The Journal of Specialised Translation 11: 52–74.Google Scholar
García, Ignacio
2010 “Is Machine Translation Ready Yet?” Target 22 (1): 7–21. DOI logoGoogle Scholar
Green, Spence, Jeffrey Heer, and Christopher D. Manning
2013 “The Efficacy of Human Post-editing for Language Translation.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ed. by Joseph A. Konstan, Ed H. Chi, and Kristina Höök, 439–448. New York: ACM press. DOI logoGoogle Scholar
Guerberof, Ana
2009 “Productivity and Quality in the Post-Editing of Outputs from Translation Memories and Machine Translation.” Localisation Focus: The International Journal of Localisation 7 (1): 11–21.Google Scholar
2014 “Correlations Between Productivity and Quality When Post-Editing in a Professional Context.” Machine Translation 28: 165–186. DOI logoGoogle Scholar
Hu, Chang, Philip Resnik, Yakov Kronrod, Vladimir Eidelman, Olivia Buzek, and Benjamin B. Bederson
2011 “The Value of Monolingual Crowdsourcing in A Real-world Translation Scenario: Simulation Using Haitian Creole Emergency SMS Messages.” In Proceedings of the Sixth Workshop on Statistical Machine Translation, ed. by Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar F. Zaidan, 399–404. Stroudsburg. PA: Association for Computational Linguistics.Google Scholar
Hvelplund, Kristian Tangsgaard
2011 Allocation of Cognitive Resources in Translation: An Eye-tracking and Key-logging Study. PhD Dissertation, Copenhagen Business School.
Hyönä, Jukka, Jorma Tommola, and Anna-Mari Alaja
1995 “Pupil Dilation as a Measure of Processing Load in Simultaneous Interpretation and Other Language Tasks.” Quarterly Journal of Experimental Psychology 48 (3): 598–612. DOI logoGoogle Scholar
Iqbal, Shamsi T., Piotr D. Adamczyk, Sam Zheng, and Brian P. Bailey
2005 “Towards an Index of Opportunity: Understanding Changes in Mental Workload during Task Execution.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ed. by Joseph A. Konstan, Ed H. Chi, and Kristina Höök, 311–320. New York: ACM Press. DOI logoGoogle Scholar
Just, Marcel A., and Patricia A. Carpenter
1980 “A Theory of Reading: From Eye Fixations to Comprehension.” Psychological Review 87: 329–354. DOI logoGoogle Scholar
Koby, Geoffrey S., and Isabel Lacruz
2017 “Translator Quality-Translation Quality: Empirical Approaches to Assessment and Evaluation.” Linguistica Antverpiensia New Series – Themes in Translation Studies 16 (January). Antwerp, Belgium.Google Scholar
Koehn, Philipp
2010 “Enabling Monolingual Translators: Post-editing vs. Options.” In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, ed. by Ronald Kaplan, Jill Burstein, Mary Harper, and Gerald Penn, 537–545. Los Angeles, California. Association for Computational Linguistics.Google Scholar
Koglin, Arlene
2015 “An Empirical Investigation of Cognitive Effort Required to Post-edit Machine Translated Metaphors Compared to the Translation of Metaphors.” Translation and Interpreting 7 (1): 126–141.Google Scholar
Koponen, Maarit
2016 “Is Machine Translation Post-editing Worth the Effort? A Survey of Research into Post-editing and Effort.” The Journal of Specialised Translation 25: 131–148.Google Scholar
Koponen, Maarit, and Leena Salmi
2015 “On the Correctness of Machine Translation: A Machine Translation Post-editing Task.” The Journal of Specialised Translation 23: 118–136.Google Scholar
Krings, Hans P.
2001 Repairing Texts: Empirical Investigations of Machine Translation Post-editing Processes (ed. Geoffrey S. Koby). Kent, Ohio: Kent State University Press.Google Scholar
Lacruz, Isabel
2017 “Cognitive Effort in Translation, Editing and Post-Editing.” In The Handbook of Translation and Cognition, ed. by John W. Schwieter, and Aline Ferreira, 386–401. Blackwell Handbooks in Linguistics. Malden, MA: John Wiley & Sons. DOI logoGoogle Scholar
Lacruz, Isabel, Michael Denkowski, and Alon Lavie
2014 “Cognitive Effort and Cognitive Demand in Post-editing.” In Proceedings from Third Workshop on Post-Editing Technology and Practice, the 11th Conference of the Association for Machine Translation of the Americas, ed. by Sharon O’Brien, Michel Simard, and Lucia Specia, 73–87. Vancouver, BC.Google Scholar
Lacruz, Isabel, and Riitta Jääskeläinen
2018 Innovation and Expansion in Translation Process Research. Amsterdam/Philadelphia: John Benjamins. DOI logoGoogle Scholar
Lacruz, Isabel, and Gregory M. Shreve
2014 “Pauses and Cognitive Effort in Post-Editing.” In Post-editing of Machine Translation: Processes and Applications, ed. by Sharon O’Brien, Laura Winther Balling, Michael Carl, Michel Simard, and Lucia Specia, 246–273. Newcastle upon Tyne: Cambridge Scholars Publishing.Google Scholar
Lacruz, Isabel, Gregory M. Shreve, and Erik Angelone
2012 “Average Pause Ratio as an Indicator of Cognitive Effort in Post-editing: A Case Study.” In Proceedings of the AMTA 2012 Workshop on Post-editing Technology and Practice (WTTP), ed. by Sharon O’Brien, Michel Simard, and Lucia Specia, 29–38. San Diego: Association for Machine Translation in the Americas.Google Scholar
Lu, Zhi, and Juan Sun
2018 “An Eye-tracking Study of Cognitive Processing in Human Translation and Post-Editing.” Foreign Language Teaching and Research 50 (5): 760–769.Google Scholar
Mesa-Lao, Bartolomé
2014 “Gaze Behavior on Source Texts: An Exploratory Study Comparing Translation and Post-editing.” In Post-editing of Machine Translation: Processes and Applications, ed. by Sharon O’Brien, Laura Winther Balling, Michael Carl, Michel Simard, and Lucia Specia, 219–245. Newcastle: Cambridge Scholar Publishers.Google Scholar
Mitchell, Linda, Johann Roturier, and Sharon O’Brien
2013 “Community-based Post-editing of Machine-translated Content: Monolingual vs. Bilingual.” In Proceedings of Machine Translation Summit XIV Workshop on Post-editing Technology and Practice, ed. by Sharon O’Brien, Michel Simard, and Lucia Specia, 35–43. Nice, France.Google Scholar
Nitzke, Jean, and Silvia Hansen-Shirra
2021 A short guide to post-editing. Berlin: Language Science Press.Google Scholar
O’Brien, Sharon
2006 “Eye-tracking and Translation Memory Matches.” Perspectives Studies in Translatology 14 (3): 185–205.Google Scholar
2008 “Processing Fuzzy Matches in Translation Memory Tools: An Eye Tracking Analysis.” In Looking at Eyes: Eye-tracking Studies of Reading and Translation Processing, ed. by Susanne Göpferich, Arnt Lykke Jakobsen, and Inger M. Mees, 79–101. Copenhagen: Samfundslitteratur Press.Google Scholar
2011 “Towards Predicting Post-editing Productivity.” Machine Translation 25 (3): 197–215. DOI logoGoogle Scholar
Plitt, Mirko, and François Masselot
2010 “A Productivity Test of Statistical Machine Translation Post-editing in a Typical Localisation Context.” The Prague Bulletin of Mathematical Linguistics 93: 7–16. DOI logoGoogle Scholar
Sánchez-Gijón, Pilar, Joss Moorkens, and Andy Way
2019 “Post-editing Neural Machine Translation Versus Translation Memory Segments.” Machine Translation 33: 31–59. DOI logoGoogle Scholar
Sjørup, Annette Camilla
2013 Cognitive Effort in Metaphor Translation: An Eye-tracking and Key-logging Study. PhD Dissertation, Copenhagen Business School.
Snover, Matthew, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul
2006 “A Study of Translation Edit Rate with Targeted Human Annotation.” In Proceedings of Association for Machine Translation in the Americas, 223–231. Cambridge, MA: Association for Machine Translation in the Americas.Google Scholar
Sousa, Sheila C. M., Wilker Aziz, and Lucia Specia
2011 “Assessing the Post-editing Effort for Automatic and Semi-automatic Translations of DVD Subtitles.” In Proceedings of the Recent Advances in Natural Language Processing Conference, ed. by Ruslan Mitkov, and Galia Angelova, 97–103. Hissar, Bulgaria: RANLP 2011 Organising Committee.Google Scholar
Specia, Lucia, Nicola Cancedda, and Marc Dymetman
2010 “A Dataset for Assessing Machine Translation Evaluation Metrics.” In Proceedings of the Seventh Conference on International Language Resources and Evaluation, ed. by Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner, and Daniel Tapias. Valletta, Malta. European Language Resources Association.Google Scholar
Specia, Lucia, Marco Turchi, Nicola Cancedda, Nello Cristianini, and Marc Dymetman
2009 “Estimating the Sentence-level Quality of Machine Translation Systems.” In Proceedings of the 13th Annual Conference of the European Association for Machine Translation, ed. by Lluís Màrquez, and Harold Somers, 28–37. Barcelona: European Association for Machine Translation.Google Scholar
Sun, Sanjun
2019 “Measuring Difficulty in Translation and Post-Editing: A Review.” In Researching cognitive processes of translation. New Frontiers in Translation Studies, ed. by Defeng Li, Victoria Lai Cheng Lei, and Yuanjian He, 139–168. Singapore: Springer. DOI logoGoogle Scholar
Tatsumi, Midori
2009 “Correlation between Automatic Evaluation Metric Scores, Post-editing Speed, and Some Other Factors.” In Proceedings of the MT Summit XII: Posters, 332–339. Ottawa: Association for Machine Translation.Google Scholar
Vieira, Lucas Nunes
2019 “Post-Editing of Machine Translation.” In The Routledge Handbook of Translation and Technology, ed. by Minako O’Hagan, 319–335. New York, NY: Routledge. DOI logoGoogle Scholar
Wagner, Elaine
1985 “Post-editing SYSTRAN, a Challenge for Commission Translators.” Terminologie et Traduction 3: 1–7.Google Scholar
Wang, Xiangling, Tingting Wang, Ricardo Muñoz Martín, and Yanfang Jia
2021 “Investigating Usability in Postediting Neural Machine Translation: Evidence from Translation Trainees’ Self-perception and Performance.” Across Languages and Cultures 22 (1): 100–123. DOI logoGoogle Scholar
Zhechev, Ventsislav
2014 “Analysing the Post-editing of Machine Translation at Autodesk.” In Post-editing of Machine Translation: Processes and Applications, ed. by Sharon O’Brien, Laura Winther Balling, Michael Carl, Michel Simard, and Lucia Specia, 2–13. Newcastle: Cambridge Scholar Publishers.Google Scholar