Cover not available
Part of
Translation in Transition: Human and machine intelligence
Edited by Isabel Lacruz
[American Translators Association Scholarly Monograph Series XX] 2023
► pp. 57–82

Chapter 4
An eye-tracking study of productivity and effort in Chinese-to-English translation and post-editing

| Shandong University of Finance and Economics
| Guangdong University of Foreign Studies
| Kent State University
| Guangzhou University of Chinese Medicine
| Beijing Foreign Studies University
| Guangdong University of Finance | Guangdong University of Foreign Studies
| Guangzhou Municipal Foreign Affairs Office
For several language pairs, an emerging consensus finds that post-editing of machine translations is faster and less cognitively effortful than from-scratch human translation, resulting in increased translator productivity and decreased translator fatigue. These benefits have yet to be robustly established in some language pairs that are linguistically and culturally remote with very different writing systems. We carry out a systematic Chinese-to-English study using keystroke logger timing measures and eye-tracking measures of cognitive effort, taking into account translator education levels, different source text domains, and quality of the translation product. We observe significant post-editing productivity gains for more highly educated participants and for more straightforward and less technical texts. Measures of cognitive effort show significantly reduced cognitive effort in post-editing.
Keywords: cognitive effort, post-editing, human translation, eye-tracking, key-logging, Chinese-to-English
Article outline
  • 1.Introduction
  • 2.Effort in post-editing
    • 2.1General considerations on post-editing
    • 2.2Definition and measurement of effort in post-editing and translation
  • 3.Methodology
    • 3.1Overview
    • 3.2Participants
    • 3.3Materials
    • 3.4Apparatus
    • 3.5Procedure
      • 3.5.1Environment
      • 3.5.2Task execution
  • 4.Results and discussion
    • 4.1Data quality
    • 4.2Analyses
      • 4.2.1Average processing time
        • Discussion
      • 4.2.2Quality as a potential confounding variable
      • 4.2.3Pupil dilation
        • Discussion
      • 4.2.4Average fixation duration
        • Average fixation duration in source and target text combined
        • Discussion
        • Average fixation duration in source text only
        • Average fixation duration in target text only
        • Discussion (ST only and TT only)
  • 5.Conclusion
  • Acknowledgements
  • References
Published online: 26 July 2023
DOI logo
https://doi.org/10.1075/ata.xx.04sun
References (54)
References
Albrecht, Joshua S., Rebecca Hwa, and G. Elisabeta Marai. 2009. “Correcting Automatic Translations through Collaborations between MT and Monolingual Target-language Users.” In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, 60–68. Athens, Greece. DOI logoGoogle Scholar
Balling, Laura W., and Michael Carl. 2014. “Production Time across Languages and Tasks: A Large-scale Analysis Using the CRITT Translation Process Database.” In The Development of Translation Competence: Theories and Methodologies from Psycholinguistics and Cognitive Science, ed. by John W. Schwieter, and Aline Ferreira, 239–268. Newcastle: Cambridge Scholar Publishers.Google Scholar
Blatz, John, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueng. 2004. “Confidence Estimation for Machine Translation.” Mental Imagery 33: 9–40. DOI logoGoogle Scholar
Callison-Burch, Chris, Philipp Koehn, Christof Monz, Kay Peterson, Mark A. Przybocki, and Omar F. Zaidan. 2010. “Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation.” In Joint Fifth Workshop on Statistical Machine Translation and Metrics MATR, ed. by Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, and Omar Zaidan, 17–53. Association for Computational Linguistics.Google Scholar
Callison-Burch, Chris, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. “Findings of the 2012 Joint Workshop on Statistical Machine Translation.” In Proceedings of the Seventh Workshop on Statistical Machine Translation, ed. by Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia, 10–51. Association for Computational Linguistics.Google Scholar
Carl, Michael, Srinivas Bangalore, and Moritz Schaeffer. 2016. New Directions in Empirical Translation Process Research: Exploring the CRITT TPRDB. Heidelberg: Springer. DOI logoGoogle Scholar
Carl, Michael, Barbara Dragsted, Jakob Elming, Daniel Hardt, and Arnd Lykke Jakobsen. 2011. “The Process of Post-editing: A Pilot Study.” In Proceedings of the 8th International NLPSC Workshop. Special Theme: Human-machine Interaction in Translation, ed. by Bernadette Sharp, Michael Zock, Michael Carl, and Arnt Lykke Jakobsen, 131–142. Frederiksberg: Samfundslitteratur.Google Scholar
Carl, Michael, Silke Gutermuth, and Silvia Hansen-Schirra. 2015. “Post-editing machine translation: A usability test for professional translation settings..” In Psycholinguistic and Cognitive Inquiries into Translation and Interpreting, ed. by Aline Ferreira, and John W. Schwieter, 145–174. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Da Silva, Igor Antônio Lourenço, Marcia Schmaltz, Fabio Alves, Adriana Pagano, Wong, Lidia Chao, Ana Luisa Varani Leal, Paulo Quaresma, and Ciao Garcia. 2015. “Translating and Post-editing in the Chinese-Portuguese Language Pair: Insights from an Exploratory Study of Key Logging and Eye Tracking.” Translation Spaces 4 (1): 145–169.Google Scholar
Daems, Joke, Sonia Vandepitte, Robert J. Hartsuiker, and Lieve Macken. 2017. “Translation Methods and Experience: A Comparative Analysis of Human Translation and Post-editing with Students and Professional Translators.” Meta 62 (2): 245–270. DOI logoGoogle Scholar
Doherty, Stephen. 2012. Investigating the Effects of Controlled Language on the Reading and Comprehension of Machine Translated Texts: A Mixed-methods Approach. PhD Dissertation, Dublin City University.
Feng, Quangong, and Qiliang Cui. 2016. “Research Focuses and Trends in Post-editing of Machine Translation.” Shanghai Journal of Translators 6: 67–89.Google Scholar
Fiederer, Rebecca, and Sharon O’Brien. 2009. “Quality and Machine Translation: A Realistic Objective?” The Journal of Specialised Translation 11: 52–74.Google Scholar
García, Ignacio. 2010. “Is Machine Translation Ready Yet?” Target 22 (1): 7–21. DOI logoGoogle Scholar
Green, Spence, Jeffrey Heer, and Christopher D. Manning. 2013. “The Efficacy of Human Post-editing for Language Translation.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ed. by Joseph A. Konstan, Ed H. Chi, and Kristina Höök, 439–448. New York: ACM press. DOI logoGoogle Scholar
Guerberof, Ana. 2009. “Productivity and Quality in the Post-Editing of Outputs from Translation Memories and Machine Translation.” Localisation Focus: The International Journal of Localisation 7 (1): 11–21.Google Scholar
. 2014. “Correlations Between Productivity and Quality When Post-Editing in a Professional Context.” Machine Translation 28: 165–186. DOI logoGoogle Scholar
Hu, Chang, Philip Resnik, Yakov Kronrod, Vladimir Eidelman, Olivia Buzek, and Benjamin B. Bederson. 2011. “The Value of Monolingual Crowdsourcing in A Real-world Translation Scenario: Simulation Using Haitian Creole Emergency SMS Messages.” In Proceedings of the Sixth Workshop on Statistical Machine Translation, ed. by Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar F. Zaidan, 399–404. Stroudsburg. PA: Association for Computational Linguistics.Google Scholar
Hvelplund, Kristian Tangsgaard. 2011. Allocation of Cognitive Resources in Translation: An Eye-tracking and Key-logging Study. PhD Dissertation, Copenhagen Business School.
Hyönä, Jukka, Jorma Tommola, and Anna-Mari Alaja. 1995. “Pupil Dilation as a Measure of Processing Load in Simultaneous Interpretation and Other Language Tasks.” Quarterly Journal of Experimental Psychology 48 (3): 598–612. DOI logoGoogle Scholar
Iqbal, Shamsi T., Piotr D. Adamczyk, Sam Zheng, and Brian P. Bailey. 2005. “Towards an Index of Opportunity: Understanding Changes in Mental Workload during Task Execution.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ed. by Joseph A. Konstan, Ed H. Chi, and Kristina Höök, 311–320. New York: ACM Press. DOI logoGoogle Scholar
Just, Marcel A., and Patricia A. Carpenter. 1980. “A Theory of Reading: From Eye Fixations to Comprehension.” Psychological Review 87: 329–354. DOI logoGoogle Scholar
Koby, Geoffrey S., and Isabel Lacruz. 2017. “Translator Quality-Translation Quality: Empirical Approaches to Assessment and Evaluation.” Linguistica Antverpiensia New Series – Themes in Translation Studies 16 (January). Antwerp, Belgium.Google Scholar
Koehn, Philipp. 2010. “Enabling Monolingual Translators: Post-editing vs. Options.” In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, ed. by Ronald Kaplan, Jill Burstein, Mary Harper, and Gerald Penn, 537–545. Los Angeles, California. Association for Computational Linguistics.Google Scholar
Koglin, Arlene. 2015. “An Empirical Investigation of Cognitive Effort Required to Post-edit Machine Translated Metaphors Compared to the Translation of Metaphors.” Translation and Interpreting 7 (1): 126–141.Google Scholar
Koponen, Maarit. 2016. “Is Machine Translation Post-editing Worth the Effort? A Survey of Research into Post-editing and Effort.” The Journal of Specialised Translation 25: 131–148.Google Scholar
Koponen, Maarit, and Leena Salmi. 2015. “On the Correctness of Machine Translation: A Machine Translation Post-editing Task.” The Journal of Specialised Translation 23: 118–136.Google Scholar
Krings, Hans P. 2001. Repairing Texts: Empirical Investigations of Machine Translation Post-editing Processes (ed. Geoffrey S. Koby). Kent, Ohio: Kent State University Press.Google Scholar
Lacruz, Isabel. 2017. “Cognitive Effort in Translation, Editing and Post-Editing.” In The Handbook of Translation and Cognition, ed. by John W. Schwieter, and Aline Ferreira, 386–401. Blackwell Handbooks in Linguistics. Malden, MA: John Wiley & Sons. DOI logoGoogle Scholar
Lacruz, Isabel, Michael Denkowski, and Alon Lavie. 2014. “Cognitive Effort and Cognitive Demand in Post-editing.” In Proceedings from Third Workshop on Post-Editing Technology and Practice, the 11th Conference of the Association for Machine Translation of the Americas, ed. by Sharon O’Brien, Michel Simard, and Lucia Specia, 73–87. Vancouver, BC.Google Scholar
Lacruz, Isabel, and Riitta Jääskeläinen. 2018. Innovation and Expansion in Translation Process Research. Amsterdam/Philadelphia: John Benjamins. DOI logoGoogle Scholar
Lacruz, Isabel, and Gregory M. Shreve. 2014. “Pauses and Cognitive Effort in Post-Editing.” In Post-editing of Machine Translation: Processes and Applications, ed. by Sharon O’Brien, Laura Winther Balling, Michael Carl, Michel Simard, and Lucia Specia, 246–273. Newcastle upon Tyne: Cambridge Scholars Publishing.Google Scholar
Lacruz, Isabel, Gregory M. Shreve, and Erik Angelone. 2012. “Average Pause Ratio as an Indicator of Cognitive Effort in Post-editing: A Case Study.” In Proceedings of the AMTA 2012 Workshop on Post-editing Technology and Practice (WTTP), ed. by Sharon O’Brien, Michel Simard, and Lucia Specia, 29–38. San Diego: Association for Machine Translation in the Americas.Google Scholar
Lu, Zhi, and Juan Sun. 2018. “An Eye-tracking Study of Cognitive Processing in Human Translation and Post-Editing.” Foreign Language Teaching and Research 50 (5): 760–769.Google Scholar
Mesa-Lao, Bartolomé. 2014. “Gaze Behavior on Source Texts: An Exploratory Study Comparing Translation and Post-editing.” In Post-editing of Machine Translation: Processes and Applications, ed. by Sharon O’Brien, Laura Winther Balling, Michael Carl, Michel Simard, and Lucia Specia, 219–245. Newcastle: Cambridge Scholar Publishers.Google Scholar
Mitchell, Linda, Johann Roturier, and Sharon O’Brien. 2013. “Community-based Post-editing of Machine-translated Content: Monolingual vs. Bilingual.” In Proceedings of Machine Translation Summit XIV Workshop on Post-editing Technology and Practice, ed. by Sharon O’Brien, Michel Simard, and Lucia Specia, 35–43. Nice, France.Google Scholar
Nitzke, Jean, and Silvia Hansen-Shirra. 2021. A short guide to post-editing. Berlin: Language Science Press.Google Scholar
O’Brien, Sharon. 2006. “Eye-tracking and Translation Memory Matches.” Perspectives Studies in Translatology 14 (3): 185–205.Google Scholar
. 2008. “Processing Fuzzy Matches in Translation Memory Tools: An Eye Tracking Analysis.” In Looking at Eyes: Eye-tracking Studies of Reading and Translation Processing, ed. by Susanne Göpferich, Arnt Lykke Jakobsen, and Inger M. Mees, 79–101. Copenhagen: Samfundslitteratur Press.Google Scholar
. 2011. “Towards Predicting Post-editing Productivity.” Machine Translation 25 (3): 197–215. DOI logoGoogle Scholar
Plitt, Mirko, and François Masselot. 2010. “A Productivity Test of Statistical Machine Translation Post-editing in a Typical Localisation Context.” The Prague Bulletin of Mathematical Linguistics 93: 7–16. DOI logoGoogle Scholar
Sánchez-Gijón, Pilar, Joss Moorkens, and Andy Way. 2019. “Post-editing Neural Machine Translation Versus Translation Memory Segments.” Machine Translation 33: 31–59. DOI logoGoogle Scholar
Sjørup, Annette Camilla. 2013. Cognitive Effort in Metaphor Translation: An Eye-tracking and Key-logging Study. PhD Dissertation, Copenhagen Business School.
Snover, Matthew, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. “A Study of Translation Edit Rate with Targeted Human Annotation.” In Proceedings of Association for Machine Translation in the Americas, 223–231. Cambridge, MA: Association for Machine Translation in the Americas.Google Scholar
Sousa, Sheila C. M., Wilker Aziz, and Lucia Specia. 2011. “Assessing the Post-editing Effort for Automatic and Semi-automatic Translations of DVD Subtitles.” In Proceedings of the Recent Advances in Natural Language Processing Conference, ed. by Ruslan Mitkov, and Galia Angelova, 97–103. Hissar, Bulgaria: RANLP 2011 Organising Committee.Google Scholar
Specia, Lucia, Nicola Cancedda, and Marc Dymetman. 2010. “A Dataset for Assessing Machine Translation Evaluation Metrics.” In Proceedings of the Seventh Conference on International Language Resources and Evaluation, ed. by Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner, and Daniel Tapias. Valletta, Malta. European Language Resources Association.Google Scholar
Specia, Lucia, Marco Turchi, Nicola Cancedda, Nello Cristianini, and Marc Dymetman. 2009. “Estimating the Sentence-level Quality of Machine Translation Systems.” In Proceedings of the 13th Annual Conference of the European Association for Machine Translation, ed. by Lluís Màrquez, and Harold Somers, 28–37. Barcelona: European Association for Machine Translation.Google Scholar
Sun, Sanjun. 2019. “Measuring Difficulty in Translation and Post-Editing: A Review.” In Researching cognitive processes of translation. New Frontiers in Translation Studies, ed. by Defeng Li, Victoria Lai Cheng Lei, and Yuanjian He, 139–168. Singapore: Springer. DOI logoGoogle Scholar
Tatsumi, Midori. 2009. “Correlation between Automatic Evaluation Metric Scores, Post-editing Speed, and Some Other Factors.” In Proceedings of the MT Summit XII: Posters, 332–339. Ottawa: Association for Machine Translation.Google Scholar
Vieira, Lucas Nunes. 2019. “Post-Editing of Machine Translation.” In The Routledge Handbook of Translation and Technology, ed. by Minako O’Hagan, 319–335. New York, NY: Routledge. DOI logoGoogle Scholar
Wagner, Elaine. 1985. “Post-editing SYSTRAN, a Challenge for Commission Translators.” Terminologie et Traduction 3: 1–7.Google Scholar
Wang, Xiangling, Tingting Wang, Ricardo Muñoz Martín, and Yanfang Jia. 2021. “Investigating Usability in Postediting Neural Machine Translation: Evidence from Translation Trainees’ Self-perception and Performance.” Across Languages and Cultures 22 (1): 100–123. DOI logoGoogle Scholar
Zhechev, Ventsislav. 2014. “Analysing the Post-editing of Machine Translation at Autodesk.” In Post-editing of Machine Translation: Processes and Applications, ed. by Sharon O’Brien, Laura Winther Balling, Michael Carl, Michel Simard, and Lucia Specia, 2–13. Newcastle: Cambridge Scholar Publishers.Google Scholar