The three measurements for post-editing effort as proposed by Krings (2001) have been adopted by many researchers in subsequent studies and publications. These measurements comprise temporal effort (the speed or productivity rate of post-editing, often measured in words per second at the segment level), technical effort (the number of actual edits performed by the post-editor, sometimes approximated using the Translation Edit Rate metric (Snover et al. 2006), again usually at the segment level), and cognitive effort. Cognitive effort has been measured using think-aloud protocols, pause measurement, and, increasingly, eye-tracking. This chapter provides a review of studies of post-editing effort using eye-tracking, noting the influence of publications by Danks et al. (1997), and O’Brien (2006, 2008), before describing a single study in detail. The detailed study examines whether predicted affort indicators affect post-editing effort and results were previously published as Moorkens et al. (2015). This chapter focuses instead on methodology and the logistics of running an eye-tracking study recording over 70 sessions. Most of the eye-tracking data analysed were unused in the previous publication, and the small amount presented was not described in detail due to space constraints. In this study average fixation count per segment correlates very strongly with temporal effort, and average fixation duration correlates strongly with technical effort, a result that we compare with other studies of post-editing effort.
Article outline
1.Introduction
2.The task of post-editing
3.Eye-tracking measures used in studies of post-editing
2003 “Post-editing.” In Computers and Translation, ed. by Harold Somers, 297–318. Amsterdam: John Benjamins.
ALPAC
1966Languages and machines: computers in translation and linguistics. A report by the Automatic Language Processing Advisory Committee. Washington, DC.
Alves, Fabio, Arlene Koglin, Bartolomé Mesa-Lao, Mercedes García Martínez, Norma B. de Lima Fonseca, Arthur de Melo Sá, José Luiz Gonçalves, Karina Sarto Szpak, Kyoko Sekino, and Marceli Aquino
2016 “Analysing the Impact of Interactive Machine Translation on Post-editing Effort.” In New Directions in Empirical Translation Process Research, ed. by Michael Carl, Srinivas Bangalore, and Moritz Schaeffer, 77–94. Heidelberg: Springer.
Bentivogli, Luisa, Arianna Bisazza, Mauro Cettolo, and Marcello Federico
2016 “Neural versus Phrase-Based Machine Translation Quality: a Case Study.” In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP). Austin, Texas.
Beyer, Robert T.
1965 “Hurdling the Language Barrier.” Physics Today 18 (1): 46–52.
Bruckner, Christine, and Mirko Plitt
2001 “Evaluating the Operational Benefit of Using Machine Translation Output as Translation Memory Input.” In Proceedings of MT Summit VIII, 61–65.
Caffrey, Colm
2009Relevant Abuse? Investigating the Effects of an Abusive Subtitling Procedure on the Perception of TV Anime Using Eye Tracker and Questionnaire. PhD dissertation, Dublin City University.
Carl, Michael, Barbara Dragsted, Jakob Elming, Daniel Hardt, and Arnt Lykke Jakobsen
2011 “The Process of Post-Editing: a Pilot Study.” In Proceedings of NLPSC 2011.
Carl, Michael, Silke Gutermuth, and Silvia Hansen-Schirra
2015 “Post-editing machine translation: Efficiency, strategies, and revision processes in professional translation settings. Psycholinguistic and Cognitive Inquiries into Translation and Interpreting.” In Psycholinguistic and Cognitive Inquiries into Translation and Interpreting, ed. by Aline Ferreira, and John W. Schwieter, 145–174. Amsterdam: John Benjamins.
Castilho, Sheila, Joss Moorkens, Federico Gaspari, Iacer Calixto, John Tinsley, and Andy Way
2017 “Is Neural Machine Translation the New State of the Art?” Prague Bulletin of Mathematical Linguistics 108 (1): 109–120.
Chang, Vincent Chieh-Ying
2009Testing Applicability of Eye-tracking and fMRI to Translation and Interpreting Studies: An Investigation into Directionality. PhD dissertation. Imperial College, London.
Daems, Joke, Sonia Vandepitte, Robert J. Hartsuiker, and Lieve Macken
2017 “Identifying the Machine Translation Error Types with the Greatest Impact on Post-Editing Effort.” Frontiers in Psychology 8: 1282.
Danks, Joseph H., Gregory M. Shreve, Stephen B. Fountain, and Michael K. McBeath
1997Cognitive Processes in Translation and Interpreting. Thousand Oaks, California: Sage Publications.
Denkowski, Michael, and Alon Lavie
2014 “Meteor Universal: Language Specific Translation Evaluation for Any Target Language.” In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation.
2003Eye Tracking Methodology: Theory and Practice. Heidelberg: Springer.
García, Ignacio
2011 “Translating by Post-editing: Is It the Way Forward?” Machine Translation 25: 217–237.
Gaspari, Federico, Antonio Toral, Sudip Kumar Naskar, Declan Groves, Andy Way
2014 “Perception vs Reality: Measuring Machine Translation Post-Editing Productivity.” In Proceedings of AMTA 2014 Workshop on Post-editing Technology and Practice, Vancouver, 60–72.
Gonçalves, José Luiz
2016 “Investigating Saccades as an Index of Cognitive Effort in Post-editing and Translation.” In Proceedings of EST Congress 2016, Aarhus, Denmark.
Hansen-Schirra, Silvia
2017 “EEG and Universal Language Processing in Translation.” In The Handbook of Translation and Cognition, ed. by John W. Schwieter, and Aline Ferreira, 232–247. Hoboken: John Wiley & Sons.
Hokamp, Chris, and Qun Liu
2015 “HandyCAT.” In Proceedings of European Association for Machine Translation (EAMT) 2015, Antalya, 216.
Hutchins, William John
1992 “Météo.” In An Introduction to Machine Translation, ed. by W. John Hutchins, and Harold L. Somers, 207–220. London: Academic Press.
Hvelplund, Kristian T.
2011Allocation of Cognitive Resources in Translation: An Eye-tracking and Key-logging Study. PhD dissertation, Copenhagen Business School.
Jakobsen, Arnt Lykke
1999 “Logging Target Text Production with Translog.” Copenhagen Studies in Language 24: 9–20.
Koglin, Arlene
2015 “An Empirical Investigation of Cognitive Effort Required to Post-edit Machine Translated Metaphors Compared to the Translation of Metaphors.” Translation and Interpreting 7 (1): 126–141.
Krings, Hans P.
2001Repairing Texts: Empirical Investigations of Machine Translation Post-editing Processes. Translated by G. S. Koby. Kent, OH: The Kent State University Press.
Lacruz, Isabel, and Gregory M. Shreve
2014 “Pauses and Cognitive Effort in Post-editing.” In Post-editing of Machine Translation: Processes and Applications, ed. by Sharon O’Brien, Laura Winther Balling, Michael Carl, Michel Simard, and Lucia Specia, 246–272. Newcastle-Upon-Tyne: Cambridge Scholars.
Läubli, Samuel, Mark Fishel, Gary Massey, Maureen Ehrensberger-Dow, and Martin Volk
2013 “Assessing Post-Editing Efficiency in a Realistic Translation Environment.” In Proceedings of MT Summit XIV Workshop on Post-editing Technology and Practice, Nice, 83–91.
Läubli, Samuel, and Ulrich Germann
2016 “Statistical Modelling and Automatic Tagging of Human Translation Processes.” In New Directions in Empirical Translation Process Research, ed. by Michael Carl, Srinivas Bangalore, and Moritz Schaeffer, 77–94. Heidelberg: Springer.
Lommel, Arle R., and Donald A. DePalma
2016Europe’s Leading Role in Machine Translation: How Europe Is Driving the Shift to MT. Boston: Common Sense Advisory Report.
Moorkens, Joss, Sharon O’Brien, Igor Antonio Lourenco Silva, Norma Fonseca, and Fabio Alves
2015 “Correlations of Perceived Post-editing Effort with Measurements of Actual Effort.” Machine Translation 29 (3–4): 267–284.
Moorkens, Joss
2017 “Under Pressure: Translation in Times of Austerity.” Perspectives: Studies in Translation Theory and Practice, 25 (3): 464–477.
Moorkens, Joss, and Sharon O’Brien
2015 “Post-editing Evaluations: Trade-offs Between Novice and Professional Participants.” In Proceedings of European Association for Machine Translation Conference (EAMT) 2015, Antalya, Turkey, 75–81.
Mossop, Brian
2007Revising and Editing for Translators. Manchester: St. Jerome.
Nitzke, Jean, Katharina Oster
2016 “Comparing Translation and Post-editing: An Annotation Schema for Activity Units.” In New Directions in Empirical Translation Process Research, ed. by Michael Carl, Srinivas Bangalore, and Moritz Schaeffer, 77–94. Heidelberg: Springer.
Nunes Vieira, Lucas
2015Cognitive Effort in Post-Editing of Machine Translation: Evidence from Eye Movements, Subjective Ratings, and Think-Aloud Protocols. PhD dissertation. Newcastle University.
O’Brien, Sharon
2005 “Methodologies for Measuring the Correlations Between Post-editing Effort and Machine Translatability.” Machine Translation 19 (1): 37–58.
O’Brien, Sharon
2006 “Eye Tracking and Translation Memory Matches.” Perspectives: Studies in Translatology 14 (3): 185–205.
O’Brien, Sharon
2008 “Processing Fuzzy Matches in Translation Memory Tools: An Eye-tracking Analysis.” In Looking at Eyes. Eye Tracking Studies of Reading and Translation Processing, ed. by Susanne Göpferich, Arnt Lykke Jakobsen and Inger Mees, 79–102. Frederiksberg: Samfundslitteratur.
O’Brien, Sharon
2010 “Introduction to Post-editing: Who, What, How and Where to Next?” In Proceedings of AMTA 2010, the 9th Conference of the Association for Machine Translation in the Americas, Denver, USA.
2010 “A Productivity Test of Statistical Machine Translation Post-editing in a Typical Localization Context.” Prague Bulletin of Mathematical Linguistics 93: 7–16.
Saldanha, Gabriela, and Sharon O’Brien
2014Research Methodologies in Translation Studies. London: Routledge.
Schmidtke, Dag
2016 “Large Scale Machine Translation Publishing, With Acceptable Quality, for Microsoft Support Content.” In Proceedings of AMTA 2016 Workshop on Interacting with Machine Translation (iMT 2016). Austin, Texas.
Senez, Dorothy
1998 “Post-editing Service for Machine Translation Users at the European Commission.” Translating and the Computer 20. Proceedings of the Twentieth International Conference. 12–13 November 1998.
Shreve, Gregory M., and Bruce Diamond
1997 “Cognitive Processes in Translation and Interpreting: Critical Issues.” In Cognitive Processes in Translation and Interpreting, ed. by Joseph H. Danks, Gregory M. Shreve, Stephen B. Fountain, and Michael K. McBeath, 233–251. Thousand Oaks, California: Sage Publications.
Snover, Matthew, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul
2006 “A Study of Translation Edit Rate with Targeted Human Annotation.” In Proceedings of Association for Machine Translation in the Americas, 223–231 8–12August 2006, Cambridge, Massachusetts, USA.
Specia, Lucia, Nicola Cancedda, Marc Dymetman, Marco Turchi, and Nello Cristianini
2009 “Estimating the Sentence-level Quality of Machine Translation Systems.” In Proceedings of the 13th Annual Conference of the EAMT, Barcelona, 28–35.
Specia, Lucia
2011 “Exploiting Objective Annotations for Measuring Translation Post-editing Effort.” In Proceedings of the 15th conference of EAMT, Leuven, 73–80.
Su, Keh-Yih, Ming-Wen Wu, and Jing-Shin Chang
1992 “A New Quantitative Quality Measure for Machine Translation Systems.” In Proceedings of COLING-92, Nantes, 433–439.
Teixeira, Carlos S. C.
2014 “Perceived vs. Measured Performance in the Post-editing of Suggestions from Machine Translation and Translation Memories.” In Proceedings of AMTA 2014 Workshop on Post-editing Technology and Practice, Vancouver, 45–59.
Turian, Joseph P., Luke Shen, and I. Dan Melamed
2003 “Evaluation of Machine Translation and Its Evaluation.” In Proceedings of MT Summit 2003, 386–393. New Orleans, Louisiana.
Vasconcellos, Muriel, and Marjorie León
1985 “SPANAM and ENGSPAN: Machine Translation at the Pan American Health Organization.” Computational Linguistics 11 (2–3): 122–136.
Wagner, Emma
1985 “Post-editing Systran: A Challenge for Commission Translators.” Terminologie et Traduction 3: 1–7.
Way, Andy
2013 “Traditional and Emerging Use-cases for Machine Translation.” In Proceedings of Translating and the Computer 35.
Wilms, Franz-Josef M.
1981 “Von SUSY zu SUSY-BSA: Forderungen and ein anwenderbezogenes MU-System.” Sprache und Datenverarbeitung 5: 38–43.
Cited by
Cited by 8 other publications
Cui, Ying, Xiao Liu & Yuqin Cheng
2023. A Comparative Study on the Effort of Human Translation and Post-Editing in Relation to Text Types: An Eye-Tracking and Key-Logging Experiment. SAGE Open 13:1 ► pp. 215824402311558 ff.
Cumbreño, Cristina & Nora Aranberri
2021. What Do You Say? Comparison of Metrics for Post-editing Effort. In Explorations in Empirical Translation Process Research [Machine Translation: Technologies and Applications, 3], ► pp. 57 ff.
2019. Post-editing neural machine translation versus translation memory segments. Machine Translation 33:1-2 ► pp. 31 ff.
Tardel, Anke
2021. Measuring Effort in Subprocesses of Subtitling. In Explorations in Empirical Translation Process Research [Machine Translation: Technologies and Applications, 3], ► pp. 81 ff.
Walker, Callum
2021. The Cognitive Paradigm in Translation Studies. In An Eye-Tracking Study of Equivalent Effect in Translation, ► pp. 13 ff.
This list is based on CrossRef data as of 16 september 2023. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.