Cover not available
Part of
Translation in Transition: Between cognition, computing and technology
Edited by Arnt Lykke Jakobsen and Bartolomé Mesa-Lao
[Benjamins Translation Library 133] 2017
► pp. 161–186

Chapter 6
From process to product

Links between post-editing effort and post-edited quality

| University of Bristol

Post-editing of machine translation (MT) is now increasingly implemented in the human translation workflow after studies in both industry and academia have demonstrated the efficacy of this practice. Post-editing still involves open questions, however, such as how best to train post-editors and how to estimate the effort required by post-editing tasks. In attempting to address some of these questions, many previous studies investigate the post-editing process, but less research has focused on the post-edited product. This chapter examines the link between the process and product of post-editing by checking to see how post-editing effort data relates to the quality of post-edited texts, assessed in terms of fluency (linguistic quality) and adequacy (translation accuracy). A statistical analysis indicated that the association between editing operations and the fluency of post-edited texts is dependent on the quality of the raw MT output. Interestingly, a negative association was observed between the number of eye fixations on the text and the quality of the post-edited translations. The chapter shows empirical evidence supporting the distinction between the concepts of translation fluency and adequacy, and postulates that automatic processes play a central role in post-editing performance.

Article outline
  • 1.Introduction
  • 2.Related work
  • 3.Methods and procedure
    • 3.1Study materials and the post-editing task
    • 3.2Post-editors
    • 3.3Assessing post-edited quality
    • 3.4Post-editing effort data
  • 4.Results and discussion
    • 4.1Fluency and adequacy
    • 4.2Rater agreement
    • 4.3Post-editing effort and post-edited quality
  • 5.Conclusion
    • 5.1Research findings
    • 5.2Suggestions for future research
  • Acknowledgments
  • Notes
  • References
Published online: 30 September 2017
DOI logo
https://doi.org/10.1075/btl.133.06vie
References
Aziz, Wilker, Sheila Castilho, and Lucia Specia
2012 “PET: a Tool for Post-Editing and Assessing Machine Translation.” In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC 2012), 21–27 May 2012, Istanbul, Turkey. ed. by Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Jan Odijk, and Stelios Piperidis, 3982–3987.
Baayen, R. Harald
2008 Analysing Linguistic Data: a Practical Introduction to Statistics. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Callison-Burch, Chris, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder
2007 “(Meta-) Evaluation of Machine Translation.” In Proceedings of the Second Workshop on Statistical Machine Translation , Prague, Czech Republic, 23 June 2007, 136–158. DOI logo
Carl, Michael, Barbara Dragsted, Jakob Elming, Daniel Hardt, and Arnt L. Jakobsen
2011 “The Process of Post-Editing: a Pilot Study.” In Proceedings of the 8th International NLPCS Workshop: Human-Machine Interaction in Translation, ed. by Bernardette Sharp, Michael Zock, Michael Carl, and Arnt L. Jakobsen, 131–142. Copenhagen: Samfundslitteratur.Google Scholar
Christensen, Rune Haubo B
2010 ordinal: Regression Models for Ordinal Data . R Package Version no. 22.Google Scholar
Daems, Joke, Lieve Macken, and Sonia Vandepitte
2013 “Quality as the Sum of its Parts: a Two-Step Approach for the Identification of Translation Problems and Translation Quality Assessment for HT and MT+PE.” In Proceedings of the MT Summit XIV Workshop on Post-Editing Technology and Practice (WPTP2), 2–6 September 2013, Nice, France, ed. by Sharon O’Brien, Michel Simard, and Lucia Specia, 63–71.
de Almeida, Giselle
2013  Translating the Post-Editor: an Investigation of Post-Editing Changes and Correlations with Professional Experience across Two Romance Languages . PhD Thesis, Dublin City University.
de Groot, Annette M.B
2000 “A Complex-Skill Approach to Translation and Interpreting.” In Tapping and Mapping the Processes of Translation and Interpreting: Outlooks on Empirical Research, ed. by Sonja Tirkkonen-Condit, and Riitta Jääskeläinen, 53–68. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Denkowski, Michael, and Alon Lavie
2011 “Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems.” In Proceedings of the Sixth Workshop on Statistical Machine Translation (EMNLP 2011), 30–31 July 2011, Edinburgh, UK, 85–91.
Depraetere, Ilse, Nathalie De Sutter, and Arda Tezcan
2014 “Post-Edited Quality, Post-Editing Behaviour and Human Evaluation: a Case Study.” In Post-Editing of Machine Translation: Processes and Applications, ed. by Sharon O'Brien, Laura Winther Balling, Michael Carl, Michel Simard, and Lucia Specia, 78–108. Newcastle upon Tyne: Cambridge Scholars Publishing.Google Scholar
Duchowski, Andrew T
2007 Eye Tracking Methodology: Theory and Practice. London: Springer-Verlag.Google Scholar
Green, Spence, Jason Chuang, Jeffrey Heer, and Christopher D. Manning
2014 “Predictive Translation Memory: a Mixed-Initiative System for Human Language Translation.” In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology , 05–08 October 2014, Honolulu, USA, 177–187.
Green, Spence, Jeffrey Heer, and Christopher D. Manning
2013 “The Efficacy of Human Post-Editing for Language Translation.” In ACM Human Factors in Computing Systems (CHI), 27 April–2 May 2013, Paris, France, 439–448.
Gopher, Daniel, and Emanuel Donchin
1986 “Workload: an Examination of the Concept.” In Handbook of Perception and Human Performance, Vol. 2, ed. by Kenneth R. Boff, Lloyd Kaufman, and James P. Thomas, 1–49. New York: John Wiley and Sons.Google Scholar
Guerberof, Ana
2014 “The Role of Professional Experience in Post-Editing from a Quality and Productivity Perspective.” In Post-Editing of Machine Translation: Processes and Applications, ed. by Sharon O'Brien, Laura Winther Balling, Michael Carl, Michel Simard and Lucia Specia, 51–76. Newcastle upon Tyne: Cambridge Scholars Publishing.Google Scholar
Guzmán, Rafael
2007 “Manual MT Post-Editing: If It’s Not Broken, Don’t Fix It!” Translation Journal 11 (4). Accessed 26 June 2014. [URL].Google Scholar
Hogarth, Robin M
2005 “Deciding Analytically or Trusting Your Intuition? The Advantages and Disadvantages of Analytic and Intuitive Thought.” In The Routines of Decision Making, ed. by Tilmann Betsch, and Susanne Haberstroh, 67–82. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
Hvelplund, Kristian Tangsgaard
2011  Allocation of Cognitive Resources in Translation: An Eye-Tracking and Key-Logging Study . PhD Thesis, Copenhagen Business School.
Jensen, K.T. Hvelplund, Annette Sjørup, and Laura Winther Balling
2009 “Effects of L1 Syntax on L2 Translation.” In Methodology, Technology and Innovation in Translation Process Research, ed. by Inger M. Mees, Fabio Alves, and Susanne Göpferich, 319–339. Copenhagen: Samfundslitteratur.Google Scholar
Koponen, Maarit
2012 “Comparing Human Perceptions of Post-Editing Effort with Post-Editing Operations.” In Proceedings of the Seventh Workshop on Statistical Machine Translation (NAACL 2012), 7–8 June 2012, Montreal, Canada, 181–190.
Koponen, Maarit, Wilker Aziz, Luciana Ramos, and Lucia Specia
2012 “Post-Editing Time as a Measure of Cognitive Effort.” In Proceedings of the AMTA 2012 Workshop on Post-Editing Technology and Practice (WPTP 2012) , 28 October 2012, San Diego, USA, ed. by Sharon O'Brien, Michel Simard, and Lucia Specia, 10.
Krings, Hans P
2001 Repairing Texts: Empirical Investigations of Machine Translation Post-Editing Processes, ed. by Geoffrey Koby. Kent, Ohio: Kent State University Press.Google Scholar
LDC
2005 Linguistic Data Annotation Specification: Assessment of Fluency and Adequacy in Translations. Revision 1.5.Google Scholar
Lin, Chin-Yew, and Franz Josef Och
2004 “ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation.” In Proceedings of the 20th International Conference on Computational Linguistics , 23–27 August 2004, Geneva, Switzerland, 7.
McCutchen, Deborah
1996 “A Capacity Theory of Writing: Working Memory in Composition.” Educational Psychology Review 8:299–325. DOI logo.Google Scholar
Meara, Paul, and Barbara Buxton
1987 “An Alternative to Multiple Choice Vocabulary Tests.” Language Testing 4 (2): 142–154. DOI logoGoogle Scholar
Mitchell, Linda
2015  Community Post-Editing of Machine-Translated User-Generated Content . PhD Thesis, Dublin City University.
Norman, Donald A., and Tim Shallice
1986 “Attention to Action: Willed and Automatic Control of Behaviour.” In Consciousness and Self-Regulation: Advances in Research and Theory, ed. by Richard J. Davidson, Gary E. Schwarts, and David Shapiro, 1–18. New York: Plenum Press. DOI logoGoogle Scholar
O’Brien, Sharon
2011 “Towards Predicting Post-Editing Productivity.” Machine Translation 25 (3): 197–215. DOI logo.Google Scholar
Paas, Fred G
1992 “Training Strategies for Attaining Transfer of Problem-Solving Skill in Statistics: a Cognitive-Load Approach.” Journal of Educational Psychology 84 (4): 429–434. DOI logoGoogle Scholar
Plitt, Mirko, and François Masselot
2010 “A Productivity Test of Statistical Machine Translation Post-Editing in a Typical Localization Context.” Prague Bull Math Linguist 93: 7–16. DOI logoGoogle Scholar
Rayner, Keith
1998 “Eye Movements in Reading and Information Processing: 20 Years of Research.” Psychological Bulletin 124 (3): 372–422. DOI logoGoogle Scholar
Shih, Claire
2006 “Revision from Translators’ Point of View: an Interview Study.” Target 18 (2): 295–312. DOI logoGoogle Scholar
Snijders, Tom A.B., and Roel Bosker
1999 Multilevel Analysis: an Introduction to Basic and Advanced Multilevel Modelling. London: Sage.Google Scholar
Snover, Matthew, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul
2006 “A Study of Translation Edit Rate with Targeted Human Annotation.” In Proceedings of the 7th Conference of the Association for Machine Translation of the Americas – Visions for the Future of Machine Translation , 8–12 August 2006, Cambridge, USA, 223–231.
TAUS
2013 Adequacy/Fluency Guidelines. Accessed 17 June 2015. [URL].Google Scholar
TAUS/CNGL
2010 MT Post-Editing Guidelines. Accessed 17 June 2015. [URL].Google Scholar
Temnikova, Irina
2010 “A Cognitive Evaluation Approach for a Controlled Language Post-Editing Experiment.” In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC 2010) , 19–21 May 2010, Valetta, Malta, 3485–3490.
Tobii Technology
2012 Determining the Tobii I-VT Fixation Filter’s Default Values: Method Description and Results Discussion. Accessed 20 December 2013. [URL].Google Scholar
Unsworth, Nash, Richard P. Heitz, Josef C. Schrock, and Randall W. Engle
2005 “An Automated Version of the Operation Span Task.” Behavior Research Methods 37 (3): 498–505. DOI logoGoogle Scholar
Vieira, Lucas N
2014 “Indices of Cognitive Effort in Machine Translation Post-Editing.” Machine Translation 28 (3–4): 187–216. DOI logoGoogle Scholar
2016  Cognitive Effort in Post-Editing of Machine Translation: Evidence from Eye Movements, Subjective Ratings, and Think-Aloud Protocols . PhD Thesis, Newcastle University.
Cited by

Cited by 4 other publications

Albl-Mikasa, Michaela, Maureen Ehrensberger-Dow, Andrea Hunziker Heeb, Caroline Lehr, Michael Boos, Matthias Kobi, Lutz Jäncke & Stefan Elmer
2020. Cognitive load in relation to non-standard language input. Translation, Cognition & Behavior 3:2 ► pp. 263 ff. DOI logo
de Faria Pires, Loïc
2020. Master’s students’ post-editing perception and strategies. FORUM. Revue internationale d’interprétation et de traduction / International Journal of Interpretation and Translation 18:1 ► pp. 26 ff. DOI logo
Hatzidaki, Anna
2019. Using experimental approaches to study translation. Translation, Cognition & Behavior 2:1 ► pp. 35 ff. DOI logo
Koponen, Maarit, Leena Salmi & Markku Nikulin
2019. A product and process analysis of post-editor corrections on neural, statistical and rule-based machine translation output. Machine Translation 33:1-2 ► pp. 61 ff. DOI logo

This list is based on CrossRef data as of 12 april 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.