Cover not available
Article published In:
Translation, Cognition & Behavior
Vol. 6:1 (2023) ► pp.60–86

The impact of traditional and interactive post-editing on Machine Translation User Experience, quality, and productivity

ORCID logo | Dublin City University
ORCID logo | Dublin City University
ORCID logo | University College Dublin
This paper presents a user study with 15 professional translators in the English-Spanish combination. We present the concept of Machine Translation User Experience (MTUX) and compare the effects of traditional post-editing (TPE) and interactive post-editing (IPE) on MTUX, translation quality and productivity. Results suggest that translators prefer IPE to TPE because they are in control of the interaction in this new form of translator-computer interaction and feel more empowered in their interaction with Machine Translation. Productivity results also suggest that IPE may be an interesting alternative to TPE, given the fact that translators worked faster in IPE even though they had no experience in this new machine translation post-editing modality, but were already used to TPE.
Keywords: machine translation, user experience, human-computer interaction, interactive post-editing, translation quality evaluation
Article outline
  • 1.Introduction
  • 2.Traditional vs interactive post-editing
  • 3.Machine Translation User Experience (MTUX)
  • 4.Methodology
    • Participants
    • Texts
    • IPE workbench
    • Design of the controlled study
    • Measures for MTUX, productivity, and quality
      • Machine Translation User Experience (MTUX)
      • Translation productivity
      • Translation quality
  • 5.Results
    • Analysis approach
    • RQ1. Do the MTPE modalities (TPE or IPE) impact MTUX in a statistically significantly way?
    • RQ2. As translators have more experience in TPE than in IPE, is MTUX impacted by increased experience with the system in a specific MTPE modality?
    • RQ3. Do the MTPE modalities (TPE or IPE) impact translator performance measures (translation quality and productivity) in a statistically significantly way?
      • Translation quality
      • Translation productivity
    • RQ4. Does MTUX correlate with translator performance measures (translation quality and productivity)?
  • 6.Discussion
    • IPE showed significant MTUX benefits to TPE
    • IPE and TPE experience over time—The need for a longitudinal study
    • The relationship between MTUX and performance
  • 7.Conclusions
  • Acknowledgements
  • Notes
  • References
Available under the Creative Commons Attribution (CC BY) 4.0 license.

For any use beyond this license, please contact the publisher at rights@benjamins.nl.

Published online: 13 June 2023
DOI logo
https://doi.org/10.1075/tcb.00077.bri
References
Alabau Gonzalvo, Vicent, Ragnar Bonkb, Christian Buck, Michael Carlb, Francisco Casacuberta Nolla, Mercedes García Martínez, Jesús González Rubio, Philip Koehn, Luis Alberto Leiva Torres, Bartolomé Mesa Lao, Daniel Ortiz Martínez, Herve Saint-Amand, Germán Sanchis Trilles & Chara Tsoukalak
2013 CASMACAT: An open source workbench for advanced computer aided translation. Prague Bulletin of Mathematical Linguistics, 1001: 101–12. De Gruyter Open. DOI logoGoogle Scholar
Alabau Gonzalvo, Vicent, Michael Carl, Mercedes García Martínez & Jesús González Rubio
2016 Learning Advanced Post-Editing. In New Directions in Empirical Translation Process Research. Edited by Michael Carl, Srinivas Bangalore, Moritz Schaeffer, 95–110. Cham: Springer. DOI logoGoogle Scholar
Alves, Fabio, Karina Sarto Szpak, José Luiz Gonçalves, Kyoko Sekino, Marceli Aquino, Rodrigo Araújo e Castro, Arlene Koglin, Norma B. de Lima Fonseca, and Bartolomé Mesa Lao
2016 Investigating Cognitive Effort in Post-Editing: A Relevance-Theoretical Approach. In Eyetracking and Applied Linguistics. Edited by Silvia Hansen-Schirra & Sambor Grucza, 109–42. Language Science Press. [URL]
Artstein, Ron
2017 Inter-Annotator Agreement. In Handbook of Linguistic Annotation. Edited by Nancy Ide & James Pustejovsky, 297–313. Dordrecht: Springer Netherlands. DOI logoGoogle Scholar
Bertoldi, Nicola, Davide Caroselli & Marcello Federico
2018 The ModernMT Project. In Proceedings of the 21st Annual Conference of the European Association for Machine Translation. Edited by Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez, Miquel Esplà-Gomis, Maja Popovic, Celia Rico, André Martins, Joachim Van den Bogaert, Mikel L. Forcada, 345–347. Alicante: European Association for Machine Translation. [URL]
Bowker, Lynne & Jairo Buitrago Ciro
2018 Localizing Websites Using Machine Translation: Exploring Connections between User EXperience and Translatability. In The Human Factor in Machine Translation. Edited by Sin-wai Chan, 29–52. London: Routledge. DOI logoGoogle Scholar
Briva-Iglesias, Vicent
2021 Traducción humana vs. traducción automática: análisis contrastivo e implicaciones para la aplicación de la traducción automática en traducción jurídica. Mutatis Mutandis. Revista Latinoamericana de Traducción 14 (2): 571–600. DOI logoGoogle Scholar
2022 English-Catalan Neural Machine Translation: State-of-the-Art Technology, Quality, and Productivity. Tradumàtica: Tecnologies de La Traducció 201: 149–76. DOI logoGoogle Scholar
Briva-Iglesias, Vicent, Sharon O’Brien & Benjamin R. Cowan
2023 Measuring Machine Translation User Experience: A Comparison between AttrakDiff and User Experience Questionnaire. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation.Google Scholar
Forthcoming. Translators’ Pre-Task Perceptions of CAT Tools and MTPE, and Their Relationship with Translation Quality: Implications for Training.
Cadwell, Patrick, Sheila Castilho, Sharon O’Brien & Linda Mitchell
2016 Human Factors in Machine Translation and Post-Editing among Institutional Translators. Translation Spaces 5 (2): 222–43. DOI logoGoogle Scholar
Cadwell, Patrick, Sharon O’Brien & Carlos S. C. Teixeira
2018 Resistance and Accommodation: Factors for the (Non-) Adoption of Machine Translation among Professional Translators. Perspectives 26 (3): 301–21. DOI logoGoogle Scholar
Castilho, Sheila
2016 Measuring Acceptability of Machine Translated Enterprise Content. PhD Thesis, Dublin City University. Doctoral, Dublin City University. Faculty of Humanities and Social Sciences. [URL]
Castilho, Sheila, Stephen Doherty, Federico Gaspari & Joss Moorkens
2018 Approaches to Human and Machine Translation Quality Assessment. In Translation Quality Assessment: From Principles to Practice. Edited by Joss Moorkens, Sheila Castilho, Federico Gaspari & Stephen Doherty, 9–38. Machine Translation: Technologies and Applications. Cham: Springer International Publishing. DOI logoGoogle Scholar
Daems, Joke & Lieve Macken
2019 Interactive Adaptive SMT versus Interactive Adaptive NMT: A User Experience Evaluation. Machine Translation 33 (1): 117–34. DOI logoGoogle Scholar
Dix, Alan
2010 Human-Computer Interaction: A Stable Discipline, a Nascent Science, and the Growth of the Long Tail. Interacting with Computers 22 (1): 13–27. DOI logoGoogle Scholar
Esteban Lauzán, José, José Lorenzo Mon, Antonio Sánchez Valderrábanos & Guy Lapalme
2004 TransType2: An Innovative Computer-Assisted Translation System. In Proceedings of the ACL 2004 on Interactive Poster and Demonstration Sessions, 94–97. Barcelona, Spain: Association for Computational Linguistics. DOI logoGoogle Scholar
Etchegoyhen, Thierry, Anna Fernández Torné, Andoni Azpeitia Zaldua, Eva Martínez García & Anna Matamala Ripoll
2018 Evaluating Domain Adaptation for Machine Translation Across Scenarios. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan: European Language Resources Association. [URL]
Forlizzi, Jodi & Katja Battarbee
2004 Understanding Experience in Interactive Systems. In Proceedings of the 5th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, 261–68. DIS ’04. New York, NY, USA: Association for Computing Machinery. DOI logoGoogle Scholar
Gaspari, Federico, Antonio Toral Ruiz, Sudip Kumar Naskar, Declan Groves & Andy Way
2014 Perception vs Reality: Measuring Machine Translation Post-Editing Productivity. In Proceedings of the 11th Conference of the Association for Machine Translation in the Americas, 60–72. Vancouver, Canada: Association for Machine Translation in the Americas.Google Scholar
Green, Spence
2016 Interactive Machine Translation. In Conferences of the Association for Machine Translation in the Americas. Invited talk.Google Scholar
Green, Spence, Jason Chuang, Jeffrey Heer & Christopher D. Manning
2014 Predictive Translation Memory: A Mixed-Initiative System for Human Language Translation. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, 177–187. Honolulu Hawaii USA: ACM. DOI logoGoogle Scholar
Green, Spence, Sida I. Wang, Jason Chuang, Jeffrey Heer, Sebastian Schuster & Christopher D. Manning
2014 Human Effort and Machine Learnability in Computer Aided Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1225–1236. Doha, Qatar: Association for Computational Linguistics. DOI logoGoogle Scholar
Guerberof Arenas, Ana, Joss Moorkens & Sharon O’Brien
2021 The Impact of Translation Modality on User Experience: An Eye-Tracking Study of the Microsoft Word User Interface. Machine Translation 35 (2): 205–37. DOI logoGoogle Scholar
Guerberof Arenas, Ana
2008 Productivity and Quality in the Post-Editing of Outputs from Translation Memories and Machine Translation. Localisation Focus The International Journal of Localisation, 7 ( 1 ), 11–21.Google Scholar
ISO
2018 ISO 9241-11:2018(En), Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts 2018 [URL]
Karakanta, Alina, Luisa Bentivogli, Mauro Cettolo, Matteo Negri & Marco Turchi
2022 Post-Editing in Automatic Subtitling: A Subtitlers’ Perspective. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, 261–70. Ghent, Belgium: European Association for Machine Translation. [URL]
Knowles, Rebecca, Marina Sánchez Torrón & Philipp Koehn
2019 Neural Interactive Translation Prediction. Machine Translation 331: 135–134. DOI logoGoogle Scholar
Koehn, Philipp
2009 A Web-Based Interactive Computer Aided Translation Tool. In Proceedings of the ACL-IJCNLP 2009 Software Demonstrations. Edited by Gary Geunbae Lee, Sabine Schulte im Walde, 17–20. Suntec, Singapore: Association for Computational Linguistics. [URL]. DOI logo
Koehn, Philipp, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, Evan Herbst, Hieu Hoang
, et al 2007 Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions—ACL. Edited by Sophia Ananiadou, 177–180. Prague: Association for Computational Linguistics. DOI logoGoogle Scholar
Koponen, Maarit
2012 Comparing Human Perceptions of Post-Editing Effort with Post-Editing Operations. In Proceedings of the Seventh Workshop on Statistical Machine Translation. Edited by Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut & Lucia Specia, 181–190. Montréal, Canada: Association for Computational Linguistics.Google Scholar
Koponen, Maarit, Umut Sulubacak, Kaisa Vitikainen & Jörg Tiedemann
2020 MT for Subtitling: Investigating Professional Translators’ User Experience and Feedback. In Proceedings of 1st Workshop on Post-Editing in Modern-Day Translation. Edited by John E. Ortega, Marcello Federico, Constantin Orasan, Maja Popovic, 79–92. Virtual: AMTA. [URL]
Langlais, Philippe, George Foster & Guy Lapalme
2000 TransType: A Computer-Aided Translation Typing System. In ANLP-NAACL 2000 Workshop: Embedded Machine Translation Systems, 1–6. [URL]
Läubli, Samuel, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen & Antonio Toral Ruiz
2020 A Set of Recommendations for Assessing Human–Machine Parity in Language Translation. Journal of Artificial Intelligence Research 671 (March): 653–672–653–72. DOI logoGoogle Scholar
Läubli, Samuel & Spence Green
2019 Translation Technology Research and Human–Computer Interaction. In The Routledge Handbook of Translation and Technology, edited by Minako O’Hagan, 370–83. New York, NY, USA: Routledge. DOI logoGoogle Scholar
Läubli, Samuel, Patrick Simianer, Joern Wuebker, Geza Kovacs, Rico Sennrich & Spence Green
2022 The Impact of Text Presentation on Translator Performance. Target. International Journal of Translation Studies 34 (2): 309–342. [URL]. DOI logo
Laugwitz, Bettina, Theo Held & Martin Schrepp
2008 Construction and Evaluation of a User Experience Questionnaire. International Journal of Interactive Multimedia and Artificial Intelligence 4 (4): 63–76. DOI logoGoogle Scholar
Macklovitch, Elliott
2006 TransType2: The Last Word. In Proceedings of the 5th Edition of the International Conference on Language Resources and Evaluation. Edited by Nicoletta Calzolari, Khalid Choukri, Aldo Gangemi, Bente Maegaard, Joseph Mariani, Jan Odijk, Daniel Tapias, 1–6. Genoa, Italy: International Conference on Language Resources and Evaluation (LREC).Google Scholar
Martikainen, Hanna
2022 Ghosts in the Machine: Can Adaptive MT Help Reclaim a Place for the Human in the Loop?. In Portail HAL Sorbonne Nouvelle. [URL]
Moorkens, Joss
2020 “A Tiny Cog in a Large Machine”: Digital Taylorism in the Translation Industry. Translation Spaces 9 (1): 12–34. DOI logoGoogle Scholar
Nurminen, Mary
2019 Decision-Making, Risk, and Gist Machine Translation in the Work of Patent Professionals. In Proceedings of the 8th Workshop on Patent and Scientific Literature Translation, 32–42. Dublin: European Association for Machine Translation. [URL]
Nurminen, Mary & Niko Papula
2018 Gist MT Users: A Snapshot of the Use and Users of One Online MT Tool. In Edited by Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez, Miquel Esplà-Gomis, Maja Popovic, Celia Rico, André Martins, Joachim Van den Bogaert, Mikel L. Forcada, 199–208. Alicante: European Association for Machine Translation. [URL]
O’Brien, Sharon
2006 Pauses as Indicators of Cognitive Effort in Post-Editing Machine Translation Output. Across Languages and Cultures 7 (1): 1–21. DOI logoGoogle Scholar
2022 How to Deal with Errors in Machine Translation: Post-Editing. In Machine Translation for Everyone, 105–20. Berlin: Language Science Press. DOI logoGoogle Scholar
O’Brien, Sharon, Maureen Ehrensberger-Dow, Marcel Hasler & Megan Connolly
2017 Irritating CAT Tool Features That Matter to Translators. Hermes: Journal of Language and Communication in Business 561: 145–62.Google Scholar
Olohan, Maeve
2011 Translators and Translation Technology: The Dance of Agency. Translation Studies 4 (3): 342–57. DOI logoGoogle Scholar
Paz, Freddy & Jose Pow-Sang
2016 A Systematic Mapping Review of Usability Evaluation Methods for Software Development Process 101: 165–78. DOI logoGoogle Scholar
Pérez Macías, Lorena, María del Mar Sánchez Ramos & Celia Rico Pérez
2020 Study on the Usefulness of Machine Translation in the Migratory Context: Analysis of Translators’ Perceptions. Open Linguistics 6 (1): 68–76. DOI logoGoogle Scholar
Peris Abril, Álvaro, Miguel Domingo Ballester & Francisco Casacuberta Nolla
2017 Interactive Neural Machine Translation. Computer Speech & Language 451 (September): 201–20. DOI logoGoogle Scholar
Sadiku, Matthew N. O. & Sarhan M. Musa
2021 A Primer on Multiple Intelligences. Cham: Springer International Publishing. DOI logoGoogle Scholar
Sakamoto, Akiko
2019 Unintended Consequences of Translation Technologies: From Project Managers Perspectives’. Perspectives 27 (1): 58–73. DOI logoGoogle Scholar
Sánchez-Torrón, Marina
2017 Productivity in Post-Editing and in Neural Interactive Translation Prediction: A Study of English-to-Spanish Professional Translators. PhD Dissertation. University of Auckland. [URL]
Sánchez Gijón, Pilar, Joss Moorkens & Andy Way
2019 Post-Editing Neural Machine Translation versus Translation Memory Segments. Machine Translation 33 (1–2): 31–59. DOI logoGoogle Scholar
Sanchis Trilles, Germán, Vicent Alabau Gonzalvo, Christian Buck, Michael Carl, Francisco Casacuberta Nolla, Mercedes García Martínez, Ulrich Germann, Jesús González Rubio, Robin L. Hill, Philipp Koehn, Luis A. Leiva Torres, Bartolomé Mesa Lao, Daniel Ortiz Martínez, Herve Saint-Amand, Chara Tsoukala & Enrique Vidal Ruiz
2014 Interactive Translation Prediction versus Conventional Post-Editing in Practice: A Study with the CasMaCat Workbench, Machine Translation 28 (3): 217–235. DOI logoGoogle Scholar
Schrepp, Martin, Andreas Hinderks & Jörg Thomaschewski
2014 Applying the User Experience Questionnaire (UEQ) in Different Evaluation Scenarios. In Design, User Experience, and Usability. Theories, Methods, and Tools for Designing the User Experience, edited by Aaron Marcus, 383–392. Lecture Notes in Computer Science. Cham: Springer. DOI logoGoogle Scholar
Torregrosa Rivero, Daniel
2018 Black-box interactive translation prediction. PhD dissertation, Universidad de Alicante. [URL]
Vermeeren, Arnold P. O. S., Effie Lai-Chong Law, Virpi Roto, Marianna Obrist, Jettie Hoonhout & Kaisa Väänänen-Vainio-Mattila
2010 User Experience Evaluation Methods: Current State and Development Needs. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries, 521–30. NordiCHI ’10. New York, NY, USA: Association for Computing Machinery. DOI logoGoogle Scholar
Vieira, Lucas Nunes
2019 Post-Editing of Machine Translation. In The Routledge Handbook of Translation and Technology. Edited by Minako O’Hagan, 319–335. Routledge. DOI logoGoogle Scholar
Wang, Xiangling, Tingting Wang, Ricardo Muñoz Martín & Yanfang Jia
2021 Investigating Usability in Postediting Neural Machine Translation: Evidence from Translation Trainees’ Self-Perception and Performance. Across Languages and Cultures 22 (1): 100–123. DOI logoGoogle Scholar
Weinberg, Bruce A.
2004 ‘Experience and Technology Adoption’. SSRN Scholarly Paper. Rochester, NY. DOI logoGoogle Scholar