Cover not available
Article published In:
Translation, Cognition & Behavior
Vol. 6:1 (2023) ► pp.60–86

The impact of traditional and interactive post-editing on Machine Translation User Experience, quality, and productivity

ORCID logo | Dublin City University
ORCID logo | Dublin City University
ORCID logo | University College Dublin
This paper presents a user study with 15 professional translators in the English-Spanish combination. We present the concept of Machine Translation User Experience (MTUX) and compare the effects of traditional post-editing (TPE) and interactive post-editing (IPE) on MTUX, translation quality and productivity. Results suggest that translators prefer IPE to TPE because they are in control of the interaction in this new form of translator-computer interaction and feel more empowered in their interaction with Machine Translation. Productivity results also suggest that IPE may be an interesting alternative to TPE, given the fact that translators worked faster in IPE even though they had no experience in this new machine translation post-editing modality, but were already used to TPE.
Keywords: machine translation, user experience, human-computer interaction, interactive post-editing, translation quality evaluation
Article outline
  • 1.Introduction
  • 2.Traditional vs interactive post-editing
  • 3.Machine Translation User Experience (MTUX)
  • 4.Methodology
    • Participants
    • Texts
    • IPE workbench
    • Design of the controlled study
    • Measures for MTUX, productivity, and quality
      • Machine Translation User Experience (MTUX)
      • Translation productivity
      • Translation quality
  • 5.Results
    • Analysis approach
    • RQ1. Do the MTPE modalities (TPE or IPE) impact MTUX in a statistically significantly way?
    • RQ2. As translators have more experience in TPE than in IPE, is MTUX impacted by increased experience with the system in a specific MTPE modality?
    • RQ3. Do the MTPE modalities (TPE or IPE) impact translator performance measures (translation quality and productivity) in a statistically significantly way?
      • Translation quality
      • Translation productivity
    • RQ4. Does MTUX correlate with translator performance measures (translation quality and productivity)?
  • 6.Discussion
    • IPE showed significant MTUX benefits to TPE
    • IPE and TPE experience over time—The need for a longitudinal study
    • The relationship between MTUX and performance
  • 7.Conclusions
  • Acknowledgements
  • Notes
  • References
Available under the Creative Commons Attribution (CC BY) 4.0 license.

For any use beyond this license, please contact the publisher at [email protected].

Published online: 13 June 2023
DOI logo
https://doi.org/10.1075/tcb.00077.bri
References (60)
References
Alabau Gonzalvo, Vicent, Ragnar Bonkb, Christian Buck, Michael Carlb, Francisco Casacuberta Nolla, Mercedes García Martínez, Jesús González Rubio, Philip Koehn, Luis Alberto Leiva Torres, Bartolomé Mesa Lao, Daniel Ortiz Martínez, Herve Saint-Amand, Germán Sanchis Trilles & Chara Tsoukalak. 2013. CASMACAT: An open source workbench for advanced computer aided translation. Prague Bulletin of Mathematical Linguistics, 1001: 101–12. De Gruyter Open. DOI logoGoogle Scholar
Alabau Gonzalvo, Vicent, Michael Carl, Mercedes García Martínez & Jesús González Rubio. 2016. Learning Advanced Post-Editing. In New Directions in Empirical Translation Process Research. Edited by Michael Carl, Srinivas Bangalore, Moritz Schaeffer, 95–110. Cham: Springer. DOI logoGoogle Scholar
Alves, Fabio, Karina Sarto Szpak, José Luiz Gonçalves, Kyoko Sekino, Marceli Aquino, Rodrigo Araújo e Castro, Arlene Koglin, Norma B. de Lima Fonseca, and Bartolomé Mesa Lao. 2016. Investigating Cognitive Effort in Post-Editing: A Relevance-Theoretical Approach. In Eyetracking and Applied Linguistics. Edited by Silvia Hansen-Schirra & Sambor Grucza, 109–42. Language Science Press. [URL]
Artstein, Ron. 2017. Inter-Annotator Agreement. In Handbook of Linguistic Annotation. Edited by Nancy Ide & James Pustejovsky, 297–313. Dordrecht: Springer Netherlands. DOI logoGoogle Scholar
Bertoldi, Nicola, Davide Caroselli & Marcello Federico. 2018. The ModernMT Project. In Proceedings of the 21st Annual Conference of the European Association for Machine Translation. Edited by Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez, Miquel Esplà-Gomis, Maja Popovic, Celia Rico, André Martins, Joachim Van den Bogaert, Mikel L. Forcada, 345–347. Alicante: European Association for Machine Translation. [URL]
Bowker, Lynne & Jairo Buitrago Ciro. 2018. Localizing Websites Using Machine Translation: Exploring Connections between User EXperience and Translatability. In The Human Factor in Machine Translation. Edited by Sin-wai Chan, 29–52. London: Routledge. DOI logoGoogle Scholar
Briva-Iglesias, Vicent. 2021. Traducción humana vs. traducción automática: análisis contrastivo e implicaciones para la aplicación de la traducción automática en traducción jurídica. Mutatis Mutandis. Revista Latinoamericana de Traducción 14 (2): 571–600. DOI logoGoogle Scholar
. 2022. English-Catalan Neural Machine Translation: State-of-the-Art Technology, Quality, and Productivity. Tradumàtica: Tecnologies de La Traducció 201: 149–76. DOI logoGoogle Scholar
Briva-Iglesias, Vicent, Sharon O’Brien & Benjamin R. Cowan. 2023. Measuring Machine Translation User Experience: A Comparison between AttrakDiff and User Experience Questionnaire. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation.Google Scholar
. Forthcoming. Translators’ Pre-Task Perceptions of CAT Tools and MTPE, and Their Relationship with Translation Quality: Implications for Training.
Cadwell, Patrick, Sheila Castilho, Sharon O’Brien & Linda Mitchell. 2016. Human Factors in Machine Translation and Post-Editing among Institutional Translators. Translation Spaces 5 (2): 222–43. DOI logoGoogle Scholar
Cadwell, Patrick, Sharon O’Brien & Carlos S. C. Teixeira. 2018. Resistance and Accommodation: Factors for the (Non-) Adoption of Machine Translation among Professional Translators. Perspectives 26 (3): 301–21. DOI logoGoogle Scholar
Castilho, Sheila. 2016. Measuring Acceptability of Machine Translated Enterprise Content. PhD Thesis, Dublin City University. Doctoral, Dublin City University. Faculty of Humanities and Social Sciences. [URL]
Castilho, Sheila, Stephen Doherty, Federico Gaspari & Joss Moorkens. 2018. Approaches to Human and Machine Translation Quality Assessment. In Translation Quality Assessment: From Principles to Practice. Edited by Joss Moorkens, Sheila Castilho, Federico Gaspari & Stephen Doherty, 9–38. Machine Translation: Technologies and Applications. Cham: Springer International Publishing. DOI logoGoogle Scholar
Daems, Joke & Lieve Macken. 2019. Interactive Adaptive SMT versus Interactive Adaptive NMT: A User Experience Evaluation. Machine Translation 33 (1): 117–34. DOI logoGoogle Scholar
Dix, Alan. 2010. Human-Computer Interaction: A Stable Discipline, a Nascent Science, and the Growth of the Long Tail. Interacting with Computers 22 (1): 13–27. DOI logoGoogle Scholar
Esteban Lauzán, José, José Lorenzo Mon, Antonio Sánchez Valderrábanos & Guy Lapalme. 2004. TransType2: An Innovative Computer-Assisted Translation System. In Proceedings of the ACL 2004 on Interactive Poster and Demonstration Sessions, 94–97. Barcelona, Spain: Association for Computational Linguistics. DOI logoGoogle Scholar
Etchegoyhen, Thierry, Anna Fernández Torné, Andoni Azpeitia Zaldua, Eva Martínez García & Anna Matamala Ripoll. 2018. Evaluating Domain Adaptation for Machine Translation Across Scenarios. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan: European Language Resources Association. [URL]
Forlizzi, Jodi & Katja Battarbee. 2004. Understanding Experience in Interactive Systems. In Proceedings of the 5th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, 261–68. DIS ’04. New York, NY, USA: Association for Computing Machinery. DOI logoGoogle Scholar
Gaspari, Federico, Antonio Toral Ruiz, Sudip Kumar Naskar, Declan Groves & Andy Way. 2014. Perception vs Reality: Measuring Machine Translation Post-Editing Productivity. In Proceedings of the 11th Conference of the Association for Machine Translation in the Americas, 60–72. Vancouver, Canada: Association for Machine Translation in the Americas.Google Scholar
Green, Spence. 2016. Interactive Machine Translation. In Conferences of the Association for Machine Translation in the Americas. Invited talk.Google Scholar
Green, Spence, Jason Chuang, Jeffrey Heer & Christopher D. Manning. 2014. Predictive Translation Memory: A Mixed-Initiative System for Human Language Translation. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, 177–187. Honolulu Hawaii USA: ACM. DOI logoGoogle Scholar
Green, Spence, Sida I. Wang, Jason Chuang, Jeffrey Heer, Sebastian Schuster & Christopher D. Manning. 2014. Human Effort and Machine Learnability in Computer Aided Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1225–1236. Doha, Qatar: Association for Computational Linguistics. DOI logoGoogle Scholar
Guerberof Arenas, Ana, Joss Moorkens & Sharon O’Brien. 2021. The Impact of Translation Modality on User Experience: An Eye-Tracking Study of the Microsoft Word User Interface. Machine Translation 35 (2): 205–37. DOI logoGoogle Scholar
Guerberof Arenas, Ana. 2008. Productivity and Quality in the Post-Editing of Outputs from Translation Memories and Machine Translation. Localisation Focus The International Journal of Localisation, 7 ( 1 ), 11–21.Google Scholar
ISO. 2018. ISO 9241-11:2018(En), Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts. 2018. [URL]
Karakanta, Alina, Luisa Bentivogli, Mauro Cettolo, Matteo Negri & Marco Turchi. 2022. Post-Editing in Automatic Subtitling: A Subtitlers’ Perspective. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, 261–70. Ghent, Belgium: European Association for Machine Translation. [URL]
Knowles, Rebecca, Marina Sánchez Torrón & Philipp Koehn. 2019. Neural Interactive Translation Prediction. Machine Translation 331: 135–134. DOI logoGoogle Scholar
Koehn, Philipp. 2009. A Web-Based Interactive Computer Aided Translation Tool. In Proceedings of the ACL-IJCNLP 2009 Software Demonstrations. Edited by Gary Geunbae Lee, Sabine Schulte im Walde, 17–20. Suntec, Singapore: Association for Computational Linguistics. [URL]. DOI logo
Koehn, Philipp, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, Evan Herbst, Hieu Hoang, et al 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions—ACL. Edited by Sophia Ananiadou, 177–180. Prague: Association for Computational Linguistics. DOI logoGoogle Scholar
Koponen, Maarit. 2012. Comparing Human Perceptions of Post-Editing Effort with Post-Editing Operations. In Proceedings of the Seventh Workshop on Statistical Machine Translation. Edited by Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut & Lucia Specia, 181–190. Montréal, Canada: Association for Computational Linguistics.Google Scholar
Koponen, Maarit, Umut Sulubacak, Kaisa Vitikainen & Jörg Tiedemann. 2020. MT for Subtitling: Investigating Professional Translators’ User Experience and Feedback. In Proceedings of 1st Workshop on Post-Editing in Modern-Day Translation. Edited by John E. Ortega, Marcello Federico, Constantin Orasan, Maja Popovic, 79–92. Virtual: AMTA. [URL]
Langlais, Philippe, George Foster & Guy Lapalme. 2000. TransType: A Computer-Aided Translation Typing System. In ANLP-NAACL 2000 Workshop: Embedded Machine Translation Systems, 1–6. [URL]
Läubli, Samuel, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen & Antonio Toral Ruiz. 2020. A Set of Recommendations for Assessing Human–Machine Parity in Language Translation. Journal of Artificial Intelligence Research 671 (March): 653–672–653–72. DOI logoGoogle Scholar
Läubli, Samuel & Spence Green. 2019. Translation Technology Research and Human–Computer Interaction. In The Routledge Handbook of Translation and Technology, edited by Minako O’Hagan, 370–83. New York, NY, USA: Routledge. DOI logoGoogle Scholar
Läubli, Samuel, Patrick Simianer, Joern Wuebker, Geza Kovacs, Rico Sennrich & Spence Green. 2022. The Impact of Text Presentation on Translator Performance. Target. International Journal of Translation Studies 34 (2): 309–342. [URL]. DOI logo
Laugwitz, Bettina, Theo Held & Martin Schrepp. 2008. Construction and Evaluation of a User Experience Questionnaire. International Journal of Interactive Multimedia and Artificial Intelligence 4 (4): 63–76. DOI logoGoogle Scholar
Macklovitch, Elliott. 2006. TransType2: The Last Word. In Proceedings of the 5th Edition of the International Conference on Language Resources and Evaluation. Edited by Nicoletta Calzolari, Khalid Choukri, Aldo Gangemi, Bente Maegaard, Joseph Mariani, Jan Odijk, Daniel Tapias, 1–6. Genoa, Italy: International Conference on Language Resources and Evaluation (LREC).Google Scholar
Martikainen, Hanna. 2022. Ghosts in the Machine: Can Adaptive MT Help Reclaim a Place for the Human in the Loop?. In Portail HAL Sorbonne Nouvelle. [URL]
Moorkens, Joss. 2020. “A Tiny Cog in a Large Machine”: Digital Taylorism in the Translation Industry. Translation Spaces 9 (1): 12–34. DOI logoGoogle Scholar
Nurminen, Mary. 2019. Decision-Making, Risk, and Gist Machine Translation in the Work of Patent Professionals. In Proceedings of the 8th Workshop on Patent and Scientific Literature Translation, 32–42. Dublin: European Association for Machine Translation. [URL]
Nurminen, Mary & Niko Papula. 2018. Gist MT Users: A Snapshot of the Use and Users of One Online MT Tool. In Edited by Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez, Miquel Esplà-Gomis, Maja Popovic, Celia Rico, André Martins, Joachim Van den Bogaert, Mikel L. Forcada, 199–208. Alicante: European Association for Machine Translation. [URL]
O’Brien, Sharon. 2006. Pauses as Indicators of Cognitive Effort in Post-Editing Machine Translation Output. Across Languages and Cultures 7 (1): 1–21. DOI logoGoogle Scholar
. 2022. How to Deal with Errors in Machine Translation: Post-Editing. In Machine Translation for Everyone, 105–20. Berlin: Language Science Press. DOI logoGoogle Scholar
O’Brien, Sharon, Maureen Ehrensberger-Dow, Marcel Hasler & Megan Connolly. 2017. Irritating CAT Tool Features That Matter to Translators. Hermes: Journal of Language and Communication in Business 561: 145–62.Google Scholar
Olohan, Maeve. 2011. Translators and Translation Technology: The Dance of Agency. Translation Studies 4 (3): 342–57. DOI logoGoogle Scholar
Paz, Freddy & Jose Pow-Sang. 2016. A Systematic Mapping Review of Usability Evaluation Methods for Software Development Process 101: 165–78. DOI logoGoogle Scholar
Pérez Macías, Lorena, María del Mar Sánchez Ramos & Celia Rico Pérez. 2020. Study on the Usefulness of Machine Translation in the Migratory Context: Analysis of Translators’ Perceptions. Open Linguistics 6 (1): 68–76. DOI logoGoogle Scholar
Peris Abril, Álvaro, Miguel Domingo Ballester & Francisco Casacuberta Nolla. 2017. Interactive Neural Machine Translation. Computer Speech & Language 451 (September): 201–20. DOI logoGoogle Scholar
Sadiku, Matthew N. O. & Sarhan M. Musa. 2021. A Primer on Multiple Intelligences. Cham: Springer International Publishing. DOI logoGoogle Scholar
Sakamoto, Akiko. 2019. Unintended Consequences of Translation Technologies: From Project Managers Perspectives’. Perspectives 27 (1): 58–73. DOI logoGoogle Scholar
Sánchez-Torrón, Marina. 2017. Productivity in Post-Editing and in Neural Interactive Translation Prediction: A Study of English-to-Spanish Professional Translators. PhD Dissertation. University of Auckland. [URL]
Sánchez Gijón, Pilar, Joss Moorkens & Andy Way. 2019. Post-Editing Neural Machine Translation versus Translation Memory Segments. Machine Translation 33 (1–2): 31–59. DOI logoGoogle Scholar
Sanchis Trilles, Germán, Vicent Alabau Gonzalvo, Christian Buck, Michael Carl, Francisco Casacuberta Nolla, Mercedes García Martínez, Ulrich Germann, Jesús González Rubio, Robin L. Hill, Philipp Koehn, Luis A. Leiva Torres, Bartolomé Mesa Lao, Daniel Ortiz Martínez, Herve Saint-Amand, Chara Tsoukala & Enrique Vidal Ruiz. 2014. Interactive Translation Prediction versus Conventional Post-Editing in Practice: A Study with the CasMaCat Workbench, Machine Translation 28 (3): 217–235. DOI logoGoogle Scholar
Schrepp, Martin, Andreas Hinderks & Jörg Thomaschewski. 2014. Applying the User Experience Questionnaire (UEQ) in Different Evaluation Scenarios. In Design, User Experience, and Usability. Theories, Methods, and Tools for Designing the User Experience, edited by Aaron Marcus, 383–392. Lecture Notes in Computer Science. Cham: Springer. DOI logoGoogle Scholar
Torregrosa Rivero, Daniel. 2018. Black-box interactive translation prediction. PhD dissertation, Universidad de Alicante. [URL]
Vermeeren, Arnold P. O. S., Effie Lai-Chong Law, Virpi Roto, Marianna Obrist, Jettie Hoonhout & Kaisa Väänänen-Vainio-Mattila. 2010. User Experience Evaluation Methods: Current State and Development Needs. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries, 521–30. NordiCHI ’10. New York, NY, USA: Association for Computing Machinery. DOI logoGoogle Scholar
Vieira, Lucas Nunes. 2019. Post-Editing of Machine Translation. In The Routledge Handbook of Translation and Technology. Edited by Minako O’Hagan, 319–335. Routledge. DOI logoGoogle Scholar
Wang, Xiangling, Tingting Wang, Ricardo Muñoz Martín & Yanfang Jia. 2021. Investigating Usability in Postediting Neural Machine Translation: Evidence from Translation Trainees’ Self-Perception and Performance. Across Languages and Cultures 22 (1): 100–123. DOI logoGoogle Scholar
Weinberg, Bruce A. 2004. ‘Experience and Technology Adoption’. SSRN Scholarly Paper. Rochester, NY. DOI logoGoogle Scholar
Cited by (3)

Cited by three other publications

Daems, Joke
2024. Students’ Attitudes Towards Interactive and Adaptive Translation Technology: Four years of Working with Lilt. In New Advances in Translation Technology [New Frontiers in Translation Studies, ], ► pp. 239 ff. DOI logo
Hao, Yu, Ke Hu & Anthony Pym
2024. Who’s afraid of literary post-editing? Performances and Reflections of Student Translators. In New Advances in Translation Technology [New Frontiers in Translation Studies, ], ► pp. 263 ff. DOI logo
Yang, Yanxia
2024. Understanding machine translation fit for language learning: The mediating effect of machine translation literacy. Education and Information Technologies DOI logo

This list is based on CrossRef data as of 27 september 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.