Article in:
Translation and Interpreting Studies: Online-First Articles

Measuring the usability of machine translation in the classroom context

| Hunan University
| Hunan University
| Hunan University
Usability is a key factor for increasing adoption of machine translation. This study aims to measure the usability of machine translation in the classroom context by comparing translation students’ machine translation post-editing output with their manual translation in two comparable translation tasks. Three dimensions of usability were empirically measured: efficiency, effectiveness, and satisfaction. The findings suggest that machine translation post-editing is more efficient than human translation and post-editing produces fewer errors than human translation. While the types of errors vary, errors in terms of accuracy outnumber those related to fluency. In addition, participants perceive the amount of time and work that is saved when post-editing to be greater benefit than the overall utility of post-editing. Likewise, students report a strong desire to learn post-editing skills in training programs.
Keywords: usability, machine translation, post-editing, human translation
Published online: 25 August 2020
https://doi.org/10.1075/tis.18047.yan
References

References

Bevan, Nigel, James Carter, and Susan Harker
2015 “ISO 9241-11 revised: What have we learnt about usability since 1998?” In Human Computer Interaction: Design and Evaluation, ed. by Kurosu Masaaki, 143–151. Cham: Springer. CrossrefGoogle Scholar
Bowker, Lynne
2020 “Fit-for-purpose translation.” In The Routledge Handbook of Translation and Technology, ed. by Minako O’Hagan, 453–468. London: Routledge.Google Scholar
Bowker, Lynne and Jairo Buitrago Ciro
2015 “Investigating the usefulness of machine translation for newcomers at the public library.” Translation and Interpreting Studies 10(2): 165–186. CrossrefGoogle Scholar
Cadwell, Patrick, Sharon O’Brien, and Carlos Teixeira
2018 “Resistance and accommodation: Factors for the (non-) adoption of machine translation among professional translators.” Perspectives 26(3): 301–321. CrossrefGoogle Scholar
Carl, Michael
2012 “Translog-II: a program for recording user activity data for empirical reading and writing research.” In The 8th International Conference on Language Resources and Evaluation, ed. by Nicoletta Calzolari, et al., 21–27. Istanbul.Google Scholar
Carl, Michael and Toledo Cristina Báez
2019 “Machine translation errors and the translation process: A study across different languages.” The Journal of Specialised Translation 31: 107–132.Google Scholar
Cohen, Jacob
1988 Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Hillsdale, NJ: Erlbaum.Google Scholar
Daems, Joke, et al.
2016 “The effectiveness of consulting external resources during translation and post-editing of general text types.” In New Directions in Empirical Translation Process, ed. by Carl Michael, Bangalore Srinivas, and Schaeffer Moritz, 111–133. Cham: Springer. CrossrefGoogle Scholar
Daems, Joke, Sonia Vandepitte, Robert Hartsuiker, and Lieve Macken
2017a “Translation methods and experience: A comparative analysis of human translation and post-editing with student and professional translators.” Meta 62(2): 246–270. CrossrefGoogle Scholar
2017b “Identifying the machine translation error types with the greatest impact on post-editing effort.” Frontiers in Psychology 8: 1282. CrossrefGoogle Scholar
Davis, Fred
1989 “Perceived usefulness, perceived ease of use, and user acceptance of information technology.” MIS Quarterly 13(3):319–340. CrossrefGoogle Scholar
Davis, Fred, Richard Bagozzi, and Paul Warshaw
1989 “User acceptance of computer technology: A comparison of two theoretical models.” Management Science 35(8): 982–1003. CrossrefGoogle Scholar
De Almeida, Giselle and Sharon O’Brien
2010 “Analysing post-editing performance: Correlations with years of translation experience.” In Proceedings of the 14th Annual Conference of the European Association for Machine Translation, ed. by François Yvon and Viggo Hansen, 1–8. Saint-Raphaël.Google Scholar
Doherty, Stephen and Dorothy Kenny
2014 “The design and evaluation of a statistical machine translation syllabus for translation students.” The Interpreter and Translator Trainer 8(2): 295–315. CrossrefGoogle Scholar
Doherty, Stephen and Sharon O’Brien
2014 “Assessing the usability of raw machine translated output: A user-centered study using eye tracking.” International Journal of Human-Computer Interaction 30(1): 40–51. CrossrefGoogle Scholar
Ducar, Cynthia and Deborah Houk Schocket
2018 “Machine translation the L2 classroom: pedagogical solutions for making peace with Google Translate.” Foreign Language Annals 51(4): 779–795. CrossrefGoogle Scholar
Fiederer, Rebecca and Sharon O’Brien
2009 “Quality and machine translation: A realistic objective?” The Journal of Specialised Translation 11: 52–74.Google Scholar
Flanagan, Marian and Tina Paulsen Christensen
2014 “Testing post-editing guidelines: How translation trainees interpret them and how to tailor then for translator training purposes.” The Interpreter and Translator Trainer 8(2):257–275. CrossrefGoogle Scholar
Flesch, Rudolf
1948 “A new readability yardstick.” Journal of Applied Psychology 32: 221–223. CrossrefGoogle Scholar
García, Ignacio
2010 “Is machine translation ready yet?” Target 22(1): 7–21. CrossrefGoogle Scholar
2011 “Translating by post-editing: Is it the way forward?” Machine Translation 25: 217–237. CrossrefGoogle Scholar
Germann, Ulrich
2008 “Yawat: Yet another world alignment tool.” In Proceedings of 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies, 20–23. Ohio: Columbus.Google Scholar
Gile, Daniel
1994 “Methodological aspects of interpretation and translation research.” In Bridge the Gap: Empirical Research in Simultaneous Interpretation, ed. by Sylvie Lambert and Barbara Moser-Mercer, 39–56. Philadelphia, PA: John Benjamins. CrossrefGoogle Scholar
Guerberof Arenas, Ana
2012 “Productivity and Quality in the Post-editing of Outputs from Translation memories and Machine translation.” Ph.D. dissertation. Universitat Rovira i Virgili, Tarragona.Google Scholar
2014 “Correlations between productivity and quality when post-editing in a professional context.” Machine Translation 28 (3–4): 165–186. CrossrefGoogle Scholar
Hansen, Gyde
2008 “The dialogue in translation process research.” In Translation and Cultural Diversity: Selected proceedings of the XVII FIT World Congress, 386–397. Shanghai: Foreign Language Press.Google Scholar
2013 “The translation process as object of research.” In The Routledge Handbook of Translation Studies, ed. by Carmen Millán and Francesca Bartrina, 88–101. London/New York: Routledge.Google Scholar
Harrati, Nouzha, et al.
2016 “Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis.” Computers in Human Behavior 61: 463–471. CrossrefGoogle Scholar
ISO9241-11
2018 “Ergonomics of human-system interaction-Part 11: Usability: Definitions and concepts.” ISO 9241-11.Google Scholar
Jia, Yanfang, Michael Carl, and Xiangling Wang
2019 “How does the post-editing of neural machine translation compare with from-scratch translation? A product and process study.” The Journal of Specialised Translation 31: 60–85.Google Scholar
Kenny, Dorothy
2018 “Sustaining disruption? The transition from statistical to neural machine translation.” Revista Tradumàtica 16: 59–70. CrossrefGoogle Scholar
Kenny, Dorothy and Stephen Doherty
2014 “Statistical machine translation in the translation curriculum: Overcoming obstacles and empowering translators.” The Interpreter and Translator Trainer 8(2): 276–294. CrossrefGoogle Scholar
Kingscott, Geoffrey
2002 “Technical translation and related disciplines.” Perspectives 10(4): 247–255. CrossrefGoogle Scholar
Koponen, Maarit
2010 “Assessing machine translation quality with error analysis.” In Electronic Proceedings of the KaTu Symposium on Translation and Interpreting Studies 4: 1–12.Google Scholar
2015 “How to teach machine translation post-editing? Experiences from a post-editing course.” In 4th Workshop on Post-Editing Technology and Practice (WPTP4), 2–15. Miami: Florida.Google Scholar
2016 “Is machine translation post-editing worth the effort? A survey of research into post-editing and effort.” The Journal of Specialised Translation 25: 131–147.Google Scholar
Kortum, Philip and Frederick Oswald
2017 “The impact of personality on the subjective assessment of usability.” International Journal of Human-Computer Interaction 34: 177–186. CrossrefGoogle Scholar
Krüger, Ralph
2019 “A model for measuring the usability of computer-assisted translation tools.” In Challenging Boundaries: New Approaches to Specialized Communication, ed. by Heike Elisabeth Jüngst, Lisa Link, Klaus Schubert, and Christiane Zehrer, 93–117. Berlin: Frank & Timme.Google Scholar
Lacruz, Isable, Michael Denkowski, and Alon Lavie
2014 “Cognitive demand and cognitive effort in PE.” In Third Workshop on PE Technology and Practice, ed. by Sharon O’Brien, Michel Simard, and Lucia Specia, 73–84. AMTA.Google Scholar
Lewis, James R.
2012 “Usability testing.” In Handbook of Human Factors and Ergonomics, ed. by Gavriel Salvendy, 1267–1312. New York: Wiley. CrossrefGoogle Scholar
Lexile
2007 The Lexile Framework for Reading: Theoretical Framework and Development (Tech. Rep). Durham, NC: MetaMetrics, Inc.Google Scholar
Lommel, Arle, Hans Uszkoreit, and Burchardt Aljoscha
2014 “Multidimensional Quality Metrics (MQM): A framework for declaring and describing translation quality metrics.” Tradumàtica 12: 455–463. CrossrefGoogle Scholar
Mariana, Valerie, Troy Cox, and Alan Melby
2015 “The multidimensional quality metrics (MQM) framework: a new framework for translation quality assessment.” The Journal of Specialised Translation 23: 137–161.Google Scholar
Mellinger, Christopher D.
2017 “Translators and machine translation: Knowledge and skills gaps in translator pedagogy.” The Interpreter and Translator Trainer 11(4): 280–293. CrossrefGoogle Scholar
Mellinger, Christopher D. and Gregory M. Shreve
2016 “Match evaluation and over-editing in a translation memory environment.” In Reembedding Translation Process Research, ed. by Ricardo Muñoz Martín, 132–148. Amsterdam: John Benjamins. CrossrefGoogle Scholar
Mellinger, Christopher D. and Thomas A. Hanson
2017 Quantitative Research Methods in Translation and Interpreting Studies. New York: Routledge.Google Scholar
2018 “Interpreter traits and the relationship with technology and visibility.” Translation and Interpreting Studies 13(3): 366–392. CrossrefGoogle Scholar
MetaMetrics
2018 About Lexile ® Measures for Reading. https://​lexile​.com​/educators​/understanding​-lexile​-measures​/about​-lexile​-measures​-for​-reading. Last accessed 23 May 2020.
Moorkens, Joss
2018 “What to expect from neural machine translation: A practical in-class translation evaluation exercise.” The Interpreter and Translator Trainer 12(4): 375–387. CrossrefGoogle Scholar
Moorkens, Joss, Antonio Toral, Sheila Castilho, and Andy Way
2018 “Translators’ perceptions of literary post-editing using statistical and neural machine translation.” Translation Space 7(2): 240–262. CrossrefGoogle Scholar
O’Brien, Sharon
2004 “Machine translatability and post-editing effort: how do they relate?” Translating and the Computer 26:1–31.Google Scholar
2007 “An empirical investigation of temporal and technical post-editing effort.” Translation and Interpreting Studies 2(1): 83–136. CrossrefGoogle Scholar
2011 “Towards predicting post-editing productivity.” Machine Translation 25: 197–215. CrossrefGoogle Scholar
Plitt, Mirko and François Masselot
2010 “A productivity test of statistical machine translation PE in a typical localization context.” Prague Bulletin of Mathematical Linguistics 93: 7–16. CrossrefGoogle Scholar
Pym, Anthony
2013 “Translation skill-sets in a machine-translation age.” Meta 58(3): 487–503. CrossrefGoogle Scholar
R Core Team
2018 “R: A language and environment for statistical computing.” R Foundation for Statistical Computing. Vienna. www​.R​-project​.org. Last accessed 23 May 2020.
Raita, Eeva and Antti Oulasvira
2011 “Too good to be bad: Favorable product expectations boost subjective usability ratings.” Interacting with Computers 23: 363–371. CrossrefGoogle Scholar
Rossi, Caroline
2017 “Introducing statistical machine translation in translator training: From users and perceptions to course design, and back again.” Tradumàtica 15: 48–62. CrossrefGoogle Scholar
Rossi, Caroline and Jean-Pierre Chevrot
2019 “Uses and perceptions of machine translation at the European Commission.” The Journal of Specialised Translation 31: 201–216.Google Scholar
Sakamoto, Akiko
2019 “Unintended consequences of translation technologies: from project managers’ perspectives.” Perspectives 27(1): 58–73. CrossrefGoogle Scholar
Sánchez-Gijón, Pilar and Olga Torres-Hostench
2014 “MT Post-editing into the mother tongue or into a foreign language? Spanish-to-English MT translation output post-edited by translation trainees.” In Proceedings of the Third Workshop on Post-editing Technology and Practice, ed. by Sharon O’Brien, Michel Simard and Lucia Specia, 5–17. Vancouver.Google Scholar
Shuttleworth, Mark
2002 “Combing MT and TM on a technology-oriented translation masters: aims and perspectives.” In Proceedings of the 6th EAMT Workshop on Teaching Machine Translation, 123–129. Manchester.Google Scholar
Suojanen, Tytti, Kaisa Koskinen, and Tiina Tuominen
2015 User-Centered Translation. London/New York: Routledge.Google Scholar
Temizöz, Ö.
2016 “Postediting machine translation output: subject-matter experts versus professional translators.” Perspectives 24(4): 2–18. CrossrefGoogle Scholar
Temnikova, Irina
2010 “Cognitive evaluation approach for a controlled language PE experiment.” In Proceedings of the 7th International Conference on Language Resources and Evaluation, ed. by Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner and Daniel Tapias, 3485–3490. Valletta.Google Scholar
Tirkkonen-Condit, Sonja
1990 “Professional vs. Non professional translation: A think-aloud protocol study.” In Learning, Keeping and Using Language: Selected papers from the 8th World Congress of Applied Linguistics, ed. by M. A. K. Halliday, John Gibbons, and Howard Nicholas, 381–394. Amsterdam: John Benjamins. CrossrefGoogle Scholar
Thode, Henry
2002 Testing for Normality. New York: Marcel Dekker. CrossrefGoogle Scholar
Trace, Jonathan, Gerriet Janssen, and Valerie Meier
2015 “Measuring the impact of rater negotiation in writing performance assessment.” Language Testing 34: 3–22. CrossrefGoogle Scholar
Van der Heijden, Hans
2004 “User acceptance of hedonic information systems.” MIS Quarterly 28(4): 695–704. CrossrefGoogle Scholar
Wang, Huashu
2018 “The development of translation technology in the era of big data.” In Restructuring Translation Education: Implications from China for the Rest of the World, ed. by Feng Yue, et al., 13–26. Singapore: Springer.Google Scholar
Wu, Jen-Her and Shu-Ching Wang
2005 “What drives mobile commerce? An empirical evaluation of the revised technology acceptance model.” Information & Management 42: 719–729. CrossrefGoogle Scholar
Yamada, Masaru
2019 “The impact of Google neural machine translation on post-editing by student translators.” The Journal of Specialized Translation 31: 87–105.Google Scholar
Yang, Yanxia and Xiangling Wang
2019 “Modeling the intention to use machine translation for student translators: An extension of technology acceptance model.” Computers & Education 133: 116–126. CrossrefGoogle Scholar
Zaharias, Panagiotis
2009 “Developing a usability evaluation method for e-learning applications: From functional usability to motivation to learn.” International Journal of Human-computer Interaction 25(1): 75–98. CrossrefGoogle Scholar
Zhai, Yuming, Aurélien Max, and Anne Vilnat
2018 “Construction of a multilingual corpus annotated with translation relations.” In Proceedings of the first workshop on linguistic resources for natural language processing, ed. by Peter Machonis, Anabela Barreiro, Kristina Kocijan, and Max Silberztein, 102–111. New Mexico: Santa Fe.Google Scholar