Cover not available
Article published In:
Translation and Interpreting Studies
Vol. 16:1 (2021) ► pp.101–123

Measuring the usability of machine translation in the classroom context

| Hunan University,
| Hunan University,
| Hunan University,
Usability is a key factor for increasing adoption of machine translation. This study aims to measure the usability of machine translation in the classroom context by comparing translation students’ machine translation post-editing output with their manual translation in two comparable translation tasks. Three dimensions of usability were empirically measured: efficiency, effectiveness, and satisfaction. The findings suggest that machine translation post-editing is more efficient than human translation and post-editing produces fewer errors than human translation. While the types of errors vary, errors in terms of accuracy outnumber those related to fluency. In addition, participants perceive the amount of time and work that is saved when post-editing to be greater benefit than the overall utility of post-editing. Likewise, students report a strong desire to learn post-editing skills in training programs.
Keywords: usability, machine translation, post-editing, human translation
Article outline
  • Introduction
  • Relevant research
    • Usability and its assessment
    • Usability and machine translation
    • Usability assessment framework of MT
  • Methodology
    • Participants
    • Instruments
      • Translog-II
      • Questionnaire
      • Quality assessment metrics
    • Translation tasks
      • Translation direction
      • Text selection
      • Metrics for text difficulty, length, and readability
      • Scenario
      • Data collection and procedure
  • Results
    • Efficiency
    • Effectiveness
    • Error count
    • Error type
    • Satisfaction
  • Discussion
  • Conclusion
  • Acknowledgements
  • Note
  • References
Published online: 25 August 2020
DOI logo
https://doi.org/10.1075/tis.18047.yan
References (75)
References
Bevan, Nigel, James Carter, and Susan Harker. 2015. “ISO 9241-11 revised: What have we learnt about usability since 1998?” In Human Computer Interaction: Design and Evaluation, ed. by Kurosu Masaaki, 143–151. Cham: Springer. DOI logoGoogle Scholar
Bowker, Lynne. 2020. “Fit-for-purpose translation.” In The Routledge Handbook of Translation and Technology, ed. by Minako O’Hagan, 453–468. London: Routledge.Google Scholar
Bowker, Lynne and Jairo Buitrago Ciro. 2015. “Investigating the usefulness of machine translation for newcomers at the public library.” Translation and Interpreting Studies 10(2): 165–186. DOI logoGoogle Scholar
Cadwell, Patrick, Sharon O’Brien, and Carlos Teixeira. 2018. “Resistance and accommodation: Factors for the (non-) adoption of machine translation among professional translators.” Perspectives 26(3): 301–321. DOI logoGoogle Scholar
Carl, Michael. 2012. “Translog-II: a program for recording user activity data for empirical reading and writing research.” In The 8th International Conference on Language Resources and Evaluation, ed. by Nicoletta Calzolari, et al., 21–27. Istanbul.Google Scholar
Carl, Michael and Toledo Cristina Báez. 2019. “Machine translation errors and the translation process: A study across different languages.” The Journal of Specialised Translation 311: 107–132.Google Scholar
Cohen, Jacob. 1988. Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Hillsdale, NJ: Erlbaum.Google Scholar
Daems, Joke, et al. 2016. “The effectiveness of consulting external resources during translation and post-editing of general text types.” In New Directions in Empirical Translation Process, ed. by Carl Michael, Bangalore Srinivas, and Schaeffer Moritz, 111–133. Cham: Springer. DOI logoGoogle Scholar
Daems, Joke, Sonia Vandepitte, Robert Hartsuiker, and Lieve Macken. 2017a. “Translation methods and experience: A comparative analysis of human translation and post-editing with student and professional translators.” Meta 62(2): 246–270. DOI logoGoogle Scholar
. 2017b. “Identifying the machine translation error types with the greatest impact on post-editing effort.” Frontiers in Psychology 81: 1282. DOI logoGoogle Scholar
Davis, Fred. 1989. “Perceived usefulness, perceived ease of use, and user acceptance of information technology.” MIS Quarterly 13(3):319–340. DOI logoGoogle Scholar
Davis, Fred, Richard Bagozzi, and Paul Warshaw. 1989. “User acceptance of computer technology: A comparison of two theoretical models.” Management Science 35(8): 982–1003. DOI logoGoogle Scholar
De Almeida, Giselle and Sharon O’Brien. 2010. “Analysing post-editing performance: Correlations with years of translation experience.” In Proceedings of the 14th Annual Conference of the European Association for Machine Translation, ed. by François Yvon and Viggo Hansen, 1–8. Saint-Raphaël.Google Scholar
Doherty, Stephen and Dorothy Kenny. 2014. “The design and evaluation of a statistical machine translation syllabus for translation students.” The Interpreter and Translator Trainer 8(2): 295–315. DOI logoGoogle Scholar
Doherty, Stephen and Sharon O’Brien. 2014. “Assessing the usability of raw machine translated output: A user-centered study using eye tracking.” International Journal of Human-Computer Interaction 30(1): 40–51. DOI logoGoogle Scholar
Ducar, Cynthia and Deborah Houk Schocket. 2018. “Machine translation the L2 classroom: pedagogical solutions for making peace with Google Translate.” Foreign Language Annals 51(4): 779–795. DOI logoGoogle Scholar
Fiederer, Rebecca and Sharon O’Brien. 2009. “Quality and machine translation: A realistic objective?” The Journal of Specialised Translation 111: 52–74.Google Scholar
Flanagan, Marian and Tina Paulsen Christensen. 2014. “Testing post-editing guidelines: How translation trainees interpret them and how to tailor then for translator training purposes.” The Interpreter and Translator Trainer 8(2):257–275. DOI logoGoogle Scholar
Flesch, Rudolf. 1948. “A new readability yardstick.” Journal of Applied Psychology 321: 221–223. DOI logoGoogle Scholar
García, Ignacio. 2010. “Is machine translation ready yet?” Target 22(1): 7–21. DOI logoGoogle Scholar
. 2011. “Translating by post-editing: Is it the way forward?” Machine Translation 251: 217–237. DOI logoGoogle Scholar
Germann, Ulrich. 2008. “Yawat: Yet another world alignment tool.” In Proceedings of 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies, 20–23. Ohio: Columbus. DOI logoGoogle Scholar
Gile, Daniel. 1994. “Methodological aspects of interpretation and translation research.” In Bridge the Gap: Empirical Research in Simultaneous Interpretation, ed. by Sylvie Lambert and Barbara Moser-Mercer, 39–56. Philadelphia, PA: John Benjamins. DOI logoGoogle Scholar
Guerberof Arenas, Ana. 2012. “Productivity and Quality in the Post-editing of Outputs from Translation memories and Machine translation.” Ph.D. dissertation. Universitat Rovira i Virgili, Tarragona.Google Scholar
. 2014. “Correlations between productivity and quality when post-editing in a professional context.” Machine Translation 28 (3–4): 165–186. DOI logoGoogle Scholar
Hansen, Gyde. 2008. “The dialogue in translation process research.” In Translation and Cultural Diversity: Selected proceedings of the XVII FIT World Congress, 386–397. Shanghai: Foreign Language Press.Google Scholar
. 2013. “The translation process as object of research.” In The Routledge Handbook of Translation Studies, ed. by Carmen Millán and Francesca Bartrina, 88–101. London/New York: Routledge.Google Scholar
Harrati, Nouzha, et al. 2016. “Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis.” Computers in Human Behavior 611: 463–471. DOI logoGoogle Scholar
ISO9241-11. 2018. “Ergonomics of human-system interaction-Part 11: Usability: Definitions and concepts.” ISO 9241-11.Google Scholar
Jia, Yanfang, Michael Carl, and Xiangling Wang. 2019. “How does the post-editing of neural machine translation compare with from-scratch translation? A product and process study.” The Journal of Specialised Translation 311: 60–85.Google Scholar
Kenny, Dorothy. 2018. “Sustaining disruption? The transition from statistical to neural machine translation.” Revista Tradumàtica 161: 59–70. DOI logoGoogle Scholar
Kenny, Dorothy and Stephen Doherty. 2014. “Statistical machine translation in the translation curriculum: Overcoming obstacles and empowering translators.” The Interpreter and Translator Trainer 8(2): 276–294. DOI logoGoogle Scholar
Kingscott, Geoffrey. 2002. “Technical translation and related disciplines.” Perspectives 10(4): 247–255. DOI logoGoogle Scholar
Koponen, Maarit. 2010. “Assessing machine translation quality with error analysis.” In Electronic Proceedings of the KaTu Symposium on Translation and Interpreting Studies 41: 1–12.Google Scholar
. 2015. “How to teach machine translation post-editing? Experiences from a post-editing course.” In 4th Workshop on Post-Editing Technology and Practice (WPTP4), 2–15. Miami: Florida.Google Scholar
. 2016. “Is machine translation post-editing worth the effort? A survey of research into post-editing and effort.” The Journal of Specialised Translation 251: 131–147.Google Scholar
Kortum, Philip and Frederick Oswald. 2017. “The impact of personality on the subjective assessment of usability.” International Journal of Human-Computer Interaction 341: 177–186. DOI logoGoogle Scholar
Krüger, Ralph. 2019. “A model for measuring the usability of computer-assisted translation tools.” In Challenging Boundaries: New Approaches to Specialized Communication, ed. by Heike Elisabeth Jüngst, Lisa Link, Klaus Schubert, and Christiane Zehrer, 93–117. Berlin: Frank & Timme.Google Scholar
Lacruz, Isable, Michael Denkowski, and Alon Lavie. 2014. “Cognitive demand and cognitive effort in PE.” In Third Workshop on PE Technology and Practice, ed. by Sharon O’Brien, Michel Simard, and Lucia Specia, 73–84. AMTA.Google Scholar
Lewis, James R. 2012. “Usability testing.” In Handbook of Human Factors and Ergonomics, ed. by Gavriel Salvendy, 1267–1312. New York: Wiley. DOI logoGoogle Scholar
Lexile. 2007. The Lexile Framework for Reading: Theoretical Framework and Development (Tech. Rep). Durham, NC: MetaMetrics, Inc.Google Scholar
Lommel, Arle, Hans Uszkoreit, and Burchardt Aljoscha. 2014. “Multidimensional Quality Metrics (MQM): A framework for declaring and describing translation quality metrics.” Tradumàtica 121: 455–463. DOI logoGoogle Scholar
Mariana, Valerie, Troy Cox, and Alan Melby. 2015. “The multidimensional quality metrics (MQM) framework: a new framework for translation quality assessment.” The Journal of Specialised Translation 231: 137–161.Google Scholar
Mellinger, Christopher D. 2017. “Translators and machine translation: Knowledge and skills gaps in translator pedagogy.” The Interpreter and Translator Trainer 11(4): 280–293. DOI logoGoogle Scholar
Mellinger, Christopher D. and Gregory M. Shreve. 2016. “Match evaluation and over-editing in a translation memory environment.” In Reembedding Translation Process Research, ed. by Ricardo Muñoz Martín, 132–148. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Mellinger, Christopher D. and Thomas A. Hanson. 2017. Quantitative Research Methods in Translation and Interpreting Studies. New York: Routledge.Google Scholar
. 2018. “Interpreter traits and the relationship with technology and visibility.” Translation and Interpreting Studies 13(3): 366–392. DOI logoGoogle Scholar
MetaMetrics. 2018. About Lexile ® Measures for Reading. [URL]. Last accessed 23 May 2020.
Moorkens, Joss. 2018. “What to expect from neural machine translation: A practical in-class translation evaluation exercise.” The Interpreter and Translator Trainer 12(4): 375–387. DOI logoGoogle Scholar
Moorkens, Joss, Antonio Toral, Sheila Castilho, and Andy Way. 2018. “Translators’ perceptions of literary post-editing using statistical and neural machine translation.” Translation Space 7(2): 240–262. DOI logoGoogle Scholar
O’Brien, Sharon. 2004. “Machine translatability and post-editing effort: how do they relate?” Translating and the Computer 261:1–31.Google Scholar
. 2007. “An empirical investigation of temporal and technical post-editing effort.” Translation and Interpreting Studies 2(1): 83–136. DOI logoGoogle Scholar
. 2011. “Towards predicting post-editing productivity.” Machine Translation 251: 197–215. DOI logoGoogle Scholar
Plitt, Mirko and François Masselot. 2010. “A productivity test of statistical machine translation PE in a typical localization context.” Prague Bulletin of Mathematical Linguistics 931: 7–16. DOI logoGoogle Scholar
Pym, Anthony. 2013. “Translation skill-sets in a machine-translation age.” Meta 58(3): 487–503. DOI logoGoogle Scholar
R Core Team 2018. “R: A language and environment for statistical computing.” R Foundation for Statistical Computing. Vienna. [URL]. Last accessed 23 May 2020.
Raita, Eeva and Antti Oulasvira. 2011. “Too good to be bad: Favorable product expectations boost subjective usability ratings.” Interacting with Computers 231: 363–371. DOI logoGoogle Scholar
Rossi, Caroline. 2017. “Introducing statistical machine translation in translator training: From users and perceptions to course design, and back again.” Tradumàtica 151: 48–62. DOI logoGoogle Scholar
Rossi, Caroline and Jean-Pierre Chevrot. 2019. “Uses and perceptions of machine translation at the European Commission.” The Journal of Specialised Translation 311: 201–216.Google Scholar
Sakamoto, Akiko. 2019. “Unintended consequences of translation technologies: from project managers’ perspectives.” Perspectives 27(1): 58–73. DOI logoGoogle Scholar
Sánchez-Gijón, Pilar and Olga Torres-Hostench. 2014. “MT Post-editing into the mother tongue or into a foreign language? Spanish-to-English MT translation output post-edited by translation trainees.” In Proceedings of the Third Workshop on Post-editing Technology and Practice, ed. by Sharon O’Brien, Michel Simard and Lucia Specia, 5–17. Vancouver.Google Scholar
Shuttleworth, Mark. 2002. “Combing MT and TM on a technology-oriented translation masters: aims and perspectives.” In Proceedings of the 6th EAMT Workshop on Teaching Machine Translation, 123–129. Manchester.Google Scholar
Suojanen, Tytti, Kaisa Koskinen, and Tiina Tuominen. 2015. User-Centered Translation. London/New York: Routledge.Google Scholar
Temizöz, Ö. 2016. “Postediting machine translation output: subject-matter experts versus professional translators.” Perspectives 24(4): 2–18. DOI logoGoogle Scholar
Temnikova, Irina. 2010. “Cognitive evaluation approach for a controlled language PE experiment.” In Proceedings of the 7th International Conference on Language Resources and Evaluation, ed. by Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner and Daniel Tapias, 3485–3490. Valletta.Google Scholar
Tirkkonen-Condit, Sonja. 1990. “Professional vs. Non professional translation: A think-aloud protocol study.” In Learning, Keeping and Using Language: Selected papers from the 8th World Congress of Applied Linguistics, ed. by M. A. K. Halliday, John Gibbons, and Howard Nicholas, 381–394. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Thode, Henry. 2002. Testing for Normality. New York: Marcel Dekker. DOI logoGoogle Scholar
Trace, Jonathan, Gerriet Janssen, and Valerie Meier. 2015. “Measuring the impact of rater negotiation in writing performance assessment.” Language Testing 341: 3–22. DOI logoGoogle Scholar
Van der Heijden, Hans. 2004. “User acceptance of hedonic information systems.” MIS Quarterly 28(4): 695–704. DOI logoGoogle Scholar
Wang, Huashu. 2018. “The development of translation technology in the era of big data.” In Restructuring Translation Education: Implications from China for the Rest of the World, ed. by Feng Yue, et al., 13–26. Singapore: Springer.Google Scholar
Wu, Jen-Her and Shu-Ching Wang. 2005. “What drives mobile commerce? An empirical evaluation of the revised technology acceptance model.” Information & Management 421: 719–729. DOI logoGoogle Scholar
Yamada, Masaru. 2019. “The impact of Google neural machine translation on post-editing by student translators.” The Journal of Specialized Translation 311: 87–105.Google Scholar
Yang, Yanxia and Xiangling Wang. 2019. “Modeling the intention to use machine translation for student translators: An extension of technology acceptance model.” Computers & Education 1331: 116–126. DOI logoGoogle Scholar
Zaharias, Panagiotis. 2009. “Developing a usability evaluation method for e-learning applications: From functional usability to motivation to learn.” International Journal of Human-computer Interaction 25(1): 75–98. DOI logoGoogle Scholar
Zhai, Yuming, Aurélien Max, and Anne Vilnat. 2018. “Construction of a multilingual corpus annotated with translation relations.” In Proceedings of the first workshop on linguistic resources for natural language processing, ed. by Peter Machonis, Anabela Barreiro, Kristina Kocijan, and Max Silberztein, 102–111. New Mexico: Santa Fe.Google Scholar
Cited by (6)

Cited by six other publications

Daems, Joke
2024. Students’ Attitudes Towards Interactive and Adaptive Translation Technology: Four years of Working with Lilt. In New Advances in Translation Technology [New Frontiers in Translation Studies, ], ► pp. 239 ff. DOI logo
Yang, Yanxia
2024. Understanding machine translation fit for language learning: The mediating effect of machine translation literacy. Education and Information Technologies DOI logo
Yang, Yanxia, Runze Liu, Xingmin Qian & Jiayue Ni
2023. Performance and perception: machine translation post-editing in Chinese-English news translation by novice translators. Humanities and Social Sciences Communications 10:1 DOI logo
Gong, Yan
2022. Study on Machine Translation Teaching Model Based on Translation Parallel Corpus and Exploitation for Multimedia Asian Information Processing. ACM Transactions on Asian and Low-Resource Language Information Processing DOI logo
Zhou, Xiangyan, Xiangling Wang & Xiaodong Liu
2022. The impact of task complexity and translating self-efficacy belief on students’ translation performance: Evidence from process and product data. Frontiers in Psychology 13 DOI logo
Chernovaty, Leonid & Natalia Kovalchuk
2021. Psycholinguistic Aspects of the Development of Students’ Critical Approach to the Solution of Terminological Problems in Online Translation Learning . East European Journal of Psycholinguistics 8:2 DOI logo

This list is based on CrossRef data as of 5 july 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.