Article In:
Translation Spaces: Online-First Articles

The Machine Translation Post-Editing Annotation System (MTPEAS)

A standardized and user-friendly taxonomy for student post-editing quality assessment

| UCLouvain
| UCLouvain
ORCID logo | UCLouvain

Machine translation post-editing quality evaluation has received relatively little attention in translation pedagogy to date. It is a time-consuming process that involves the comparison of three texts (source text, machine translation and student post-edited text) and the systematic identification and correction of students’ edits (or absence thereof) of machine translation (MT) output. There are as yet no widely available, standardized, user-friendly annotation systems for use in translator education. In this article, we address this gap by describing the Machine Translation Post-Editing Annotation System (MTPEAS). MTPEAS includes a taxonomy of seven categories that are presented in easy-to-understand terms: Value-adding edits, Successful edits, Unnecessary edits, Incomplete edits, Error-introducing edits, Unsuccessful edits, and Missing edits. We then assess the robustness of the MTPEAS taxonomy in a pilot study of 30 students’ post-edited texts and offer some preliminary findings on students’ MT error identification and correction skills.

Keywords: machine translation post-editing, post-editing quality evaluation, translator education, post-editing training, annotation system, inter-rater agreement
Article outline
  • 1.Introduction: Post-editing quality evaluation in translator education
  • 2.The Machine Translation Post-Editing Annotation System (MTPEAS)
    • 2.1The MTPEAS annotation system
    • 2.2The MTPEAS taxonomy
    • 2.3The MTPEAS decision tree: A user-friendly, graphical representation of the annotation system
  • 3.Pilot Study: Data and methodology
    • 3.1Data
    • 3.2Methodology
  • 4.Pilot Study: Results and discussion
    • 4.1Inter-rater agreement on segment identification
    • 4.2Inter-rater agreement on MTPEAS annotation
    • 4.3Preliminary findings
  • 5.Concluding remarks
  • Acknowledgments
  • Notes
  • References
This content is being prepared for publication; it may be subject to changes.
References
Blain, Frédéric, Jean Senellart, Holger Schwenk, Mirko Plitt, and Johann Roturier
2011 “Qualitative analysis of post-editing for high quality machine translation.” In Proceedings of Machine Translation Summit XIII: Papers, 164–171.Google Scholar
Castagnoli, Sara, Dragos Ciobanu, Natalie Kübler, Kerstin Kunz, and Alexandra Volanschi
2011 “Designing a learner translator corpus for training purposes.” In Corpora, language, teaching, and resources: From theory to practice, edited by Natalie Kübler, 221–248. Peter Lang.Google Scholar
Castilho, Sheila, Stephen Doherty, Federico Gaspari, and Joss Moorkens
2018 “Approaches to Human and Machine Translation Quality Assessment.” In Translation Quality Assessment, edited by Joss Moorkens, Sheila Castilho, Federico Gaspari, and Stephen Doherty, 9–38. Cham: Springer. DOI logoGoogle Scholar
Daems, Joke, Sonia Vandepitte, Robert J. Hartsuiker, and Lieve Macken
2017 “Translation Methods and Experience: A Comparative Analysis of Human Translation and Post-editing with Students and Professional Translators.” Meta 62 (2):245–270. Accessed January 19, 2024. DOI logoGoogle Scholar
De Almeida, Giselle
2013 Translating the post-editor: an investigation of post-editing changes and correlations with professional experience across two Romance languages. PhD Thesis, Dublin City University.
Doherty, Stephen, and Dorothy Kenny
2014 “The design and evaluation of a Statistical Machine Translation syllabus for translation students.” The Interpreter and Translator Trainer 8 (2): 295–315. DOI logoGoogle Scholar
Doherty, Stephen, Joss Moorkens, Federico Gaspari, and Sheila Castilho
2018 “On education and training in translation quality assessment.” In Translation Quality Assessment, edited by Joss Moorkens, Sheila Castilho, Federico Gaspari, and Stephen Doherty, 95–106. Cham: Springer. DOI logoGoogle Scholar
EMT Expert Group
2022 “Competence Framework”. Accessed January 19, 2024. [URL]
Froeliger, Nicolas, Claire Larsonneur, and Giuseppe Sofo
(eds) 2023 Traduction humaine et traitement automatique des langues: Vers un nouveau consensus?/Human Translation and Natural Language Processing: Forging a New Consensus? Studi e ricerche 35. Edizioni Ca’ Foscari – Venice University Press: Venice.Google Scholar
Gene, Viveta, and Lucía Guerrero
2022 “A Common Machine Translation Post-Editing Training Handbook for Academia, Clients, LSPs and Post-Editors.” GALA MTPE Training Special Interest Group.Google Scholar
Ginovart Cid, Clara, and Carme Colominas Ventura
2021 “The MT post-editing skill set: Course descriptions and educators’ thoughts.” In Translation Revision and Post-editing: Industry Practices and Cognitive Processes, edited by Maarit Koponen, Brian S. Mossop, Isabelle Robert, and Giovanna Schocchera, 226–246. London/New York: Routledge.Google Scholar
Granger, Sylviane, and Marie-Aude Lefer
2023 “Learner translation corpora: Bridging the gap between learner corpus research and corpus-based translation studies.” International Journal of Learner Corpus Research 9 (1): 1–28. DOI logoGoogle Scholar
2021 “ Translation-oriented Annotation System manual (Version 2.0) .” CECL Papers 3. Louvain-la-Neuve: Centre for English Corpus Linguistics/Université catholique de Louvain. Accessed January 19, 2024. [URL]
2020 “The Multilingual Student Translation corpus: A resource for translation teaching and research.” Language Resources and Evaluation 541:1183–1199. DOI logoGoogle Scholar
Guerberof, Ana, and Joss Moorkens
2019 “Machine translation and post-editing training as part of a master’s programme.” The Journal of Specialised Translation 311: 217–238.Google Scholar
Hu, Ke, and Patrick Cadwell
2016 “A Comparative Study of Post-editing Guidelines.” Baltic J. Modern Computing 4 (2): 346–353.Google Scholar
Huertas Barros, Elsa, and Juliet Vine
(eds) 2018 New Perspectives in Assessment in Translator Training. London: Routledge.Google Scholar
Kenny, Dorothy
2020 “Technology and translator training.” In The Routledge Handbook of Translation and Technology, edited by Minako O’Hagan, 498–515. London: Routledge.Google Scholar
Kenny, Dorothy and Doherty Stephen
2014 “Statistical machine translation in the translation curriculum: overcoming obstacles and empowering translators.” The Interpreter and Translator Trainer 8 (2): 276–294. DOI logoGoogle Scholar
Konttinen, Kalle, Leena Salmi, and Maarit Koponen
2021 “Revision and post-editing competences in translation education.” In Translation Revision and Post-editing: Industry Practices and Cognitive Processes, edited by Maarit Koponen, Brian S. Mossop, Isabelle Robert, and Giovanna Schocchera, 187–202. London/New York: Routledge.Google Scholar
Koponen, Maarit
2015 “How to teach machine translation post-editing? Experiences from a post-editing course.” In Proceedings of 4th Workshop on post-editing technology and practice, Miami, USA, October 30 to November 3, 2015, 2–15. Accessed January 19, 2024. [URL]
Koponen, Maarit and Leena Salmi
2017 “Post-editing quality: Analysing the correctness and necessity of post-editor corrections.” Linguistica Antverpiensia, New Series: Themes in Translation Studies 161: 137–148.Google Scholar
Koponen, Maarit, Leena Salmi, and Markku Nikulin
2019 “A product and process analysis of post-editor corrections on neural, statistical and rule-based machine translation output.” Machine Translation 33 (1): 61–90. DOI logoGoogle Scholar
Kübler, Natalie, Alexandra Mestivier, and Mojca Pecman
2022 “Using comparable corpora for translating and post-editing complex noun phrases in specialized texts. Insights from English-to-French specialized translation.” In Extending the Scope of Corpus-Based Translation Studies, edited by Sylviane Granger and Marie-Aude Lefer, 237–266. London: Bloomsbury. DOI logoGoogle Scholar
Kunilovskaya, Maria
2015 “How far do we agree on the quality of translation?” English Studies at NBU 1 (1): 18–31. DOI logoGoogle Scholar
Landis, J. Richard, and Koch, Gary G.
1977 “The measurement of observer agreement for categorical data.” Biometrics 331: 159–174. DOI logoGoogle Scholar
Lefer, Marie-Aude, Romane Bodart, Adam Obrusník, and Justine Piette
2023 The Post-Edit Me! project. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, 493–494.Google Scholar
Lefer, Marie-Aude, Romane Bodart, Justine Piette, and Adam Obrusník
2024 “MTPE quality evaluation in translator education: the postedit.me app.” EAMT2024, Sheffield.Google Scholar
Lipnevich, Anastasiya A., and Ernesto Panadero
2021 “A review of feedback models and theories: Descriptions, definitions, and conclusions.” Frontiers in Education 61: 720195. DOI logoGoogle Scholar
Massey, Gary, Maria Piotrowska, and Mariusz Marczak
2023 “(Re-) profiling T&I education: meeting evolution with innovation”. The Interpreter and Translator Trainer 17 (3):325–331. Accessed January 19, 2024. DOI logoGoogle Scholar
Mellinger, Christopher D.
2017 “Translators and machine translation: knowledge and skills gaps in translator pedagogy.” The Interpreter and Translator Trainer 11 (4): 280–293. DOI logoGoogle Scholar
Nitzke, Jean, Anke Tardel, and Silvia Hansen-Schirra
2019 “Training the modern translator – the acquisition of digital competencies through blended learning”. The Interpreter and Translator Trainer 13 (3): 292–306. DOI logoGoogle Scholar
O’Brien, Sharon
2002 “Teaching post-editing: a proposal for course content.” In Proceedings of the 6th EAMT Workshop: Teaching machine translation, Manchester, England, November 14–15, 2002, 99–106. Accessed January 19, 2024. [URL]
Pavlović, Nataša, and Goranka Antunović
2021 “Towards acquiring post-editing abilities through research-informed practical tasks.” Strani jezici: časopis za primijenjenu lingvistiku 50 (2): 185–205. DOI logoGoogle Scholar
Plaza Lara, Cristina
2019 “SWOT Analysis of the Inclusion of Machine Translation and Post -Editing in the Master’s Degrees Offered in the EMT Network.” The Journal of Specialised Translation 311: 260–280.Google Scholar
Rico, Celia, and Diana Gonzalez Pastor
2022 “The Role of Machine Translation in Translation Education: A Thematic Analysis of Translator Educators’ Beliefs.” Translation & Interpreting 14 (1): 177–197. DOI logoGoogle Scholar
Schumacher, Perrine
2023 La post-édition de traduction automatique en contexte d’apprentissage. PhD Thesis, University of Liège and University of Geneva.
Stasimioti, Maria, and Vilelmini Sosoni
2021 “Investigating post-editing: A mixed-methods study with experienced and novice translators in the English-Greek language pair.” In Translation, interpreting, cognition: The way out of the box, edited by Tra&Co Group, 79–104. Berlin: Language Science Press. DOI logoGoogle Scholar
Strobl, Carola
2022 “Revision behaviour of translation students in response to electronic feedback. Insights from a pilot study using Hypal4MUST.” Paper presented at EST conference 2022 , Oslo, Norway, 22–25 June 2022.
Yamada, Masaru
2019 “The impact of Google neural machine translation on post-editing by student translators.” The Journal of Specialised Translation 31 (1): 87–106.Google Scholar