Translators’ perceptions of literary post-editing using statistical and neural machine translation
In the context of recent improvements in the quality of machine translation (MT) output and new use cases being found for that
output, this article reports on an experiment using statistical and neural MT systems to translate literature. Six professional
translators with experience of literary translation produced English-to-Catalan translations under three conditions: translation
from scratch, neural MT post-editing, and statistical MT post-editing. They provided feedback before and after the translation via
questionnaires and interviews. While all participants prefer to translate from scratch, mostly due to the freedom to be creative
without the constraints of segment-level segmentation, those with less experience find the MT suggestions useful.
Article outline
- Introduction
- Related literature
- Perceptions of post-editing
- Neural MT
- MT for literature
- Methodology
- MT systems and training
- Test data: Warbreaker
- Participant profiles
- Experimental setup
- Results
- Measurements and perceptions of post-editing effort
- Temporal effort
- Technical effort
- Users’ perceptions of the task
- Creativity vs. productivity
- Segmentation and context
- Source vs. target (adequacy vs. fluency)
- SMT vs. NMT
- Perception vs. reality
- Alternative methods of machine assistance
- Conclusion
- Acknowledgements
- Notes
-
References
References
Aziz, Wilker, Sheila Castilho M. de Sousa, and Lucia Specia
2012 “
PET: a tool for post-editing and assessing machine translation.” In
The Eighth International Conference on Language Resources and Evaluation, LREC ‘12, Istanbul, Turkey. May 2012.
[URL]
Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio
2014 Neural Machine Translation by Jointly Learning to Align and Translate. Conference paper presented at the ICLR 2015. arXiv preprint, arXiv:1409.0473.
Bar-Hillel, Yehoshua
1960 “
The Present Status of Automatic Translation of Languages.”
Advances in Computers 11: 91–163.


Bentivogli, Luisa, Arianna Bisazza, Mauro Cettolo, Marcello Federico
2017 “
Neural versus Phrase-Based MT Quality: An In-Depth Analysis on English-German and English-French”.
Computer Speech & Language 491: 52–70.


Besacier, Laurent
2014 “
Traduction automatisée d’une œuvre littéraire: une étude pilote.” In
Traitement Automatique du Langage Naturel (TALN). Marseille, France.

Bird, Steven
2006 “
NLTK: The Natural Language Toolkit.” In
Proceedings of the COLING/ACL on Interactive Presentation Sessions, 69–72. Sydney, Australia.


Bojar, Ondřej, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, Marcos Zampieri
2016 “
Findings of the 2016 Conference on Machine Translation (WMT16).” In
Proceedings of the First Conference on Machine Translation 21: 131–198. Berlin, Germany: Association for Computational Linguistics.

Bowker, Lynne
2007 “
Translation Memory and ‘Text’.” In
Lexicography, Terminology, and Translation. Text-Based Studies in Honour of Ingrid Meyer, edited by
Lynne Bowker, 175–187. Ottawa: University of Ottawa Press.

Cadwell, Patrick, Sharon O’Brien, and Carlos S. C. Teixeira
2017 “
Resistance and Accommodation: Factors for the (Non-) Adoption of Machine Translation among Professional Translators.”
Perspectives: Studies in Translation Theory and Practice 26 (3): 301–321.


Cadwell, Patrick, Sheila Castilho, Sharon O’Brien, and Linda Mitchell
Carl, Michael, Silke Gutermuth, and Silvia Hansen-Schirra
Castilho, Sheila, and Sharon O’Brien
Castilho, Sheila, Joss Moorkens, Federico Gaspari, Iacer Calixto, John Tinsley, and Andy Way
2017 “
Is Neural Machine Translation the New State of the Art?”
The Prague Bulletin of Mathematical Linguistics 1081: 109–120.


Castilho, Sheila, Joss Moorkens, Federico Gaspari, Rico Sennrich, Vilelmini Sosoni, Panayota Georgakopoulou, Pintu Lohar, Andy Way, Antonio Valerio Miceli Barone, and Maria Gialama
2017 “
A Comparative Quality Evaluation of PBSMT and NMT using Professional Translators.” Conference paper presented at the MT Summit 2017. Nagoya, Japan.
Catford, John C.
1965 A Linguistic Theory of Translation: An Essay in Applied Linguistics. London: Oxford University Press.

Church, Kenneth W., and Eduard H. Hovy
1993 “
Good Applications for Crummy Machine Translation.”
Machine Translation 8 (4): 239–258.


Daems, Joke, Orphée De Clercq, and Lieve Macken
2017 “
Translationese and Post-Editese: How Comparable is Comparable Quality?”
Linguistica Antverpiensia, New Series: Themes in Translation Studies 161: 89–103.

De Almeida, Giselle, and Sharon O’Brien
2010 “
Analysing Post-Editing Performance: Correlations with Years of Translation Experience.” In
Proceedings of the 14th Annual Conference of the European Association for Machine Translation held in St. Raphaël, France.

Durrani, Nadir, Helmut Schmid, and Alexander Fraser
2011 “
A Joint Sequence Translation Model with Integrated Reordering.” In
Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, from June 19 to 24, in Portland, Oregon, 1045–1054.

Gaspari, Federico, Antonio Toral, Sudip Kumar Naskar, Declan Groves, and Andy Way
2014 “
Perception vs Reality: Measuring Machine Translation Post-Editing Productivity.” In
Proceedings of AMTA 2014 Workshop on Post-editing Technology and Practice, Vancouver, 60–72.

Genzel, Dmitriy, Jakob Uszkoreit, and Franz Och
2010 “
‘Poetic’ Statistical Machine Translation: Rhyme and Meter.” In the
EMNLP ’10 Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, Cambridge, Massachusetts, 158–166. Stroudsburg: Association for Computational Linguistics.

Green, Spence, Jeffrey Heer, and Christopher D. Manning
2013 “
The Efficacy of Human Post-Editing for Language Translation.” In the
CHI ’13 Proceedings of the SIGCHI Conference Factors in Computing Systems. New York: ACM Press.


Greene, Erica, Tugba Bodrumlu, and Kevin Knight
2010 “
Automatic Analysis of Rhythmic Poetry with Applications to Generation and Translation.” In the
EMNLP ’10 Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, Cambridge, Massachusetts, 524–533. Stroudsburg: Association for Computational Linguistics.

Guerberof, Ana
2012 “
Productivity and Quality in the Post-Editing of Outputs from Translation Memories and Machine Translation.” PhD Dissertation. Universitat Rovira i Virgili.

Hassan, Hany, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou
2018 “
Achieving Human Parity on Automatic Chinese to English News Translation.” Redmond: Microsoft AI & Research. arXiv:1803.05567.

Heyn, Matthias
1998 “
Translation Memories: Insights and Prospects.” In
Unity in Diversity? Current Trends in Translation Studies, edited by
Lynne Bowker,
Michael Cronin,
Dorothy Kenny, and
Jennifer Pearson, 123–36. Manchester: St. Jerome Publishing.

Jones, Ruth, and Ann Irvine
2013 “
The (Un)Faithful Machine Translator.” In
Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, Sofia, Bulgaria, 96–101. Stroudsburg: Association for Computational Linguistics,
[URL]
Kelly, Nataly
2014 “
Why So Many Translators Hate Translation Technology.”
Huffington Post. The Blog.
[URL]
Klubička, Filip, Antonio Toral, and Víctor M. Sánchez-Cartagena
2017 “
Fine-Grained Human Evaluation of Neural Versus Phrase-Based Machine Translation.”
The Prague Bulletin of Mathematical Linguistics 1081: 121–132.


Koehn, P., and R. Knowles
2017 “
Six Challenges for Neural Machine Translation.” In
Proceedings of the First Workshop on Neural Machine Translation, Vancouver, BC, Canada, 28–39.
[URL].

Koponen, Maarit
2016 “
Is Post-Editing Worth the Effort? A Survey of Research into Post-Editing and Effort.”
JosTrans: Journal of Specialised Translation 251: 131–148.

Krings, Hans P.
2001 Repairing Texts: Empirical Investigations of Machine Translation Post-Editing Processes. Ohio: Kent State University Press.

Lacruz, Isabel, and Gregory M. Shreve
2014 “
Pauses and Cognitive Effort in Post-Editing.” In
Post-Editing of Machine Translation: Processes and Applications, edited by
Sharon O’Brien,
Laura Winther Balling,
Michael Carl,
Michel Simard, and
Lucia Specia, 287–314. Newcastle-Upon-Tyne: Cambridge Scholars.

LeBlanc, Matthieu
2013 “
Translators on Translation Memory (TM). Results of an Ethnographic Study in Three Translation Services and Agencies.”
The International Journal for Translation and Interpreting Research 5 (2): 1–13.


Lommel, Arle, and Donald A. DePalma
2016 “
Europe’s Leading Role in Machine Translation: How Europe Is Driving the Shift to MT.” Technical Report. Boston: Common Sense Advisory.

Martín, Juan Alberto Alonso, and Anna Civil Serra
2014 “
Integration of a Machine Translation System into the Editorial Process Flow of a Daily Newspaper.”
Procesamiento del Lenguaje Natural Revista 531: 193–196.

Moorkens, Joss
2017 “
Under Pressure: Translation in Times of Austerity.”
Perspectives 25 (3): 464–477.


Moorkens, Joss, and Sharon O’Brien
2015 “
Post-Editing Evaluations: Trade-offs between Novice and Professional Participants.” In
Proceedings of the 18th Annual Conference of the European Association for Machine Translation (EAMT 2015), edited by
İIknur Durgar El-Kahlout,
Mehmed Özkan,
Felipe Sánchez-Martínez,
Gema Ramírez-Sánchez,
Fred Hollowood, and
Andy Way, 75–81.

Moorkens, Joss, and Sharon O’Brien
2017 “
Assessing User Interface Needs of Post-Editors of Machine Translation.” In
Human Issues in Translation Technology: The IATIS Yearbook, edited by
Dorothy Kenny, 109–130, Oxford, United Kingdom: Routledge.

Moorkens, Joss, Sharon O’Brien, Igor A. L. Silva, Norma Fonseca, and Fabio Alves
2015 “
Correlations of perceived post-editing effort with measurements of actual effort.”
Machine Translation 29 (3–4): 267–284.


Nida, Eugene
1964 Towards a Science of Translating. Leiden: Brill.

Nitzke, Jean
2016 “
Monolingual Post-Editing: An Exploratory Study on Research Behaviour and Target Text Quality.” In
Eye-tracking and Applied Linguistics, edited by
Silvia Hansen-Schirra, and
Sambor Grucza, 83–109. Berlin: Language Science Press.


PACTE group
2005 “
Investigating Translation Competence: Conceptual and Methodological Issues.”
Meta 50 (2): 609–619.


Plitt, Mirko, and François Masselot
2010 “
A Productivity Test of Statistical Machine Translation Post-Editing in a Typical Localisation Context.”
The Prague Bulletin of Mathematical Linguistics 931: 7–16.


Pym, Anthony
2008 “
Professional Corpora: Teaching Strategies for Work with Online Documentation, Translation Memories and Content Management.”
Chinese Translator’s Journal 29 (2): 41–45.

Reiss, Katharina
1981 “
Type, Kind and Individuality of Text: Decision Making in Translation.”
Poetics Today 2 (4): 121–131.


Sennrich, Rico, Barry Haddow, and Alexandra Birch
2016a “
Improving Neural Machine Translation Models with Monolingual Data.” In
Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics from August 7 to August 12, 2016, 86–96. Berlin, Germany.

Sennrich, Rico, Barry Haddow, and Alexandra Birch
2016b “
Neural Machine Translation of Rare Words with Subword Units.” In
Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics from August 7 to August 12, 2016, 1715–1725. Berlin, Germany.

Sennrich, Rico, Barry Haddow, and Alexandra Birch
2016c “
Controlling Politeness in Neural Machine Translation via Side Constraints.” In
Proceedings of NAACL-HLT 2016, 351–40.

Sennrich, Rico, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde
2017 “
Nematus: A Toolkit for Neural Machine Translation.” In
Proceedings of the Software Demonstrations from the 15th Conference of the European Chapter of the Association for Computational Linguistics, 65–68.
[URL].

Snover, Matthew, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul
2006 “
A Study of Translation Edit Rate with Targeted Human Annotation.”
Proceedings of Association for Machine Translation in the Americas.

Specia, Lucia
2011 “
Exploiting Objective Annotations for Measuring Translation Post-Editing Effort.” In Proceedings of the 15th Conference of the European Association for
Machine Translation, 73–80. Leuven, Belgium.

Specia, Lucia, and Kashif Shah
2018 “
Machine Translation Quality Estimation: Applications and Future Perspectives.” In
Translation Quality Assessment: From Principles to Practice, edited by
Joss Moorkens,
Sheila Castilho,
Federico Gaspari, and
Stephen Doherty, 201–236. Heidelberg: Springer International Publishing.


Taivalkoski-Shilov, Kristiina
2018 “
Ethical Issues Regarding Machine(-assisted) Translation of Literary Texts.”
Perspectives: Studies in Translation Theory and Practice (online first).
Special Issue: Voice, Translation, and Ethics, ed. by
Cecilia Alvstad,
Annjo K. Greenall,
Hanne Jansen, and
Kristiina Taivalkoski-Shilov.


Teixeira, Carlos S. C.
2014 “
Perceived vs. Measured Performance in the Post-Editing of Suggestions from Machine Translation and Translation Memories.” In
Proceedings of the Third Workshop on Post-Editing Technology and Practice (WPTP-3), edited by
Sharon O’Brien,
Michel Simard, and
Lucia Specia, 45–59.

Thouin, Benoît
1982 “
The METEO System.” In
Practical Experience of Machine Translation: Proceedings of Translating and the Computer 1981, edited by
Veronica Lawson, 39–44, Amsterdam: North-Holland Publishing.

Toral, Antonio, and Andy Way
2015 “
Translating Literary Text between Related Languages using SMT.” In
Proceedings of NAACL-HLT Fourth Workshop on Computational Linguistics for Literature, 123–132. Denver, Colorado.


Toral, Antonio, and Andy Way
2018 “
What level of quality can Neural Machine Translation attain on literary text?” In
Translation Quality Assessment: From Principles to Practice, edited by
Joss Moorkens,
Sheila Castilho,
Federico Gaspari, and
Stephen Doherty, 263–287. Heidelberg: Springer International Publishing.


Toral, Antonio, and Victor M. Sánchez-Cartagena
2017 “
A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions.” In
Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017. Valencia, Spain.


Toral, Antonio, Martijn Wieling, and Andy Way
2018 “
Post-editing Effort of a Novel with Statistical and Neural Machine Translation.”
Frontiers in Digital Humanities 51:9.


Vasconcellos, Muriel
1985 “
Machine Aids to Translation: A Holistic Scenario for Maximizing the Technology.” In
Overcoming Language Barriers: The Human/Machine Relationship, Proceedings of the IV Annual Conference on Language and Communication held from December 13 to December 14, 1985 in New York, edited by
Humphrey Tonkin, and
Karen Johnson-Weiner, 27–34. New York: Center for Research and Documentation on World Problems.

Vaswani, Ashish, Yinggong Zhao, Victoria Fossum and David Chiang
2013 “
Decoding with Large-Scale Neural Language Models Improves Translation.” In
Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Washington, 1387–1392. Stroudsburg: Association for Computational Linguistics.

Viera, Lucas Nunes
2014 “
Indices of cognitive effort in machine translation post-editing.”
Machine Translation 28 (3–4):187–216.


Wagner, Elizabeth
1985 “
Post-Editing Systran-A Challenge for Commission Translators.”
Terminologie et Traduction 31: 1–7.

Way, Andy
2013 “
Traditional and Emerging Use-Cases for Machine Translation.” In
Proceedings of Translating and the Computer 351. London, United Kingdom.

Way, Andy
2018a “
Quality Expectations of Machine Translation.” In
Translation Quality Assessment: From Principles to Practice, edited by
Joss Moorkens,
Sheila Castilho,
Federico Gaspari, and
Stephen Doherty, 159–178. Heidelberg: Springer International Publishing.


Way, Andy
2018b “
Machine Translation: Where We Are at Today.” In
The Bloomsbury Companion to Language Industry Studies, edited by
Erik Angelone,
Gary Massey, and
Maureen Ehrensberger-Dow. London: Bloomsbury.

Wu, Yonghui, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean
2016 “
Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation”. arXiv preprint 1609.08144,
[URL]
Cited by
Cited by 21 other publications
Abdel Latif, Muhammad M. M.
2020.
Researching Professional Translator/Interpreter Experiences and Roles. In
Translator and Interpreter Education Research [
New Frontiers in Translation Studies, ],
► pp. 125 ff.

Castilho, Sheila & Natália Resende
2022.
Post-Editese in Literary Translations.
Information 13:2
► pp. 66 ff.

Chang-Soo Lee
2019.
문학번역에서의 기계번역과 인간번역 문체에 대한 전산문체학적 비교 연구.
The Journal of Translation Studies 20:2
► pp. 111 ff.

2022.
How do machine translators measure up to human literary translators in stylometric tests?.
Digital Scholarship in the Humanities 37:3
► pp. 813 ff.

Dorst, Aletta G., Susana Valdez & Heather Bouman
Guerberof-Arenas, Ana & Antonio Toral
Jiménez-Crespo, Miguel A.
Kenny, Dorothy & Marion Winters
Nunes Vieira, Lucas & Elisa Alonso
2020.
Translating perceptions and managing expectations: an analysis of management and production perspectives on machine translation.
Perspectives 28:2
► pp. 163 ff.

O’Brien, Sharon & Maureen Ehrensberger-Dow
Ragni, Valentina & Lucas Nunes Vieira
2022.
What has changed with neural machine translation? A critical review of human factors.
Perspectives 30:1
► pp. 137 ff.

Rico, Celia
2022.
Mind the gap.
Babel. Revue internationale de la traduction / International Journal of Translation 68:5
► pp. 697 ff.

Sakamoto, Akiko
2022.
Translation and Technology. In
The Cambridge Handbook of Translation,
► pp. 55 ff.

Sakamoto, Akiko & Masaru Yamada
Slessor, Stephen
2020.
Tenacious technophobes or nascent technophiles? A survey of the technological practices and needs of literary translators.
Perspectives 28:2
► pp. 238 ff.

Sánchez-Gijón, Pilar
Tardel, Anke
2021.
Measuring Effort in Subprocesses of Subtitling. In
Explorations in Empirical Translation Process Research [
Machine Translation: Technologies and Applications, 3],
► pp. 81 ff.

Yang, Yanxia, Xiangling Wang & Qingqing Yuan
Zhao, Shengfang
2021.
Post-editing Neural Machine Translation Versus Human Translation for Chinese Essays: A Pilot Study. In
Diverse Voices in Chinese Translation and Interpreting [
New Frontiers in Translation Studies, ],
► pp. 399 ff.

Zheng, Jianwei & Wenjun Fan
This list is based on CrossRef data as of 25 february 2023. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.