The impact of text presentation on translator performance
Abstract
Widely used computer-aided translation (CAT) tools divide documents into segments, such as sentences, and arrange them side-by-side in a spreadsheet-like view. We present the first controlled evaluation of these design choices on translator performance, measuring speed and accuracy in three experimental text-processing tasks. We find significant evidence that sentence-by-sentence presentation enables faster text reproduction and within-sentence error identification compared to unsegmented text, and that a top-and-bottom arrangement of source and target sentences enables faster text reproduction compared to a side-by-side arrangement. For revision, on the other hand, we find that presenting unsegmented text results in the highest accuracy and time efficiency. Our findings have direct implications for best practices in designing CAT tools.
Keywords:
Publication history
Table of contents
Research into CAT-tool adoption among professional translators shows that poor usability is a major reason for resistance (LeBlanc 2013; O’Brien et al. 2017). The sentence-by-sentence presentation of texts, for example, was criticised by translators for creating an “obstructed view of the text, which in turn disrupts the workflow” (O’Brien et al. 2017, 155). However, the impact of poor usability on translator performance has rarely been tested empirically, and since the motivation for using CAT tools is primarily economic – saving time by leveraging translation suggestions rather than translating from scratch – the design of these tools is unlikely to change until research shows that alternative designs speed up translators or cause them to make fewer mistakes.
References
Arif, Ahmed
Sabbir, and Wolfgang Stuerzlinger
2009 “Analysis
of Text Entry Performance Metrics.” In 2009 IEEE Toronto
International Conference Science and Technology for Humanity (TIC–STH), 26–27 September 2009, Toronto
Canada, 100–105. Piscataway,
NJ: IEEE. 
Baayen, R.
Harald, and Petar Milin
2010 “Analyzing
Reaction Times.” International Journal of Psychological
Research 3 (2): 12–28.
Bates, Douglas, Martin Mächler, Ben Bolker, and Steve Walker
2015 “Fitting
Linear Mixed-Effects Models Using lme4.” Journal of Statistical
Software 67 (1): 1–48.
Bojar, Ondřej, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck et al.
2017 “Findings
of the 2017 Conference on Machine Translation (WMT17).” In WMT 2017:
Second Conference on Machine Translation: Proceedings, edited by Ondřej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck et al., 169–214. Stroudsburg: Association
for Computational Linguistics.
Bojar, Ondřej, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn et al.
2018 “Findings
of the 2018 Conference on Machine Translation (WMT18).” In WMT 2018:
Third Conference on Machine Translation: Proceedings of the Conference, edited
by Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck et al., 272–307. Stroudsburg: Association
for Computational Linguistics.
Bowker, Lynne
2005 “Productivity
vs Quality? A Pilot Study on the Impact of Translation Memory Systems.” Localisation
Focus 4 (1): 13–20.
Campbell, Stuart
1999 “A
Cognitive Approach to Source Text Difficulty in
Translation.” Target 11 (1): 33–63.
Carl, Michael
2010 “A
Computational Framework for a Cognitive Model of Human Translation
Processes.” In Proceedings of Translating and the Computer
32. London: Aslib.
2012 “Translog-II:
A Program for Recording User Activity Data for Empirical Reading and Writing
Research.” In Proceedings of the Eighth International Conference on
Language Resources and Evaluation (LREC ’12), edited by Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet
Uğur Doğan, Bente Maegaard, Joseph Mariani, Asuncion Moreno et al., 4108–4112. Istanbul: European
Language Resources Association.
Castilho, Sheila, Joss Moorkens, Federico Gaspari, Iacer Calixto, John Tinsley, and Andy Way
2017 “Is
Neural Machine Translation the New State of the Art?” The Prague Bulletin of Mathematical
Linguistics 108: 109–120.
Castilho, Sheila, Joss Moorkens, Federico Gaspari, Rico Sennrich, Vilelmini Sosoni, Panayota Georgakopoulou, Pintu Lohar et al.
2017 “A
Comparative Quality Evaluation of PBSMT and NMT Using Professional
Translators.” In Proceedings of Machine Translation Summit
XVI, vol. 1, edited by Sadao Kurohashi and Pascale Fung, 116–131. Nagoya: Asia-Pacific
Association for Machine Translation.
Castilho, Sheila, Joss Moorkens, Federico Gaspari, Rico Sennrich, Andy Way, and Panayota Georgakopoulou
2018 “Evaluating
MT for Massive Open Online Courses.” In Human Evaluation of
Statistical and Neural Machine Translation, edited by Andy Way and Mikel
L. Forcada, special issue
of Machine
Translation 32 (3): 255–278.
Coppers, Sven, Jan
van
den Bergh, Kris Luyten, Karin Coninx, Iulianna
van
der Lek–Ciudin, Tom Vanallemeersch, and Vincent Vandeghinste
2018 “Intellingo:
An Intelligible Translation Environment.” In Proceedings of the 2018
CHI Conference on Human Factors in Computing Systems, edited by Regan Mandryk, Mark Hancock, Mark Perry, and Anna Cox. Montreal: Association
for Computing Machinery.
do
Carmo, Félix, and Joss Moorkens
2020 “Differentiating
Editing, Post-Editing and Revision.” In Translation Revision and
Post-Editing: Industry Practices and Cognitive Processes, edited by Maarit Koponen, Brian Mossop, Isabelle
S. Robert, and Giovanna Scocchera. Abingdon: Routledge.
Dragsted, Barbara
2006 “Computer-Aided
Translation as a Distributed Cognitive Task.” In Distributed
Cognition, edited by Stevan Harnad and Itiel
E. Dror, special issue
of Pragmatics &
Cognition 14 (2): 443–464.
2010 “Coordination
of Reading and Writing Processes in Translation: An Eye on Uncharted
Territory.” In Translation and
Cognition, edited by Gregory
M. Shreve and Erik Angelone, 41–62. Amsterdam: John
Benjamins.
Ehrensberger-Dow, Maureen, Andrea
Hunziker Heeb, Gary Massey, Ursula Meidert, Silke Neumann, and Heidrun Becker
2016 “An
International Survey of the Ergonomics of Professional
Translation.” In Ergonomic Approaches to Professional Practices and
Translator Training, edited by Élisabeth Lavault-Olléon, special
issue of Journal of the Institute of Languages and Cultures of Europe, America, Africa, Asia and
Australia 27.
Engelbart, Douglas
C., and William
K. English
1968 “A
Research Center for Augmenting Human Intellect.” In AFIPS 1968 Fall
Joint Conference, Part
1, 395–410. Washington: Thompson
Book Company.
Ericsson, K.
Anders, and Herbert
A. Simon
1984 Protocol
Analysis: Verbal Reports as Data. Cambridge, MA: MIT
Press.
Federico, Marcello, Alessandro Cattelan, and Marco Trombetti
2012 “Measuring
User Productivity in Machine Translation Enhanced Computer Assisted
Translation.” In Proceedings of the 10th Conference of the
Association for Machine Translation in the Americas. San
Diego: Association for Machine Translation in the
Americas.
Francis, Wendy
S., and Silvia
P. Sáenz
2007 “Repetition
Priming Endurance in Picture Naming and Translation: Contributions of Component
Processes.” Memory &
Cognition 35 (3): 481–493.
Gelman, Andrew, and Jennifer Hill
2007 Data
Analysis Using Regression and Multilevel/Hierarchical Models. New
York: Cambridge University Press.
Green, Spence, Jason Chuang, Jeffrey Heer, and Christopher
D. Manning
2014 “Predictive
Translation Memory: A Mixed-Initiative System for Human Language
Translation.” In UIST ’14: Proceedings of the 27th Annual ACM
Symposium on User Interface Software and Technology, edited by Hrvoje Benko, Mira Dontcheva, and Daniel Wigdor, 177–187. New
York: Association for Computing Machinery.
Hassan, Hany, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang et al.
2018 “Achieving
Human Parity on Automatic Chinese to English News Translation.” arXiv
Preprint 1803.05567.
Hornbæk, Kasper, and Erik Frøkjær
2001 “Reading
of Electronic Documents: The Usability of Linear, Fisheye, and Overview+detail
Interfaces.” In CHI ’01: Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems, edited by Julie Jacko and Andrew Sears, 293–300. New
York: Association for Computing Machinery.
House, Juliane
2013 “Quality
in Translation Studies.” In The Routledge Handbook of
Translation, edited by Carmen Millán and Francesca Bartrina, 534–547. Abingdon: Routledge.
Jakobsen, Arnt
Lykke
2003 “Effects of Think Aloud on
Translation Speed, Revision, and Segmentation.” In Triangulating
Translation: Perspectives in Process Oriented Research, edited by Fabio Alves, 69–95. Amsterdam: John
Benjamins.
Junczys-Dowmunt, Marcin
2019 “Microsoft
Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine
Translation.” In Proceedings of the Fourth Conference on Machine
Translation (WMT19), vol. 2, edited
by Loïc Barrault, Ondřej Bojar, Marta
R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow et al., 225–233. Stroudsburg: Association
for Computational Linguistics.
Karimova, Sariya, Patrick Simianer, and Stefan Riezler
2018 “A
User-Study on Online Adaptation of Neural Machine Translation to Human Post-Edits.” Machine
Translation 32 (4): 309–324.
Kay, Martin
1980 “The
Proper Place of Men and Machines in Language Translation.” Research Report
CSL-80-11. Palo Alto, CA: Xerox Palo Alto Research
Center.
Krings, Hans
P.
1994 Texte Reparieren: Empirische
Untersuchungen Zum Prozeß Der Nachredaktion von Maschinenübersetzungen. Habilitation
thesis. University of Hildesheim.
2001 Repairing Texts: Empirical
Investigations of Machine Translation Post-Editing Processes. Translated by Geoffrey
S. Koby, Gregory
M. Shreve, Katja Mischerikow, and Sarah Litzer. Kent: Kent
State University Press.
Langlais, Philippe, Guy Lapalme, and Sébastien Sauvé
2001 “User
Interface Aspects of a Translation Typing System.” In Advances in
Artificial Intelligence: 14th Biennial Conference of the Canadian Society for Computational Studies of Intelligence, AI
2001, edited by Eleni Stroulia and Stan Matwin, 246–256. Berlin: Springer.
Läubli, Samuel, Rico Sennrich, and Martin Volk
2018 “Has
Machine Translation Achieved Human Parity? A Case for Document-Level
Evaluation.” In Proceedings of the 2018 Conference on Empirical
Methods in Natural Language Processing, edited by Ellen Riloff, Davide Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, 4791–4796. Stroudsburg: Association
for Computational Linguistics.
LeBlanc, Matthieu
2013 “Translators
on Translation Memory (TM): Results of an Ethnographic Study in Three Translation Services and
Agencies.” Translation &
Interpreting 5 (2): 1–13.
Macklovitch, Elliott
2006 “TransType2:
The Last Word.” In Proceedings of the Fifth International Conference
on Language Resources and Evaluation (LREC 2006), edited by Nicoletta Calzolari, Khalid Choukri, Aldo Gangemi, Bente Maegaard, Joseph Mariani, Jan Odijk, and Daniel Tapias, 167–172. European
Language Resources Association.
Green, Spence, Jeffrey Heer, and Christopher
D. Manning
2013 “The
Efficacy of Human Post-Editing for Language Translation.” In CHI
2013: Changing Perspectives, Conference Proceedings, edited by Wendy
E. Mackay, Stephen Brewster, and Susanne Bødker. New
York: Association for Computing Machinery.
Miniukovich, Aliaksei, Antonella
de Angeli, Simone Sulpizio, and Paola Venuti
2017 “Design
Guidelines for Web Readability.” In DIS ’17: Proceedings of the 2017
Conference on Designing Interactive Systems, edited by Oli Mival, Michael Smyth, and Peter Dalsgaard, 285–296. New
York: Association for Computing Machinery.
Moorkens, Joss, and Sharon O’Brien
2017 “Assessing
User Interface Needs of Post-Editors of Machine
Translation.” In Human Issues in Translation
Technology, edited by Dorothy Kenny, 109–130. Abingdon: Routledge.
Müller, Mathias, Annette Rios, Elena Voita, and Rico Sennrich
2018 “A
Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine
Translation.” In WMT 2018: Third Conference on Machine Translation:
Proceedings of the Conference, edited by Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck et al., 61–72. Stroudsburg: Association
for Computational Linguistics.
Norman, Donald
1988 The
Psychology of Everyday Things. New York: Basic
Books.
O’Brien, Sharon
2009 “Eye
Tracking in Translation Process Research: Methodological Challenges and
Solutions.” In Methodology, Technology and Innovation in Translation
Process Research: A Tribute to Arnt Lykke Jakobsen, edited by Inger
M. Mees, Fabio Alves, and Susanne Göpferich, 251–266. Frederiksberg: Samfundslitteratur.
O’Brien, Sharon, Maureen Ehrensberger-Dow, Megan Connolly, and Marcel Hasler
2017 “Irritating
CAT Tool Features That Matter to
Translators.” HERMES 56: 145–162.
Rello, Luz, Martin Pielot, and Mari-Carmen Marcos
2016 “Make
It Big!: The Effect of Font Size and Line Spacing on Online
Readability.” In Proceedings: The 32th Annual CHI Conference on Human
Factors in Computing Science, edited by Jofish Kaye, Allison Druin, Cliff Lampe, Dan Morris, Juan
Pablo Hourcade, 3637–3648. New
York: Association for Computing Machinery.
Roberts, Teresa
L.
1980 Evaluation of Computer Text
Editors. PhD diss. Stanford
University.
Roberts, Teresa
L., and Thomas
P. Moran
1982 “Evaluation
of Computer Text Editors.” In CHI ’82: Proceedings of the 1982
Conference on Human Factors in Computing Systems, edited by Jean
A. Nichols and Michael
L. Schneider, 136–141. New
York: Association for Computing Machinery.
Ruiz, Carmen, Natalia Paredes, Pedro Macizo, and Maria
Teresa Bajo
2008 “Activation
of Lexical and Syntactic Target Language Properties in Translation.” Acta
Psychologica 128 (3): 490–500.
Saffer, Dan
2005 The
Role of Metaphor in Interaction Design. Master’s thesis. Carnegie
Mellon University.
Schneider, Dominik, Marcos Zampieri, and Josef
van Genabith
2018 “Translation
Memories and the Translator: A Report on a User
Survey.” Babel 64 (5/6): 734–762.
Sennrich, Rico
2017 “How
Grammatical Is Character-Level Neural Machine Translation? Assessing MT Quality with Contrastive Translation
Pairs.” In 15th Conference of the European Chapter of the Association
for Computational Linguistics: Proceedings of the
Conference, vol. 2, edited
by Mirella Lapata, Phil Blunsom, and Alexander Koller, 376–382. Stroudsburg: Association
for Computational Linguistics.
Shih, Claire
Yi-yi
2006 “Revision from Translators’
Point of View: An Interview
Study.” Target 18 (2): 295–312.
Shneiderman, Ben
1983 “Direct
Manipulation: A Step Beyond Programming
Languages.” Computer 8 (16): 57–69.
Yu, Chen-Hsiang, and Robert
C. Miller
2010 “Enhancing
Web Page Readability for Non-Native Readers.” In CHI ’10: Proceedings
of the SIGCHI Conference on Human Factors in Computing Systems, edited by Elizabeth Mynatt, Geraldine Fitzpatrick, Scott Hudson, Keith Edwards, and Tom Rodden, 2523–2532. New
York: Association for Computing Machinery.
Zaretskaya, Anna
2015 D2.1:
User Requirement Analysis. Technical Report: EXPERT: EXPloiting Empirical appRoaches to Translation,
the European Union’s Seventh Framework Programme (FP7). http://expert-itn.eu/sites/default/files/outputs/expert_d2.1_20150210.pdf