Article published in:
Target
Vol. 34:2 (2022) ► pp. 309–342

The impact of text presentation on translator performance

| Zurich University of Applied Sciences | University of Zurich
| Lilt, Inc.
| Lilt, Inc.
| Lilt, Inc.
| University of Zurich | University of Edinburgh
| Lilt, Inc.
Widely used computer-aided translation (CAT) tools divide documents into segments, such as sentences, and arrange them side-by-side in a spreadsheet-like view. We present the first controlled evaluation of these design choices on translator performance, measuring speed and accuracy in three experimental text-processing tasks. We find significant evidence that sentence-by-sentence presentation enables faster text reproduction and within-sentence error identification compared to unsegmented text, and that a top-and-bottom arrangement of source and target sentences enables faster text reproduction compared to a side-by-side arrangement. For revision, on the other hand, we find that presenting unsegmented text results in the highest accuracy and time efficiency. Our findings have direct implications for best practices in designing CAT tools.
Keywords: computer-aided translation, human-computer interaction, user interface, text presentation, productivity
Article outline
  • 1.Introduction
  • 2.Background
    • 2.1Text and document visualisation
    • 2.2Text and document visualisation in CAT tools
    • 2.3Understanding translator performance
  • 3.Preliminary feedback from potential users
    • 3.1Method
    • 3.2Ideation
    • 3.3Concept testing
      • 3.3.1Praise and opportunities
      • 3.3.2Criticism and limitations
  • 4.Experimental methods
    • 4.1Tasks
      • 4.1.1Text reproduction (copy)
      • 4.1.2Error identification (scan)
      • 4.1.3Revision (revise)
    • 4.2Materials
      • 4.2.1Texts
      • 4.2.2User Interfaces (UIs)
    • 4.3Participants
    • 4.4Procedure
    • 4.5Data analysis
  • 5.Experimental results
    • 5.1Text reproduction (copy)
      • 5.1.1Speed
      • 5.1.2Accuracy
    • 5.2Error identification (scan)
      • 5.2.1Speed
      • 5.2.2Accuracy
    • 5.3Revision (revise)
      • 5.3.1Speed
      • 5.3.2Accuracy
    • 5.4UI preference
  • 6.Discussion and design implications
    • 6.1Segmentation
    • 6.2Orientation
    • 6.3Limitations
    • 6.4Future work
  • 7.Conclusions
  • Acknowledgements
  • Notes
  • References
Published online: 23 November 2021
https://doi.org/10.1075/target.20006.lau
References
Arif, Ahmed Sabbir, and Wolfgang Stuerzlinger
2009 “Analysis of Text Entry Performance Metrics.” In 2009 IEEE Toronto International Conference Science and Technology for Humanity (TIC–STH), 26–27 September 2009, Toronto Canada, 100–105. Piscataway, NJ: IEEE. CrossrefGoogle Scholar
Baayen, R. Harald, and Petar Milin
2010 “Analyzing Reaction Times.” International Journal of Psychological Research 3 (2): 12–28. CrossrefGoogle Scholar
Bates, Douglas, Martin Mächler, Ben Bolker, and Steve Walker
2015 “Fitting Linear Mixed-Effects Models Using lme4.” Journal of Statistical Software 67 (1): 1–48. CrossrefGoogle Scholar
Bojar, Ondřej, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck et al.
2017 “Findings of the 2017 Conference on Machine Translation (WMT17).” In WMT 2017: Second Conference on Machine Translation: Proceedings, edited by Ondřej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck et al., 169–214. Stroudsburg: Association for Computational Linguistics. CrossrefGoogle Scholar
Bojar, Ondřej, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn et al.
2018 “Findings of the 2018 Conference on Machine Translation (WMT18).” In WMT 2018: Third Conference on Machine Translation: Proceedings of the Conference, edited by Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck et al., 272–307. Stroudsburg: Association for Computational Linguistics. CrossrefGoogle Scholar
Bowker, Lynne
2005 “Productivity vs Quality? A Pilot Study on the Impact of Translation Memory Systems.” Localisation Focus 4 (1): 13–20.Google Scholar
Campbell, Stuart
1999 “A Cognitive Approach to Source Text Difficulty in Translation.” Target 11 (1): 33–63. CrossrefGoogle Scholar
Carl, Michael
2010 “A Computational Framework for a Cognitive Model of Human Translation Processes.” In Proceedings of Translating and the Computer 32. London: Aslib.Google Scholar
2012 “Translog-II: A Program for Recording User Activity Data for Empirical Reading and Writing Research.” In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC ’12), edited by Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Asuncion Moreno et al., 4108–4112. Istanbul: European Language Resources Association.Google Scholar
Castilho, Sheila, Joss Moorkens, Federico Gaspari, Iacer Calixto, John Tinsley, and Andy Way
2017 “Is Neural Machine Translation the New State of the Art?” The Prague Bulletin of Mathematical Linguistics 1081: 109–120. CrossrefGoogle Scholar
Castilho, Sheila, Joss Moorkens, Federico Gaspari, Rico Sennrich, Vilelmini Sosoni, Panayota Georgakopoulou, Pintu Lohar et al.
2017 “A Comparative Quality Evaluation of PBSMT and NMT Using Professional Translators.” In Proceedings of Machine Translation Summit XVI, vol. 11, edited by Sadao Kurohashi and Pascale Fung, 116–131. Nagoya: Asia-Pacific Association for Machine Translation.Google Scholar
Castilho, Sheila, Joss Moorkens, Federico Gaspari, Rico Sennrich, Andy Way, and Panayota Georgakopoulou
2018 “Evaluating MT for Massive Open Online Courses.” In Human Evaluation of Statistical and Neural Machine Translation, edited by Andy Way and Mikel L. Forcada, special issue of Machine Translation 32 (3): 255–278. CrossrefGoogle Scholar
Coppers, Sven, Jan van den Bergh, Kris Luyten, Karin Coninx, Iulianna van der Lek–Ciudin, Tom Vanallemeersch, and Vincent Vandeghinste
2018 “Intellingo: An Intelligible Translation Environment.” In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, edited by Regan Mandryk, Mark Hancock, Mark Perry, and Anna Cox. Montreal: Association for Computing Machinery. CrossrefGoogle Scholar
do Carmo, Félix, and Joss Moorkens
2020 “Differentiating Editing, Post-Editing and Revision.” In Translation Revision and Post-Editing: Industry Practices and Cognitive Processes, edited by Maarit Koponen, Brian Mossop, Isabelle S. Robert, and Giovanna Scocchera. Abingdon: Routledge. CrossrefGoogle Scholar
Dragsted, Barbara
2006 “Computer-Aided Translation as a Distributed Cognitive Task.” In Distributed Cognition, edited by Stevan Harnad and Itiel E. Dror, special issue of Pragmatics & Cognition 14 (2): 443–464. CrossrefGoogle Scholar
2010 “Coordination of Reading and Writing Processes in Translation: An Eye on Uncharted Territory.” In Translation and Cognition, edited by Gregory M. Shreve and Erik Angelone, 41–62. Amsterdam: John Benjamins. CrossrefGoogle Scholar
Ehrensberger-Dow, Maureen, Andrea Hunziker Heeb, Gary Massey, Ursula Meidert, Silke Neumann, and Heidrun Becker
2016 “An International Survey of the Ergonomics of Professional Translation.” In Ergonomic Approaches to Professional Practices and Translator Training, edited by Élisabeth Lavault-Olléon, special issue of Journal of the Institute of Languages and Cultures of Europe, America, Africa, Asia and Australia 271. CrossrefGoogle Scholar
Engelbart, Douglas C., and William K. English
1968 “A Research Center for Augmenting Human Intellect.” In AFIPS 1968 Fall Joint Conference, Part 1, 395–410. Washington: Thompson Book Company. CrossrefGoogle Scholar
Ericsson, K. Anders, and Herbert A. Simon
1984 Protocol Analysis: Verbal Reports as Data. Cambridge, MA: MIT Press.Google Scholar
Federico, Marcello, Alessandro Cattelan, and Marco Trombetti
2012 “Measuring User Productivity in Machine Translation Enhanced Computer Assisted Translation.” In Proceedings of the 10th Conference of the Association for Machine Translation in the Americas. San Diego: Association for Machine Translation in the Americas.Google Scholar
Francis, Wendy S., and Silvia P. Sáenz
2007 “Repetition Priming Endurance in Picture Naming and Translation: Contributions of Component Processes.” Memory & Cognition 35 (3): 481–493. CrossrefGoogle Scholar
Gelman, Andrew, and Jennifer Hill
2007 Data Analysis Using Regression and Multilevel/Hierarchical Models. New York: Cambridge University Press.Google Scholar
Green, Spence, Jason Chuang, Jeffrey Heer, and Christopher D. Manning
2014 “Predictive Translation Memory: A Mixed-Initiative System for Human Language Translation.” In UIST ’14: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, edited by Hrvoje Benko, Mira Dontcheva, and Daniel Wigdor, 177–187. New York: Association for Computing Machinery. CrossrefGoogle Scholar
Hassan, Hany, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang et al.
2018 “Achieving Human Parity on Automatic Chinese to English News Translation.” arXiv Preprint 1803.05567. CrossrefGoogle Scholar
Hornbæk, Kasper, and Erik Frøkjær
2001 “Reading of Electronic Documents: The Usability of Linear, Fisheye, and Overview+detail Interfaces.” In CHI ’01: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, edited by Julie Jacko and Andrew Sears, 293–300. New York: Association for Computing Machinery. CrossrefGoogle Scholar
House, Juliane
2013 “Quality in Translation Studies.” In The Routledge Handbook of Translation, edited by Carmen Millán and Francesca Bartrina, 534–547. Abingdon: Routledge.Google Scholar
Jakobsen, Arnt Lykke
2003 “Effects of Think Aloud on Translation Speed, Revision, and Segmentation.” In Triangulating Translation: Perspectives in Process Oriented Research, edited by Fabio Alves, 69–95. Amsterdam: John Benjamins. CrossrefGoogle Scholar
Junczys-Dowmunt, Marcin
2019 “Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation.” In Proceedings of the Fourth Conference on Machine Translation (WMT19), vol. 21, edited by Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow et al., 225–233. Stroudsburg: Association for Computational Linguistics. CrossrefGoogle Scholar
Karimova, Sariya, Patrick Simianer, and Stefan Riezler
2018 “A User-Study on Online Adaptation of Neural Machine Translation to Human Post-Edits.” Machine Translation 32 (4): 309–324. CrossrefGoogle Scholar
Kay, Martin
1980 “The Proper Place of Men and Machines in Language Translation.” Research Report CSL-80-11. Palo Alto, CA: Xerox Palo Alto Research Center.Google Scholar
Krings, Hans P.
1994 Texte Reparieren: Empirische Untersuchungen Zum Prozeß Der Nachredaktion von Maschinenübersetzungen. Habilitation thesis. University of Hildesheim.
2001 Repairing Texts: Empirical Investigations of Machine Translation Post-Editing Processes. Translated by Geoffrey S. Koby, Gregory M. Shreve, Katja Mischerikow, and Sarah Litzer. Kent: Kent State University Press.Google Scholar
Langlais, Philippe, Guy Lapalme, and Sébastien Sauvé
2001 “User Interface Aspects of a Translation Typing System.” In Advances in Artificial Intelligence: 14th Biennial Conference of the Canadian Society for Computational Studies of Intelligence, AI 2001, edited by Eleni Stroulia and Stan Matwin, 246–256. Berlin: Springer. CrossrefGoogle Scholar
Läubli, Samuel, Rico Sennrich, and Martin Volk
2018 “Has Machine Translation Achieved Human Parity? A Case for Document-Level Evaluation.” In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, edited by Ellen Riloff, Davide Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, 4791–4796. Stroudsburg: Association for Computational Linguistics. CrossrefGoogle Scholar
LeBlanc, Matthieu
2013 “Translators on Translation Memory (TM): Results of an Ethnographic Study in Three Translation Services and Agencies.” Translation & Interpreting 5 (2): 1–13. CrossrefGoogle Scholar
Macklovitch, Elliott
2006 “TransType2: The Last Word.” In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC 2006), edited by Nicoletta Calzolari, Khalid Choukri, Aldo Gangemi, Bente Maegaard, Joseph Mariani, Jan Odijk, and Daniel Tapias, 167–172. European Language Resources Association.Google Scholar
Green, Spence, Jeffrey Heer, and Christopher D. Manning
2013 “The Efficacy of Human Post-Editing for Language Translation.” In CHI 2013: Changing Perspectives, Conference Proceedings, edited by Wendy E. Mackay, Stephen Brewster, and Susanne Bødker. New York: Association for Computing Machinery. CrossrefGoogle Scholar
Miniukovich, Aliaksei, Antonella de Angeli, Simone Sulpizio, and Paola Venuti
2017 “Design Guidelines for Web Readability.” In DIS ’17: Proceedings of the 2017 Conference on Designing Interactive Systems, edited by Oli Mival, Michael Smyth, and Peter Dalsgaard, 285–296. New York: Association for Computing Machinery. CrossrefGoogle Scholar
Moorkens, Joss, and Sharon O’Brien
2017 “Assessing User Interface Needs of Post-Editors of Machine Translation.” In Human Issues in Translation Technology, edited by Dorothy Kenny, 109–130. Abingdon: Routledge.Google Scholar
Müller, Mathias, Annette Rios, Elena Voita, and Rico Sennrich
2018 “A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation.” In WMT 2018: Third Conference on Machine Translation: Proceedings of the Conference, edited by Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck et al., 61–72. Stroudsburg: Association for Computational Linguistics. CrossrefGoogle Scholar
Norman, Donald
1988 The Psychology of Everyday Things. New York: Basic Books.Google Scholar
O’Brien, Sharon
2009 “Eye Tracking in Translation Process Research: Methodological Challenges and Solutions.” In Methodology, Technology and Innovation in Translation Process Research: A Tribute to Arnt Lykke Jakobsen, edited by Inger M. Mees, Fabio Alves, and Susanne Göpferich, 251–266. Frederiksberg: Samfundslitteratur.Google Scholar
O’Brien, Sharon, Maureen Ehrensberger-Dow, Megan Connolly, and Marcel Hasler
2017 “Irritating CAT Tool Features That Matter to Translators.” HERMES 561: 145–162.Google Scholar
Rello, Luz, Martin Pielot, and Mari-Carmen Marcos
2016 “Make It Big!: The Effect of Font Size and Line Spacing on Online Readability.” In Proceedings: The 32th Annual CHI Conference on Human Factors in Computing Science, edited by Jofish Kaye, Allison Druin, Cliff Lampe, Dan Morris, Juan Pablo Hourcade, 3637–3648. New York: Association for Computing Machinery.Google Scholar
Roberts, Teresa L.
1980 Evaluation of Computer Text Editors. PhD diss. Stanford University.
Roberts, Teresa L., and Thomas P. Moran
1982 “Evaluation of Computer Text Editors.” In CHI ’82: Proceedings of the 1982 Conference on Human Factors in Computing Systems, edited by Jean A. Nichols and Michael L. Schneider, 136–141. New York: Association for Computing Machinery. CrossrefGoogle Scholar
Ruiz, Carmen, Natalia Paredes, Pedro Macizo, and Maria Teresa Bajo
2008 “Activation of Lexical and Syntactic Target Language Properties in Translation.” Acta Psychologica 128 (3): 490–500. CrossrefGoogle Scholar
Saffer, Dan
2005 The Role of Metaphor in Interaction Design. Master’s thesis. Carnegie Mellon University.
Schneider, Dominik, Marcos Zampieri, and Josef van Genabith
2018 “Translation Memories and the Translator: A Report on a User Survey.” Babel 64 (5/6): 734–762. CrossrefGoogle Scholar
Sennrich, Rico
2017 “How Grammatical Is Character-Level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs.” In 15th Conference of the European Chapter of the Association for Computational Linguistics: Proceedings of the Conference, vol. 21, edited by Mirella Lapata, Phil Blunsom, and Alexander Koller, 376–382. Stroudsburg: Association for Computational Linguistics. CrossrefGoogle Scholar
Shih, Claire Yi-yi
2006 “Revision from Translators’ Point of View: An Interview Study.” Target 18 (2): 295–312. CrossrefGoogle Scholar
Shneiderman, Ben
1983 “Direct Manipulation: A Step Beyond Programming Languages.” Computer 8 (16): 57–69. CrossrefGoogle Scholar
Yu, Chen-Hsiang, and Robert C. Miller
2010 “Enhancing Web Page Readability for Non-Native Readers.” In CHI ’10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, edited by Elizabeth Mynatt, Geraldine Fitzpatrick, Scott Hudson, Keith Edwards, and Tom Rodden, 2523–2532. New York: Association for Computing Machinery. CrossrefGoogle Scholar
Zaretskaya, Anna
2015 D2.1: User Requirement Analysis. Technical Report: EXPERT: EXPloiting Empirical appRoaches to Translation, the European Union’s Seventh Framework Programme (FP7). http://​expert​-itn​.eu​/sites​/default​/files​/outputs​/expert​_d2​.1​_20150210​.pdf