Measuring the difficulty of text translation: The combination of text-focused and translator-oriented approaches

Yanmei Liu, Binghan Zheng and Hao Zhou


This paper explores the impact of text complexity on translators’ subjective perception of translation difficulty and on their cognitive load. Twenty-six MA translation students from a UK university were asked to translate three English texts with different complexity into Chinese. Their eye movements were recorded by an eye-tracker, and their cognitive load was self-assessed with a Likert scale before translation and NASA-TLX scales after translation. The results show that: (i) the intrinsic complexity measured by readability, word frequency and non-literalness was in line with the results received from informants’ subjective assessment of translation difficulty; (ii) moderate and positive correlations existed between most items in the self-assessments and the indicator (fixation and saccade durations) obtained by the eye-tracking measurements; and (iii) the informants’ cognitive load as indicated by fixation and saccade durations (but not for pupil size) increased significantly in two of the three texts along with the increase in source text complexity.

Publication history
Table of contents

The significance of measuring the difficulty of a source text for translation pedagogy and research has received some attention in the past two decades (e.g., Hale and Campbell 2002; Jensen 2009; Mishra, Bhattacharyya, and Carl 2013; Sun and Shreve 2014). To investigate the degree of translation difficulty caused by the variable text complexity, researchers have based their examinations either on readability alone (Pavlović and Jensen 2009), or on a combination of readability and other indicators, such as word frequency, sentence structure and non-literalness (Sharmin, Spakov, Räihä, and Jakobsen 2008; Jensen 2009). Measurement has generally been centred around the level of text complexity – for instance, character length, syllable length and sentence length – while ignoring other important factors, such as conceptual complexity, text organisation, or reader’s background knowledge (Liu and Chiu 2011, 149). Nevertheless, the textual factors can account only partially for the text’s level of translation difficulty (Sun and Shreve 2014, 98), since the construct of translation difficulty originates from the interaction between task and its translator. Therefore, translation difficulty should be measured in both texts and the profiles of translators working with the texts.

Full-text access is restricted to subscribers. Log in to obtain additional credentials. For subscription information see Subscription & Price. Direct PDF access to this article can be purchased through our e-platform.


Akbari, Alireza, and Winibert Segers
2017 “Translation Difficulty: How to Measure and What to Measure.” Lebende Sprachen 62 (1): 3–29. CrossrefGoogle Scholar
Ashby, Jane, Keith Rayner, and Charles Clifton
2005 “Eye Movements of Highly Skilled and Average Readers: Differential Effects of Frequency and Predictability.” The Quarterly Journal of Experimental Psychology Section A 58 (6): 1065–1086. CrossrefGoogle Scholar
Cain, Brad
2007A Review of the Mental Workload Literature. Technical Report, Defence Research and Development Canada Toronto.Google Scholar
Campbell, Stuart
1999 “A Cognitive Approach to Source Text Difficulty in Translation.” Target 11 (1): 33–63. CrossrefGoogle Scholar
Carpenter, Patricia A., and Marcel A. Just
1989 “The Role of Working Memory in Language Comprehension.” In Complex Information Processing: The Impact of Herbert A. Simon, edited by David Klahr and Kenneth Kotovsk, 31–68. Hillsdale, NJ: Lawrence Erlbaum Associates Publishers.Google Scholar
Chaffin, Roger, Robin K. Morris, and Rachel E. Seely
2001 “Learning New Word Meanings from Context: A Study of Eye Movements.” Journal of Experimental Psychology: Learning, Memory, and Cognition 27 (1): 225–235.Google Scholar
Ehrlich, Susan F., and Keith Rayner
1981 “Contextual Effects on Word Perception and Eye Movements during Reading.” Journal of Verbal Learning and Verbal Behavior 20 (6): 641–655. CrossrefGoogle Scholar
Evans, James D.
1996Straightforward Statistics for the Behavioral Sciences. Pacific Grove, CA: Brooks/Cole Publishing Co.Google Scholar
Gibbs Jr, Raymond W.
1990 “Comprehending Figurative Referential Descriptions.” Journal of Experimental Psychology: Learning, Memory and Cognition 16 (1): 56–66.Google Scholar
Hale, Sandra, and Stuart Campbell
2002 “The Interaction between Text Difficulty and Translation Accuracy.” Babel 8 (1): 14–33. CrossrefGoogle Scholar
Hart, Sandra G., and Lowell E. Staveland
1988 “Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research.” In Human Mental Workload, edited by Peter A. Hancock and Najmedin Meshkati, 139–183. Amsterdam: North-Holland. CrossrefGoogle Scholar
Hess, Eckhard H., and James M. Polt
1964 “Pupil Size in Relation to Mental Activity during Simple Problem-Solving.” Science 143 (3611): 1190–1192. CrossrefGoogle Scholar
Hvelplund, Kristian Tangsgaard
2011Allocation of Cognitive Resources in Translation: An Eye-tracking and Key-logging Study. PhD diss. Copenhagen Business School.Google Scholar
2014 “Eye Tracking and the Translation Process: Reflections on the Analysis and Interpretation of Eye-tracking Data.” In Minding Translation / Con la traducción en mente, edited by Ricardo Muñoz Martín, 201–224. San Vicente del Raspeig: Publicaciones de la Universidad de Alicante.Google Scholar
Hyönä, Jukka, Jorma Tommola, and Anna-Mari Alaja
1995 “Pupil Dilation as a Measure of Processing Load in Simultaneous Interpretation and Other Language Tasks.” The Quarterly Journal of Experimental Psychology 48 (3): 598–612. CrossrefGoogle Scholar
Inhoff, Albrecht Werner
1984 “Two Stages of Word Processing during Eye Fixations in the Reading of Prose.” Journal of Verbal Learning & Verbal Behavior 23 (5): 612–624. CrossrefGoogle Scholar
Iqbal, Shamsi T., Xianjun Sam Zheng, and Brian P. Bailey
2004 “Task-evoked Pupillary Response to Mental Workload in Human-Computer Interaction.” In CHI’04 Extended Abstracts on Human Factors in Computing Systems, edited by Elizabeth Dykstra-Erickson and Manfred Tscheligi, 1477–1480. Vienna.Google Scholar
Irwin, David E.
2004 “Fixation Location and Fixation Duration as Indices of Cognitive Processing.” In The Interface of Language, Vision, and Action: Eye Movements and Visual World, edited by John Henderson and Fernanda Ferreira, 105–134. New York: Psychology Press.Google Scholar
Jensen, Kristian T. H.
2009 “Indicators of Text Complexity.” In Behind the Mind: Methods, Models and Results in Translation Process Research, edited by Susanne Göpferich, Arnt L. Jakobsen, and Inger M. Mees, 61–80. Copenhagen: Samfundslitteratur.Google Scholar
Johannsen, Gunnar
1979 “Workload and Workload Measurement.” In Mental Workload: Its Theory and Measurement, edited by Neville Moray, 3–11. New York: Springer Science & Business Media. CrossrefGoogle Scholar
Just, Marcel A., and Patricia A. Carpenter
1980 “A Theory of Reading: From Eye Fixations to Comprehension.” Psychological Review 87 (4): 329–354. CrossrefGoogle Scholar
1993 “The Intensity Dimension of Thought: Pupillometric Indices of Sentence Processing.” Canadian Journal of Experimental Psychology 47 (2): 310–339. CrossrefGoogle Scholar
LeBreton, James M., and Jenell L. Senter
2008 “Answers to 20 Questions about Interrater Reliability and Interrater Agreement.” Organizational Research Methods 11 (4): 815–852. CrossrefGoogle Scholar
Liu, Minhua, and Yu-Hsien Chiu
2011 “Assessing Source Material Difficulty for Consecutive Interpreting.” In Interpreting Chinese, Interpreting China, edited by Robin Setton, 135–156. Amsterdam: John Benjamins. CrossrefGoogle Scholar
Mishra, Abhijit, Pushpak Bhattacharyya, and Michael Carl
2013 “Automatically Predicting Sentence Translation Difficulty.” In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 346–351. Sofia.Google Scholar
O’Brien, Sharon
2006 “Eye-tracking and Translation Memory Matches.” Perspectives 14 (3): 185–205.Google Scholar
Paas, Fred G. W. C.
1992 “Training Strategies for Attaining Transfer of Problem-Solving Skill in Statistics: A Cognitive-Load Approach.” Journal of Educational Psychology 84 (4): 429–434. CrossrefGoogle Scholar
Paas, Fred G. W. C., and Jeroen J. G. van Merriënboer
1994a “Instructional Control of Cognitive Load in the Training of Complex Cognitive Tasks.” Educational Psychology Review 6 (4): 351–371. CrossrefGoogle Scholar
1994b “Variability of Worked Examples and Transfer of Geometrical Problem-Solving Skills: A Cognitive-Load Approach.” Journal of Educational Psychology 86 (1): 122–133. CrossrefGoogle Scholar
Pavlović, Nataša, and Kristian Jensen
2009 “Eye Tracking Translation Directionality.” In Translation Research Projects 2, edited by Anthony Pym and Alexander Perekrestenko, 93–109. Tarragona: Intercultural Studies Group.Google Scholar
Pomplun, Marc, and Sindhura Sunkara
2003 “Pupil Dilation as an Indicator of Cognitive Workload in Human-Computer Interaction.” In Proceedings of the 10th International Conference on HCI (Vol. 3), 542–546. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.Google Scholar
Rayner, Keith
1998 “Eye Movements in Reading and Information Processing: 20 Years of Research.” Psychological Bulletin 124 (3): 372–422. CrossrefGoogle Scholar
Rayner, Keith, and Arnold D. Well
1996 “Effects of Contextual Constraint on Eye Movements in Reading: A Further Examination.” Psychonomic Bulletin & Review 3 (4): 504–509. CrossrefGoogle Scholar
Rayner, Keith, and Susan A. Duffy
1986 “Lexical Complexity and Fixation Times in Reading: Effects of Word Frequency, Verb Complexity, and Lexical Ambiguity.” Memory & Cognition 14 (3):191–201. CrossrefGoogle Scholar
Rayner, Keith, and Martin H. Fischer
1996 “Mindless Reading Revisited: Eye Movements during Reading and Scanning Are Different.” Perception & Psychophysics 58 (5): 734–747. CrossrefGoogle Scholar
Rayner, Keith, and Gary E. Raney
1996 “Eye Movement Control in Reading and Visual Search: Effects of Word Frequency.” Psychonomic Bulletin & Review 3 (2): 245–248. CrossrefGoogle Scholar
Rayner, Keith, Sara C. Sereno, and Gary E. Raney
1996 “Eye Movement Control in Reading: A Comparison of Two Types of Models.” Journal of Experimental Psychology: Human Perception and Performance 22 (5): 1188–1200.Google Scholar
Schotter, Elizabeth R., and Keith Rayner
2012 “Eye Movements in Reading: Implications for Reading Subtitles.” In Eye Tracking in Audiovisual Translation, edited by Elisa Perego, 83–104. Roma: Aracne Editrice.Google Scholar
Schultheis, Holger, and Anthony Jameson
2004 “Assessing Cognitive Load in Adaptive Hypermedia Systems: Physiological and Behavioural Methods.” In Adaptive Hypermedia and Adaptive Web-based Systems, edited by Wolfgang Nejdl and Paul De Bra, 225–234. Berlin: Springer. CrossrefGoogle Scholar
Sereno, Sara C., Patrick J. O’Donnell, and Keith Rayner
2006 “Eye Movements and Lexical Ambiguity Resolution: Investigating the Subordinate-Bias Effect.” Journal of Experimental Psychology: Human Perception and Performance 32 (2): 335–350.Google Scholar
Sharmin, Selina, Oleg Spakov, Kari-Jouko Räihä, and Arnt L. Jakobsen
2008 “Where on the Screen Do Translation Students Look While Translating, and for How Long?” In Looking at Eyes: Eye-Tracking Studies of Reading and Translation Processing, edited by Arnt L. Jakobsen, Susanne Göpferich, and Inger M. Mees, 31–51. Copenhagen: Samfundslitteratur.Google Scholar
Staub, Adrian, and Keith Rayner
2007 “Eye Movements and On-Line Comprehension Processes.” In The Oxford Handbook of Psycholinguistics, edited by M. Gareth Gaskell and Gerry Altmann, 327–342. Oxford: Oxford University Press.Google Scholar
Sun, Sanjun
2015 “Measuring Translation Difficulty: Theoretical and Methodological Considerations.” Across Languages and Cultures 16 (1): 29–54. CrossrefGoogle Scholar
Sun, Sanjun, and Gregory M. Shreve
2014 “Measuring Translation Difficulty: An Empirical Study.” Target 26 (1): 98–127. CrossrefGoogle Scholar
Sweller, John
2010 “Element Interactivity and Intrinsic, Extraneous, and Germane Cognitive Load.” Educational Psychology Review 22 (2): 123–138. CrossrefGoogle Scholar
Sweller, John, Jeroen J. G. Van Merrienboer, and Fred G. W. C. Paas
1998 “Cognitive Architecture and Instructional Design.” Educational Psychology Review 10 (3): 251–296. CrossrefGoogle Scholar
Vauras, Marja, Jukka Hyönä, and Pekka Niemi
1992 “Comprehending Coherent and Incoherent Texts: Evidence from Eye Movement Patterns and Recall Performance.” Journal of Research in Reading 15 (1): 39–54. CrossrefGoogle Scholar
Williams, Rihana, and Robin Morris
2004 “Eye Movements, Word Familiarity, and Vocabulary Acquisition.” European Journal of Cognitive Psychology 16 (1–2): 312–339. CrossrefGoogle Scholar
Zola, David
1984 “Redundancy and Word Perception during Reading.” Perception & Psychophysics 36 (3): 277–284. CrossrefGoogle Scholar