Measuring translation difficulty: An empirical study

Sanjun Sun and Gregory M. Shreve

The purpose of this study was to find a method to measure difficulty in a translation task. Readability formulas have been suggested to be a useful tool and yet this needs to be empirically tested. In this study, NASA Task Load Index, a multidimensional scale for measuring mental workload, was used to assess the level of translation difficulty for the translator. It was found that a text’s readability only partially accounts for its translation difficulty level. Translation quality score was found to be an unreliable indicator of translation difficulty level, while time-on-task was significantly, but weakly, related to translation difficulty level. A formula was developed to predict a text’s translation difficulty level for a translator by using the translator’s pre-translation rating.

Table of contents

Knowing the difficulty level of a translation assignment is important in translation pedagogy, accreditation and research, as well as for the language industry. For instance, in process-oriented translation research, researchers have no standards to refer to when they choose test passages, and the texts used are diverse in terms of text type, length and, possibly, difficulty (see Krings 2001, 74). This makes it hard for one to evaluate the comparability of experimental results between these studies. For example, the use of different translation strategies in terms of type and frequency might vary depending on the translation difficulty level of the texts. Dragsted (2004) found in her empirical study that professional translators would adopt a more novice-like behavior during the translation of a difficult text than during the translation of an easy text. Thus, translation difficulty is an important variable in translation process research.

Full-text access is restricted to subscribers. Log in to obtain additional credentials. For subscription information see Subscription & Price. Direct PDF access to this article can be purchased through our e-platform.


Anagnostou, Nikolaos K
., and George R.S. Weir 2007“From Corpus-Based Collocation Frequencies to Readability Measure.” In Texts, Textbooks and Readability , ed. by George R. S. Weir, and Toshiaki Ozasa, 34–48. Glasgow: University of Strathclyde Publishing.Google Scholar
Baker, Mona
(1992) 2011  In Other Words: A Coursebook on Translation . London: Routledge.   DOI logoGoogle Scholar
Bastable, Susan Bacorn
2008 Nurse as Educator: Principles of Teaching and Learning for Nursing Practice . 3rd ed. London: Jones and Bartlett.Google Scholar
Campbell, Stuart
and Sandra Hale 2003“Translation and Interpreting Assessment in the Context of Educational Measurement.” In Translation Today: Trends and Perspectives , ed. by Gunilla M. Anderman, and Margaret Rogers, 205–224. Clevedon: Multilingual Matters. DOI logoGoogle Scholar
“What Makes a Text Difficult to Translate?” Available at: http://​www​.latrobe​.edu​.au​/alaa​/proceed​/camphale​.html (accessed Nov 1 2009).
Carrell, Patricia L
., and Ulla Connor 1991“Reading and Writing Descriptive and Persuasive Texts.” The Modern Language Journal 75 (3): 314–324. DOI logoGoogle Scholar
Chall, Jeanne Sternlicht, and Edgar Dale
1995 Readability Revisited: The New Dale-Chall Readability Formula .Cambridge, MA: Brookline Books.Google Scholar
Chen, Peter Y
., and Autumn D. Krauss 2004a“Internal Reliability.” In The Sage Encyclopedia of Social Science Research Methods , ed. by Michael S. Lewis-Beck, Alan Bryman, and Tim Futing Liao, 501–502. ThousandOaks, CA: Sage.Google Scholar
“Reliability.” In The Sage Encyclopedia of Social Science Research Methods, ed. by Michael S. Lewis-Beck, Alan Bryman, and Tim Futing Liao, 952–956. ThousandOaks, CA: Sage.   DOI logoGoogle Scholar
Cohen, Jacob
1988 Statistical Power Analysis for the Behavioral Sciences . 2nd ed.Hillsdale, NJ: L. Erlbaum Associates.Google Scholar
Cronbach, Lee J
1951“Coefficient Alpha and the Internal Structure of Tests.” Psychometrika 16 (3): 297–334. DOI logoGoogle Scholar
Dahl, Östen
2004 The Growth and Maintenance of Linguistic Complexity . Amsterdam: John Benjamins.   DOI logoGoogle Scholar
Davidson, Janet E
., and Robert J. Sternberg eds 2003 The Psychology of Problem Solving . Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Dragsted, B.
2004 Segmentation in Translation and Translation Memory Systems: An Empirical Investigation of Cognitive Segmentation and Effects of Integrating a Tm-System into the Translation Process . PhD diss. Copenhagen Business School.Google Scholar
DuBay, William H
2004“The Principles of Readability.” Available at: http://​www​.impact​-information​.com​/impactinfo​/readability02​.pdf (accessed November 1 2009).
Englund Dimitrova, Birgitta
2005 Expertise and Explicitation in the Translation Process . Amsterdam: John Benjamins.   DOI logoGoogle Scholar
Feldt, Leonard S
1961“The Use of Extreme Groups to Test for the Presence of a Relationship.” Psychometrika 26 (3): 307–316. DOI logoGoogle Scholar
Gerloff, Pamela
1988 From French to English: A Look at the Translation Process in Students, Bilinguals, and Professional Translators . PhD. diss. Harvard University.Google Scholar
Green, Samuel B
1991“How Many Subjects Does It Take to Do a Regression Analysis.” Multivariate Behavioral Research 26 (3): 499–510. DOI logoGoogle Scholar
Hale, Sandra
and Stuart Campbell 2002“The Interaction between Text Difficulty and Translation Accuracy.” Babel 48 (1): 14–33. DOI logoGoogle Scholar
Hart, Sandra G
., and Lowell E. Staveland 1988“Development of Nasa-Tlx (Task Load Index): Results of Empirical and Theoretical Research.” In Human Mental Workload , ed. by Peter A. Hancock, and Najmedin Meshkati, 139–183. Amsterdam: North-Holland. DOI logoGoogle Scholar
Higgins, Jennifer, Michael Russell
and Thomas Hoffmann 2005.”Examining the Effect of Computer-Based Passage Presentation on Reading Test Performance.” The Journal of Technology, Learning, and Assessment 4. http://​www​.jtla​.org.
House, Juliane
2009“Quality.” In Routledge Encyclopedia of Translation Studies , ed. by Mona Baker, and Gabriela Saldanha, 222–225. London: Routledge.Google Scholar
Hoyt, Linda, Margaret E. Mooney, and Brenda Parkes
, eds 2003 Exploring Informational Texts: From Theory to Practice .Portsmouth, NH: Heinemann.Google Scholar
Jääskeläinen, Riitta
1990 Features of Successful Translation Processes: A Think-aloud Protocol Study . Unpublished licentiate thesis, University of Joensuu, Savonlinna School of Translation Studies.Google Scholar
“Hard Work Will Bear Beautiful Fruit. A Comparison of Two Think-Aloud Protocol Studies.” Meta 41 (1): 60–74. DOI logoGoogle Scholar
Tapping the Process: An Explorative Study of the Cognitive and Affective Factors Involved in Translating . University of Joensuu Publications in the Humanities 22. Joensuu: University of Joensuu.Google Scholar
Jensen, Kristian T.H.
2009“Indicators of Text Complexity.” Copenhagen Studies in Language 37: 61–80.Google Scholar
Jex, Henry. R.
1988“Measuring Mental Workload: Problems, Progress, and Promises.” In Human Mental Workload, Advances in Psychology 52 , ed. by Peter. A. Hancock, and Najmedin Meshkati, 5–38. Amsterdam: North-Holland.   DOI logoGoogle Scholar
Kelley, Truman L
1939“The Selection of Upper and Lower Groups for the Validation of Test Items.” Journal of Educational Psychology 30: 17–24. DOI logoGoogle Scholar
Kiraly, Donald Charles
1995 Pathways to Translation: From Process to Pedagogy .Kent, OH: Kent State University Press.Google Scholar
Krings, Hans P
2001 Repairing Texts: Empirical Investigations of Machine Translation Post-Editing Processes . Translated by Geoffrey Koby, Gregory Shreve, Katjz Mischerikow, and Sarah Litzer. Kent, OH: Kent State University Press.Google Scholar
Liu, Minhua, and Yu-Hsien Chiu
2009“Assessing Source Material Difficulty for Consecutive Interpreting: Quantifiable Measures and Holistic Judgment.” Interpreting 11 (2): 244–266. DOI logoGoogle Scholar
Moray, Neville
1979 Mental Workload: Its Theory and Measurement . New York: Plenum Press. DOI logoGoogle Scholar
Moroney, William F
., David W. Biers, F. Thomas Eggemeier, and Jennifer A. Mitchell 1992“A Comparison of Two Scoring Procedures with the Nasa Task Load Index in a Simulated Flight Task.” In Proceedings of the IEEE 1992 National Aerospace and Electronics Conference - Naecon 1992 , Dayton, 18–22 May 1992, 734–740.   DOI logo
Nord, Christiane
2005 Text Analysis in Translation: Theory, Methodology, and Didactic Application of a Model for Translation-Oriented Text Analysis . 2nd ed. Amsterdam: Rodopi.Google Scholar
Nunnally, Jum C
1978 Psychometric Theory . 2nd ed. New York: McGraw-Hill.Google Scholar
O’Donnell, Robert D
., and F. Thomas Eggemeier 1986“Workload Assessment Methodology.” In Handbook of Perception and Human Performance. Vol. 2: Cognitive Processes and Performance , ed. by Kenneth R. Boff, Lloyd Kaufman, and James P. Thomas, 42/41–42/49. New York: Wiley.Google Scholar
Paas, Fred G
1992“Training Strategies for Attaining Transfer of Problem-Solving Skill in Statistics: A Cognitive-Load Approach.” Journal of Educational Psychology 84 (4): 429–434. DOI logoGoogle Scholar
2005“Investigating Translation Competence: Conceptual and Methodological Issues.” Meta 50 (2): 609–619. DOI logoGoogle Scholar
Pallant, Julie
2011 Spss Survival Manual . 4th ed. Crows Nest (Australia): Allen & Unwin.Google Scholar
Pedhazur, Elazar J
1997 Multiple Regression in Behavioral Research: Explanation and Prediction . 3rd ed. United States: Wadsworth/Thomson Learning.Google Scholar
Preacher, Kristopher J
., Derek D. Rucker, Robert C. MacCallum, and W. Alan Nicewander 2005“Use of the Extreme Groups Approach: A Critical Reexamination and New Recommendations.” Psychological Methods 10 (2): 178–192. DOI logoGoogle Scholar
Pretz, Jean E
., Adam J. Naples, and Robert J. Sternberg 2003“Recognizing, Defining, and Representing Problems.” In The Psychology of Problem Solving , ed. by Janet E. Davidson, and Robert J. Sternberg, 3–30. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Pym, Anthony
2010 Exploring Translation Theories . London: Routledge.Google Scholar
“Redefining Translation Competence in an Electronic Age: In Defence of a Minimalist Approach.” Meta 48 (4): 481–497. DOI logoGoogle Scholar
Rayner, Keith, and Alexander Pollatsek
1989 The Psychology of Reading .Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
Shreve, Gregory M
2002“Knowing Translation: Cognitive and Experiential Aspects of Translation Expertise from the Perspective of Expertise Studies.” In Translation Studies: Perspectives on an Emerging Discipline , ed. by Alessandra Riccardi, 150–173. Cambridge: Cambridge University Press.Google Scholar
Sweller, John, Paul L. Ayres
and Slava Kalyuga 2011 Cognitive Load Theory . New York: Springer. DOI logoGoogle Scholar
Taylor, Wilson L
1953“‘Cloze Procedure’: A New Tool for Measuring Readability.” Journalism Quarterly 30: 415–433. DOI logoGoogle Scholar
Tirkkonen-Condit, Sonja
1987“Think-Aloud Protocols in the Study of the Translation Process.” In CDEF 86: Papers from the Conference of Departments of English in Finland , ed. by Heikki Nyyssönen, Riitta Kataja, and Vesa Komulainen, 39–49. Oulu: University of Oulu.Google Scholar
Tsang, Pamela S
2006“Mental Workload.” In International Encyclopedia of Ergonomics and Human Factors , ed. by Waldemar Karwowski, 809–813. BocaRaton, FL: CRC/Taylor & Francis.Google Scholar
Vidulich, Michael A
1988“The Cognitive Psychology of Subjective Mental Workload.” In Human Mental Workload , ed. by Peter A. Hancock, and Najmedin Meshkati, 219–229. Amsterdam: North-Holland. DOI logoGoogle Scholar
Wilson, Glenn F
.. and F. Thomas Eggemeier 2006“Mental Workload Measurement.” In International Encyclopedia of Ergonomics and Human Factors , ed. by Waldemar Karwowski, 814–817. BocaRaton, FL: CRC/Taylor & Francis.Google Scholar
Wilss, Wolfram
1982 The Science of Translation: Problems and Methods . Tübinger Beitraäge zur Linguistik. Tuübingen: G. Narr.Google Scholar
Zamanian, Mostafa, and Pooneh Heydari
2012“Readability of Texts: State of the Art.” Theory and Practice in Language Studies 2 (1): 43–53. DOI logoGoogle Scholar
Zimmerman, Lynn W
2010 ESL, EFL, and Bilingual Education: Exploring Historical, Sociocultural, Linguistic, and Instructional Foundations .Charlotte, NC: Information Age Pub.Google Scholar
Zipf, George Kingsley
1935 The Psycho-Biology of Language: An Introduction to Dynamic Philology . Boston: Houghton Mifflin.Google Scholar