Measuring translation difficulty: An empirical study
Sanjun Sun and Gregory M. Shreve
Beijing Foreign Studies University | Kent State University
The purpose of this study was to find a method to measure difficulty in a translation task. Readability formulas have been suggested to be a useful tool and yet this needs to be empirically tested. In this study, NASA Task Load Index, a multidimensional scale for measuring mental workload, was used to assess the level of translation difficulty for the translator. It was found that a text’s readability only partially accounts for its translation difficulty level. Translation quality score was found to be an unreliable indicator of translation difficulty level, while time-on-task was significantly, but weakly, related to translation difficulty level. A formula was developed to predict a text’s translation difficulty level for a translator by using the translator’s pre-translation rating.
Knowing the difficulty level of a translation assignment is important in translation pedagogy, accreditation and research, as well as for the language industry. For instance, in process-oriented translation research, researchers have no standards to refer to when they choose test passages, and the texts used are diverse in terms of text type, length and, possibly, difficulty (see Krings 2001, 74). This makes it hard for one to evaluate the comparability of experimental results between these studies. For example, the use of different translation strategies in terms of type and frequency might vary depending on the translation difficulty level of the texts. Dragsted (2004) found in her empirical study that professional translators would adopt a more novice-like behavior during the translation of a difficult text than during the translation of an easy text. Thus, translation difficulty is an important variable in translation process research.
Anagnostou, Nikolaos K
., and George R.S. Weir2007“From Corpus-Based Collocation Frequencies to Readability Measure.” In Texts, Textbooks and Readability, ed. by George R. S. Weir, and Toshiaki Ozasa, 34–48. Glasgow: University of Strathclyde Publishing.
(1992) 2011 In Other Words: A Coursebook on Translation. London: Routledge.
Bastable, Susan Bacorn
2008Nurse as Educator: Principles of Teaching and Learning for Nursing Practice. 3rd ed. London: Jones and Bartlett.
and Sandra Hale2003“Translation and Interpreting Assessment in the Context of Educational Measurement.” In Translation Today: Trends and Perspectives, ed. by Gunilla M. Anderman, and Margaret Rogers, 205–224. Clevedon: Multilingual Matters.
., and Ulla Connor1991“Reading and Writing Descriptive and Persuasive Texts.”The Modern Language Journal 75 (3): 314–324.
Chall, Jeanne Sternlicht, and Edgar Dale
1995Readability Revisited: The New Dale-Chall Readability Formula.Cambridge, MA: Brookline Books.
Chen, Peter Y
., and Autumn D. Krauss2004a“Internal Reliability.” In The Sage Encyclopedia of Social Science Research Methods, ed. by Michael S. Lewis-Beck, Alan Bryman, and Tim Futing Liao, 501–502. ThousandOaks, CA: Sage.
“Reliability.” In The Sage Encyclopedia of Social Science Research Methods,
ed. by Michael S. Lewis-Beck, Alan Bryman, and Tim Futing Liao, 952–956. ThousandOaks, CA: Sage.
1988Statistical Power Analysis for the Behavioral Sciences. 2nd ed.Hillsdale, NJ: L. Erlbaum Associates.
Cronbach, Lee J
1951“Coefficient Alpha and the Internal Structure of Tests.”Psychometrika 16 (3): 297–334.
2004The Growth and Maintenance of Linguistic Complexity. Amsterdam: John Benjamins.
Davidson, Janet E
., and Robert J. Sternberg eds 2003The Psychology of Problem Solving. Cambridge: Cambridge University Press.
2004Segmentation in Translation and Translation Memory Systems: An Empirical Investigation of Cognitive Segmentation and Effects of Integrating a Tm-System into the Translation Process. PhD diss. Copenhagen Business School.
2005Expertise and Explicitation in the Translation Process. Amsterdam: John Benjamins.
Feldt, Leonard S
1961“The Use of Extreme Groups to Test for the Presence of a Relationship.”Psychometrika 26 (3): 307–316.
1988From French to English: A Look at the Translation Process in Students, Bilinguals, and Professional Translators. PhD. diss. Harvard University.
Green, Samuel B
1991“How Many Subjects Does It Take to Do a Regression Analysis.”Multivariate Behavioral Research 26 (3): 499–510.
and Stuart Campbell2002“The Interaction between Text Difficulty and Translation Accuracy.”Babel 48 (1): 14–33.
Hart, Sandra G
., and Lowell E. Staveland1988“Development of Nasa-Tlx (Task Load Index): Results of Empirical and Theoretical Research.” In Human Mental Workload, ed. by Peter A. Hancock, and Najmedin Meshkati, 139–183. Amsterdam: North-Holland.
Higgins, Jennifer, Michael Russell
and Thomas Hoffmann2005.”Examining the Effect of Computer-Based Passage Presentation on Reading Test Performance.”The Journal of Technology, Learning, and Assessment 4. http://www.jtla.org.
2009“Quality.” In Routledge Encyclopedia of Translation Studies, ed. by Mona Baker, and Gabriela Saldanha, 222–225. London: Routledge.
Hoyt, Linda, Margaret E. Mooney, and Brenda Parkes
, eds 2003Exploring Informational Texts: From Theory to Practice.Portsmouth, NH: Heinemann.
1990Features of Successful Translation Processes: A Think-aloud Protocol Study. Unpublished licentiate thesis, University of Joensuu, Savonlinna School of Translation Studies.
“Hard Work Will Bear Beautiful Fruit. A Comparison of Two Think-Aloud Protocol Studies.”Meta 41 (1): 60–74.
Tapping the Process: An Explorative Study of the Cognitive and Affective Factors Involved in Translating. University of Joensuu Publications in the Humanities 22. Joensuu: University of Joensuu.
Jensen, Kristian T.H.
2009“Indicators of Text Complexity.”Copenhagen Studies in Language 37: 61–80.
Jex, Henry. R.
1988“Measuring Mental Workload: Problems, Progress, and Promises.” In Human Mental Workload, Advances in Psychology 52, ed. by Peter. A. Hancock, and Najmedin Meshkati, 5–38. Amsterdam: North-Holland.
Kelley, Truman L
1939“The Selection of Upper and Lower Groups for the Validation of Test Items.”Journal of Educational Psychology 30: 17–24.
Kiraly, Donald Charles
1995Pathways to Translation: From Process to Pedagogy.Kent, OH: Kent State University Press.
Krings, Hans P
2001Repairing Texts: Empirical Investigations of Machine Translation Post-Editing Processes. Translated by Geoffrey Koby, Gregory Shreve, Katjz Mischerikow, and Sarah Litzer. Kent, OH: Kent State University Press.
Liu, Minhua, and Yu-Hsien Chiu
2009“Assessing Source Material Difficulty for Consecutive Interpreting: Quantifiable Measures and Holistic Judgment.”Interpreting 11 (2): 244–266.
1979Mental Workload: Its Theory and Measurement. New York: Plenum Press.
Moroney, William F
., David W. Biers, F. Thomas Eggemeier, and Jennifer A. Mitchell1992“A Comparison of Two Scoring Procedures with the Nasa Task Load Index in a Simulated Flight Task.” In Proceedings of the IEEE 1992 National Aerospace and Electronics Conference - Naecon 1992, Dayton, 18–22 May 1992, 734–740.
2005Text Analysis in Translation: Theory, Methodology, and Didactic Application of a Model for Translation-Oriented Text Analysis. 2nd ed. Amsterdam: Rodopi.
Nunnally, Jum C
1978Psychometric Theory. 2nd ed. New York: McGraw-Hill.
O’Donnell, Robert D
., and F. Thomas Eggemeier1986“Workload Assessment Methodology.” In Handbook of Perception and Human Performance. Vol. 2: Cognitive Processes and Performance, ed. by Kenneth R. Boff, Lloyd Kaufman, and James P. Thomas, 42/41–42/49. New York: Wiley.
Paas, Fred G
1992“Training Strategies for Attaining Transfer of Problem-Solving Skill in Statistics: A Cognitive-Load Approach.”Journal of Educational Psychology 84 (4): 429–434.
2005“Investigating Translation Competence: Conceptual and Methodological Issues.”Meta 50 (2): 609–619.
1997Multiple Regression in Behavioral Research: Explanation and Prediction. 3rd ed. United States: Wadsworth/Thomson Learning.
Preacher, Kristopher J
., Derek D. Rucker, Robert C. MacCallum, and W. Alan Nicewander2005“Use of the Extreme Groups Approach: A Critical Reexamination and New Recommendations.”Psychological Methods 10 (2): 178–192.
Pretz, Jean E
., Adam J. Naples, and Robert J. Sternberg2003“Recognizing, Defining, and Representing Problems.” In The Psychology of Problem Solving, ed. by Janet E. Davidson, and Robert J. Sternberg, 3–30. Cambridge: Cambridge University Press.
“Redefining Translation Competence in an Electronic Age: In Defence of a Minimalist Approach.”Meta 48 (4): 481–497.
Rayner, Keith, and Alexander Pollatsek
1989The Psychology of Reading.Hillsdale, NJ: Lawrence Erlbaum.
Shreve, Gregory M
2002“Knowing Translation: Cognitive and Experiential Aspects of Translation Expertise from the Perspective of Expertise Studies.” In Translation Studies: Perspectives on an Emerging Discipline, ed. by Alessandra Riccardi, 150–173. Cambridge: Cambridge University Press.
Sweller, John, Paul L. Ayres
and Slava Kalyuga 2011Cognitive Load Theory. New York: Springer.
Taylor, Wilson L
1953“‘Cloze Procedure’: A New Tool for Measuring Readability.”Journalism Quarterly 30: 415–433.
1987“Think-Aloud Protocols in the Study of the Translation Process.” In CDEF 86: Papers from the Conference of Departments of English in Finland, ed. by Heikki Nyyssönen, Riitta Kataja, and Vesa Komulainen, 39–49. Oulu: University of Oulu.
Tsang, Pamela S
2006“Mental Workload.” In International Encyclopedia of Ergonomics and Human Factors, ed. by Waldemar Karwowski, 809–813. BocaRaton, FL: CRC/Taylor & Francis.
Vidulich, Michael A
1988“The Cognitive Psychology of Subjective Mental Workload.” In Human Mental Workload, ed. by Peter A. Hancock, and Najmedin Meshkati, 219–229. Amsterdam: North-Holland.
Wilson, Glenn F
.. and F. Thomas Eggemeier2006“Mental Workload Measurement.” In International Encyclopedia of Ergonomics and Human Factors, ed. by Waldemar Karwowski, 814–817. BocaRaton, FL: CRC/Taylor & Francis.
1982The Science of Translation: Problems and Methods. Tübinger Beitraäge zur Linguistik. Tuübingen: G. Narr.
Zamanian, Mostafa, and Pooneh Heydari
2012“Readability of Texts: State of the Art.”Theory and Practice in Language Studies 2 (1): 43–53.
Zimmerman, Lynn W
2010ESL, EFL, and Bilingual Education: Exploring Historical, Sociocultural, Linguistic, and Instructional Foundations.Charlotte, NC: Information Age Pub.
Zipf, George Kingsley
1935The Psycho-Biology of Language: An Introduction to Dynamic Philology. Boston: Houghton Mifflin.