Measuring translation difficulty: An empirical study
Sanjun Sun and Gregory M. Shreve
Beijing Foreign Studies University | Kent State University
The purpose of this study was to find a method to measure difficulty in a translation task. Readability formulas have been suggested to be a useful tool and yet this needs to be empirically tested. In this study, NASA Task Load Index, a multidimensional scale for measuring mental workload, was used to assess the level of translation difficulty for the translator. It was found that a text’s readability only partially accounts for its translation difficulty level. Translation quality score was found to be an unreliable indicator of translation difficulty level, while time-on-task was significantly, but weakly, related to translation difficulty level. A formula was developed to predict a text’s translation difficulty level for a translator by using the translator’s pre-translation rating.
Knowing the difficulty level of a translation assignment is important in translation pedagogy, accreditation and research, as well as for the language industry. For instance, in process-oriented translation research, researchers have no standards to refer to when they choose test passages, and the texts used are diverse in terms of text type, length and, possibly, difficulty (see Krings 2001, 74). This makes it hard for one to evaluate the comparability of experimental results between these studies. For example, the use of different translation strategies in terms of type and frequency might vary depending on the translation difficulty level of the texts. Dragsted (2004) found in her empirical study that professional translators would adopt a more novice-like behavior during the translation of a difficult text than during the translation of an easy text. Thus, translation difficulty is an important variable in translation process research.
Anagnostou, Nikolaos K
., and George R.S. Weir2007“From Corpus-Based Collocation Frequencies to Readability Measure.” In Texts, Textbooks and Readability, ed. by George R. S. Weir, and Toshiaki Ozasa, 34–48. Glasgow: University of Strathclyde Publishing.
2008Nurse as Educator: Principles of Teaching and Learning for Nursing Practice. 3rd ed. London: Jones and Bartlett.
and Sandra Hale2003“Translation and Interpreting Assessment in the Context of Educational Measurement.” In Translation Today: Trends and Perspectives, ed. by Gunilla M. Anderman, and Margaret Rogers, 205–224. Clevedon: Multilingual Matters.
1995Readability Revisited: The New Dale-Chall Readability Formula.Cambridge, MA: Brookline Books.
Chen, Peter Y
., and Autumn D. Krauss2004a“Internal Reliability.” In The Sage Encyclopedia of Social Science Research Methods, ed. by Michael S. Lewis-Beck, Alan Bryman, and Tim Futing Liao, 501–502. ThousandOaks, CA: Sage.
“Reliability.” In The Sage Encyclopedia of Social Science Research Methods,
ed. by Michael S. Lewis-Beck, Alan Bryman, and Tim Futing Liao, 952–956. ThousandOaks, CA: Sage.
1988Statistical Power Analysis for the Behavioral Sciences. 2nd ed.Hillsdale, NJ: L. Erlbaum Associates.
Cronbach, Lee J
1951“Coefficient Alpha and the Internal Structure of Tests.”Psychometrika 16 (3): 297–334.
2004The Growth and Maintenance of Linguistic Complexity. Amsterdam: John Benjamins.
., and Robert J. Sternberg eds 2003The Psychology of Problem Solving. Cambridge: Cambridge University Press.
2004Segmentation in Translation and Translation Memory Systems: An Empirical Investigation of Cognitive Segmentation and Effects of Integrating a Tm-System into the Translation Process. PhD diss. Copenhagen Business School.
., and Lowell E. Staveland1988“Development of Nasa-Tlx (Task Load Index): Results of Empirical and Theoretical Research.” In Human Mental Workload, ed. by Peter A. Hancock, and Najmedin Meshkati, 139–183. Amsterdam: North-Holland.
1995Pathways to Translation: From Process to Pedagogy.Kent, OH: Kent State University Press.
Krings, Hans P
2001Repairing Texts: Empirical Investigations of Machine Translation Post-Editing Processes. Translated by Geoffrey Koby, Gregory Shreve, Katjz Mischerikow, and Sarah Litzer. Kent, OH: Kent State University Press.
1979Mental Workload: Its Theory and Measurement. New York: Plenum Press.
Moroney, William F
., David W. Biers, F. Thomas Eggemeier, and Jennifer A. Mitchell1992“A Comparison of Two Scoring Procedures with the Nasa Task Load Index in a Simulated Flight Task.” In Proceedings of the IEEE 1992 National Aerospace and Electronics Conference - Naecon 1992, Dayton, 18–22 May 1992, 734–740.
1978Psychometric Theory. 2nd ed. New York: McGraw-Hill.
O’Donnell, Robert D
., and F. Thomas Eggemeier1986“Workload Assessment Methodology.” In Handbook of Perception and Human Performance. Vol. 2: Cognitive Processes and Performance, ed. by Kenneth R. Boff, Lloyd Kaufman, and James P. Thomas, 42/41–42/49. New York: Wiley.
Paas, Fred G
1992“Training Strategies for Attaining Transfer of Problem-Solving Skill in Statistics: A Cognitive-Load Approach.”Journal of Educational Psychology 84 (4): 429–434.
2005“Investigating Translation Competence: Conceptual and Methodological Issues.”Meta 50 (2): 609–619.
1997Multiple Regression in Behavioral Research: Explanation and Prediction. 3rd ed. United States: Wadsworth/Thomson Learning.
Preacher, Kristopher J
., Derek D. Rucker, Robert C. MacCallum, and W. Alan Nicewander2005“Use of the Extreme Groups Approach: A Critical Reexamination and New Recommendations.”Psychological Methods 10 (2): 178–192.
Pretz, Jean E
., Adam J. Naples, and Robert J. Sternberg2003“Recognizing, Defining, and Representing Problems.” In The Psychology of Problem Solving, ed. by Janet E. Davidson, and Robert J. Sternberg, 3–30. Cambridge: Cambridge University Press.
“Redefining Translation Competence in an Electronic Age: In Defence of a Minimalist Approach.”Meta 48 (4): 481–497.
Rayner, Keith, and Alexander Pollatsek
1989The Psychology of Reading.Hillsdale, NJ: Lawrence Erlbaum.
Shreve, Gregory M
2002“Knowing Translation: Cognitive and Experiential Aspects of Translation Expertise from the Perspective of Expertise Studies.” In Translation Studies: Perspectives on an Emerging Discipline, ed. by Alessandra Riccardi, 150–173. Cambridge: Cambridge University Press.
Sweller, John, Paul L. Ayres
and Slava Kalyuga 2011Cognitive Load Theory. New York: Springer.
Taylor, Wilson L
1953“‘Cloze Procedure’: A New Tool for Measuring Readability.”Journalism Quarterly 30: 415–433.
1987“Think-Aloud Protocols in the Study of the Translation Process.” In CDEF 86: Papers from the Conference of Departments of English in Finland, ed. by Heikki Nyyssönen, Riitta Kataja, and Vesa Komulainen, 39–49. Oulu: University of Oulu.
Tsang, Pamela S
2006“Mental Workload.” In International Encyclopedia of Ergonomics and Human Factors, ed. by Waldemar Karwowski, 809–813. BocaRaton, FL: CRC/Taylor & Francis.
Vidulich, Michael A
1988“The Cognitive Psychology of Subjective Mental Workload.” In Human Mental Workload, ed. by Peter A. Hancock, and Najmedin Meshkati, 219–229. Amsterdam: North-Holland.
Wilson, Glenn F
.. and F. Thomas Eggemeier2006“Mental Workload Measurement.” In International Encyclopedia of Ergonomics and Human Factors, ed. by Waldemar Karwowski, 814–817. BocaRaton, FL: CRC/Taylor & Francis.
1982The Science of Translation: Problems and Methods. Tübinger Beitraäge zur Linguistik. Tuübingen: G. Narr.
Zamanian, Mostafa, and Pooneh Heydari
2012“Readability of Texts: State of the Art.”Theory and Practice in Language Studies 2 (1): 43–53.
Zimmerman, Lynn W
2010ESL, EFL, and Bilingual Education: Exploring Historical, Sociocultural, Linguistic, and Instructional Foundations.Charlotte, NC: Information Age Pub.
Zipf, George Kingsley
1935The Psycho-Biology of Language: An Introduction to Dynamic Philology. Boston: Houghton Mifflin.