Psycholinguistics and audiovisual translation

Jan-Louis Kruger
Macquarie University (Australia) / North-West University (South Africa)

Psycholinguistic investigations of translated audiovisual products have been conducted since at least the 1980s. These mainly concerned the role of subtitles in the processing of language in the context of language acquisition, literacy, and education. This article provides an overview of some of the most productive lines of research from a psycholinguistic angle in audiovisual translation (AVT), focussing on studies that investigated the positive effects of subtitles on language performance, but also on a growing body of behavioural research on the cognitive processing of the language of subtitles. The article evaluates a number of methodologies in some of the most prominent studies on the processing of subtitles, primarily making use of eye tracking, and then provides some thoughts on future directions in psycholinguistic studies on the processing of the language of AVT.

Table of contents

The link between audiovisual translation (AVT) and psycholinguistics centres on the various ways in which language is foregrounded in AVT, making this mode a useful context for studying the way individuals process language. In AVT, the processing of language is complicated by the fact that this processing occurs alongside the processing of a number of other codes due to the multimodal nature of the audiovisual text.

Full-text access is restricted to subscribers. Log in to obtain additional credentials. For subscription information see Subscription & Price. Direct PDF access to this article can be purchased through our e-platform.


Bird, Stephen A., and John Williams
2002 “The Effect of Bimodal Input on Implicit and Explicit Memory: An Investigation into the Benefits of Within-Language Subtitling.” Applied Psycholinguistics 23 (4): 509–533. DOI logoGoogle Scholar
Bisson, Marie-Josée, Walter Van Heuven, Kathy Conklin, and Richard Tunney
2012 “Processing of Native and Foreign Language Subtitles in Films: An Eye Tracking Study.” Applied Psycholinguistics 35: 399–418. DOI logoGoogle Scholar
Danan, Martine
2004 “Captioning and Subtitling: Undervalued Language Learning Strategies.” Meta 49 (1): 67–77. DOI logoGoogle Scholar
Diao, Yali, Paul Chandler, and John Sweller
2007 “The Effect of Written Text on Comprehension of Spoken English as a Foreign Language.” The American Journal of Psychology 120 (2): 237–261.Google Scholar
d’Ydewalle, Géry, and Wim De Bruycker
2007 “Eye Movements of Children and Adults While Reading Television Subtitles.” European Psychologist 12: 196–205. DOI logoGoogle Scholar
d’Ydewalle, Géry, and Ingrid Gielen
1992 “Attention Allocation with Overlapping Sound, Image, and Text.” In Eye Movements and Visual Cognition: Scene Perception and Reading, ed. by Keith Rayner, 415–427. New York: Springer-Verlag. DOI logoGoogle Scholar
d’Ydewalle, Géry, Caroline Praet, Karl Verfaillie, and Johan van Rensbergen
1991 “Watching Subtitled Television: Automatic Reading Behavior.” Communication Research 18: 650–665. DOI logoGoogle Scholar
Garza, Thomas J.
1991 “Evaluating the Use of Captioned Video Materials in Advanced Foreign Language Learning.” Foreign Language Annuals 24 (3): 239–258. DOI logoGoogle Scholar
Ghia, Elisa
2012 “The Impact of Translation Strategies on Subtitle Reading.” In Eye Tracking in Audiovisual Translation, ed. by Elisa Perego, 155–182. Roma: Aracne Editrice.Google Scholar
Just, Marcel Adam, and Patricia A. Carpenter
1980 “A Theory of Reading: From Eye Fixations to Comprehension.” Psychological Review 87 (4): 329–354. DOI logoGoogle Scholar
Keating, Gregory D.
2014 “Eye Tracking with Text.” In Research Methods in Second Language Psycholinguistics, ed. by Jill Jegerski, and Bill VanPatten, 69–92. New York: Routledge.Google Scholar
Krejtz, Izabela, Agnieszka Szarkowska, and Krzysztof Krejtz
2013 “The Effects of Shot Changes on Eye Movements in Subtitling.” Journal of Eye Movement Research 6 (5): 1–12.Google Scholar
Kruger, Jan-Louis
2013 “Subtitles in the Classroom: Balancing the Benefits of Dual Coding with the Cost of Increased Cognitive Load.” Journal for Language Teaching 47 (1): 29–53.Google Scholar
Kruger, Jan-Louis, and Faans Steyn
2014 “Subtitles and Eye Tracking: Reading and Performance.” Reading Research Quarterly 49 (1): 105–120. DOI logoGoogle Scholar
Kruger, Jan-Louis, Esté Hefer, and Gordon Matthew
2013 “Measuring the Impact of Subtitles on Cognitive Load: Eye Tracking and Dynamic Audiovisual Texts.” Proceedings of the 2013 Conference on Eye Tracking South Africa, 62–66. New York: ACM. DOI logoGoogle Scholar
Markham, Peter L.
1993 “Captioned Television Videotapes: Effects of Visual Support on Second Language Comprehension.” Journal of Educational Technology Systems 21: 183–191. DOI logoGoogle Scholar
Markham Peter, L.
1999 “Captioned Video-Tapes and Second Language Listening Word Recognition.” Foreign Language Annals 32 (3): 321–328. DOI logoGoogle Scholar
Perego, Elisa, Fabio Del Missier, Marco Porta, and Mauro Mosconi
2010 “The Cognitive Effectiveness of Subtitle Processing.” Media Psychology 13 (3): 243–272. DOI logoGoogle Scholar
Pulvermüller, Friedemann
2009 “Word Processing in the Brain as Revealed by Neuro­physiological Imaging.” In The Oxford Handbook of Psycholinguistics, ed. by M. Gareth Gaskell, 119–139. Oxford: Oxford University Press.Google Scholar
Rajendran, Dhevi, Andrew Duchowski, Pilar Orero, Juan Martínez, and Pablo Romero-Fresco
2013 “Effects of Text Chunking on Subtitling: A Quantitative and Qualitative Examination.” Perspectives: Studies in Translatology 21 (1): 5–31. DOI logoGoogle Scholar
Rayner, Keith
1998 “Eye Movements in Reading and Information Processing: 20 Years of Research.” Psychological Bulletin 124: 372–422. DOI logoGoogle Scholar
Specker, Elizabeth A.
2008L1/L2 Eye Movement Reading of Closed Captioning: A Multimodal Analysis of Multimodal Use. PhD diss. University of Arizona.Google Scholar
Vanderplank, Robert
1988 “The Value of Teletext Sub-titles in Language Learning.” ELT Journal 42 (4): 272–281. DOI logoGoogle Scholar
1990 “Paying Attention to the Words: Practical and Theoretical Problems in Watching Television Programmes with Uni-lingual (CEEFAX) Sub-titles.” System 18 (2): 221–234. DOI logoGoogle Scholar
2013 “ ‘Effects of’ and ‘Effects with’ Captions: How Exactly Does Watching a TV Programme with Same-Language Subtitles Make a Difference to Language Learners?Language Teaching 1–16.Google Scholar
VanPatten, Bill
2014 “The Psycholinguistics of SLA.” In Research Methods in Second Language Psycholinguistics (Second Language Acquisition Research Series), ed. by Jill Jegerski, and Bill VanPatten, 1–19. New York: Routledge.Google Scholar
Winke, Paula, Susan Gass, and Tetyana Syderenko
2013 “Factors Influencing the Use of Captions by Foreign Language Learners: An Eye Tracking Study.” The Modern Language Journal 97 (1): 254–275. DOI logoGoogle Scholar