Macquarie University (Australia) / North-West University (South Africa)
Abstract
Psycholinguistic investigations of translated audiovisual products have been conducted since at least the 1980s. These mainly concerned the role of subtitles in the processing of language in the context of language acquisition, literacy, and education. This article provides an overview of some of the most productive lines of research from a psycholinguistic angle in audiovisual translation (AVT), focussing on studies that investigated the positive effects of subtitles on language performance, but also on a growing body of behavioural research on the cognitive processing of the language of subtitles. The article evaluates a number of methodologies in some of the most prominent studies on the processing of subtitles, primarily making use of eye tracking, and then provides some thoughts on future directions in psycholinguistic studies on the processing of the language of AVT.
The link between audiovisual translation (AVT) and psycholinguistics centres on the various ways in which language is foregrounded in AVT, making this mode a useful context for studying the way individuals process language. In AVT, the processing of language is complicated by the fact that this processing occurs alongside the processing of a number of other codes due to the multimodal nature of the audiovisual text.
References
Bird, Stephen A., and John Williams
2002 “The Effect of Bimodal Input on Implicit and Explicit Memory: An Investigation into the Benefits of Within-Language Subtitling.” Applied Psycholinguistics 23 (4): 509–533.
Bisson, Marie-Josée, Walter Van Heuven, Kathy Conklin, and Richard Tunney
2012 “Processing of Native and Foreign Language Subtitles in Films: An Eye Tracking Study.” Applied Psycholinguistics 35: 399–418.
Danan, Martine
2004 “Captioning and Subtitling: Undervalued Language Learning Strategies.” Meta 49 (1): 67–77.
Diao, Yali, Paul Chandler, and John Sweller
2007 “The Effect of Written Text on Comprehension of Spoken English as a Foreign Language.” The American Journal of Psychology 120 (2): 237–261.
d’Ydewalle, Géry, and Wim De Bruycker
2007 “Eye Movements of Children and Adults While Reading Television Subtitles.” European Psychologist 12: 196–205.
d’Ydewalle, Géry, and Ingrid Gielen
1992 “Attention Allocation with Overlapping Sound, Image, and Text.” In Eye Movements and Visual Cognition: Scene Perception and Reading, ed. by Keith Rayner, 415–427. New York: Springer-Verlag.
d’Ydewalle, Géry, Caroline Praet, Karl Verfaillie, and Johan van Rensbergen
1991 “Watching Subtitled Television: Automatic Reading Behavior.” Communication Research 18: 650–665.
Garza, Thomas J.
1991 “Evaluating the Use of Captioned Video Materials in Advanced Foreign Language Learning.” Foreign Language Annuals 24 (3): 239–258.
Ghia, Elisa
2012 “The Impact of Translation Strategies on Subtitle Reading.” In Eye Tracking in Audiovisual Translation, ed. by Elisa Perego, 155–182. Roma: Aracne Editrice.
Just, Marcel Adam, and Patricia A. Carpenter
1980 “A Theory of Reading: From Eye Fixations to Comprehension.” Psychological Review 87 (4): 329–354.
Keating, Gregory D.
2014 “Eye Tracking with Text.” In Research Methods in Second Language Psycholinguistics, ed. by Jill Jegerski, and Bill VanPatten, 69–92. New York: Routledge.
Krejtz, Izabela, Agnieszka Szarkowska, and Krzysztof Krejtz
2013 “The Effects of Shot Changes on Eye Movements in Subtitling.” Journal of Eye Movement Research 6 (5): 1–12.
Kruger, Jan-Louis
2013 “Subtitles in the Classroom: Balancing the Benefits of Dual Coding with the Cost of Increased Cognitive Load.” Journal for Language Teaching 47 (1): 29–53.
Kruger, Jan-Louis, and Faans Steyn
2014 “Subtitles and Eye Tracking: Reading and Performance.” Reading Research Quarterly 49 (1): 105–120.
Kruger, Jan-Louis, Esté Hefer, and Gordon Matthew
2013 “Measuring the Impact of Subtitles on Cognitive Load: Eye Tracking and Dynamic Audiovisual Texts.” Proceedings of the 2013 Conference on Eye Tracking South Africa, 62–66. New York: ACM.
Markham, Peter L.
1993 “Captioned Television Videotapes: Effects of Visual Support on Second Language Comprehension.” Journal of Educational Technology Systems 21: 183–191.
Markham Peter, L.
1999 “Captioned Video-Tapes and Second Language Listening Word Recognition.” Foreign Language Annals 32 (3): 321–328.
Perego, Elisa, Fabio Del Missier, Marco Porta, and Mauro Mosconi
2010 “The Cognitive Effectiveness of Subtitle Processing.” Media Psychology 13 (3): 243–272.
Pulvermüller, Friedemann
2009 “Word Processing in the Brain as Revealed by Neurophysiological Imaging.” In The Oxford Handbook of Psycholinguistics, ed. by M. Gareth Gaskell, 119–139. Oxford: Oxford University Press.
Rajendran, Dhevi, Andrew Duchowski, Pilar Orero, Juan Martínez, and Pablo Romero-Fresco
2013 “Effects of Text Chunking on Subtitling: A Quantitative and Qualitative Examination.” Perspectives: Studies in Translatology 21 (1): 5–31.
Rayner, Keith
1998 “Eye Movements in Reading and Information Processing: 20 Years of Research.” Psychological Bulletin 124: 372–422.
Specker, Elizabeth A.
2008L1/L2 Eye Movement Reading of Closed Captioning: A Multimodal Analysis of Multimodal Use. PhD diss. University of Arizona.
Vanderplank, Robert
1988 “The Value of Teletext Sub-titles in Language Learning.” ELT Journal 42 (4): 272–281.
Vanderplank, Robert
1990 “Paying Attention to the Words: Practical and Theoretical Problems in Watching Television Programmes with Uni-lingual (CEEFAX) Sub-titles.” System 18 (2): 221–234.
Vanderplank, Robert
2013 “ ‘Effects of’ and ‘Effects with’ Captions: How Exactly Does Watching a TV Programme with Same-Language Subtitles Make a Difference to Language Learners?” Language Teaching 1–16.
VanPatten, Bill
2014 “The Psycholinguistics of SLA.” In Research Methods in Second Language Psycholinguistics (Second Language Acquisition Research Series), ed. by Jill Jegerski, and Bill VanPatten, 1–19. New York: Routledge.
Winke, Paula, Susan Gass, and Tetyana Syderenko
2013 “Factors Influencing the Use of Captions by Foreign Language Learners: An Eye Tracking Study.” The Modern Language Journal 97 (1): 254–275.
