Article published In:
Audiovisual Translation: Theoretical and methodological challengesEdited by Yves Gambier and Sara Ramos Pinto
[Target 28:2] 2016
► pp. 276–287
Edited by Yves Gambier and Sara Ramos Pinto
[Target 28:2] 2016
► pp. 276–287
Psycholinguistics and audiovisual translation
Psycholinguistic investigations of translated audiovisual products have been conducted since at least the 1980s. These mainly concerned the role of subtitles in the processing of language in the context of language acquisition, literacy, and education. This article provides an overview of some of the most productive lines of research from a psycholinguistic angle in audiovisual translation (AVT), focussing on studies that investigated the positive effects of subtitles on language performance, but also on a growing body of behavioural research on the cognitive processing of the language of subtitles. The article evaluates a number of methodologies in some of the most prominent studies on the processing of subtitles, primarily making use of eye tracking, and then provides some thoughts on future directions in psycholinguistic studies on the processing of the language of AVT.
Keywords: audiovisual translation (AVT), psycholinguistics, eye tracking, cognitive load, reading research
Article outline
- 1.Introduction
- 2.Performance and reception studies
- 3.Physiological studies
- 4.Eye tracking
- 5.Conclusion: future psycholinguistic methodologies for AVT
-
References
References
Bird, Stephen A., and John Williams
2002 “The Effect of Bimodal Input on Implicit and Explicit Memory: An Investigation into the Benefits of Within-Language Subtitling.” Applied Psycholinguistics 23 (4): 509–533. 
Bisson, Marie-Josée, Walter Van Heuven, Kathy Conklin, and Richard Tunney
2012 “Processing of Native and Foreign Language Subtitles in Films: An Eye Tracking Study.” Applied Psycholinguistics 351: 399–418.
Danan, Martine
2004 “Captioning and Subtitling: Undervalued Language Learning Strategies.” Meta 49 (1): 67–77.
Diao, Yali, Paul Chandler, and John Sweller
2007 “The Effect of Written Text on Comprehension of Spoken English as a Foreign Language.” The American Journal of Psychology 120 (2): 237–261.
d’Ydewalle, Géry, and Wim De Bruycker
2007 “Eye Movements of Children and Adults While Reading Television Subtitles.” European Psychologist 121: 196–205.
d’Ydewalle, Géry, and Ingrid Gielen
1992 “Attention Allocation with Overlapping Sound, Image, and Text.” In Eye Movements and Visual Cognition: Scene Perception and Reading, ed. by Keith Rayner, 415–427. New York: Springer-Verlag.
d’Ydewalle, Géry, Caroline Praet, Karl Verfaillie, and Johan van Rensbergen
1991 “Watching Subtitled Television: Automatic Reading Behavior.” Communication Research 181: 650–665.
Garza, Thomas J.
1991 “Evaluating the Use of Captioned Video Materials in Advanced Foreign Language Learning.” Foreign Language Annuals 24 (3): 239–258.
Ghia, Elisa
2012 “The Impact of Translation Strategies on Subtitle Reading.” In Eye Tracking in Audiovisual Translation, ed. by Elisa Perego, 155–182. Roma: Aracne Editrice.
Just, Marcel Adam, and Patricia A. Carpenter
1980 “A Theory of Reading: From Eye Fixations to Comprehension.” Psychological Review 87 (4): 329–354.
Keating, Gregory D.
2014 “Eye Tracking with Text.” In Research Methods in Second Language Psycholinguistics, ed. by Jill Jegerski, and Bill VanPatten, 69–92. New York: Routledge.
Krejtz, Izabela, Agnieszka Szarkowska, and Krzysztof Krejtz
2013 “The Effects of Shot Changes on Eye Movements in Subtitling.” Journal of Eye Movement Research 6 (5): 1–12.
Kruger, Jan-Louis
2013 “Subtitles in the Classroom: Balancing the Benefits of Dual Coding with the Cost of Increased Cognitive Load.” Journal for Language Teaching 47 (1): 29–53.
Kruger, Jan-Louis, and Faans Steyn
2014 “Subtitles and Eye Tracking: Reading and Performance.” Reading Research Quarterly 49 (1): 105–120.
Kruger, Jan-Louis, Esté Hefer, and Gordon Matthew
2013 “Measuring the Impact of Subtitles on Cognitive Load: Eye Tracking and Dynamic Audiovisual Texts.” Proceedings of the 2013 Conference on Eye Tracking South Africa, 62–66. New York: ACM.
Markham, Peter L.
1993 “Captioned Television Videotapes: Effects of Visual Support on Second Language Comprehension.” Journal of Educational Technology Systems 211: 183–191.
Markham Peter, L.
1999 “Captioned Video-Tapes and Second Language Listening Word Recognition.” Foreign Language Annals 32 (3): 321–328.
Perego, Elisa, Fabio Del Missier, Marco Porta, and Mauro Mosconi
2010 “The Cognitive Effectiveness of Subtitle Processing.” Media Psychology 13 (3): 243–272.
Pulvermüller, Friedemann
2009 “Word Processing in the Brain as Revealed by Neurophysiological Imaging.” In The Oxford Handbook of Psycholinguistics, ed. by M. Gareth Gaskell, 119–139. Oxford: Oxford University Press.
Rajendran, Dhevi, Andrew Duchowski, Pilar Orero, Juan Martínez, and Pablo Romero-Fresco
2013 “Effects of Text Chunking on Subtitling: A Quantitative and Qualitative Examination.” Perspectives: Studies in Translatology 21 (1): 5–31.
Rayner, Keith
1998 “Eye Movements in Reading and Information Processing: 20 Years of Research.” Psychological Bulletin 1241: 372–422.
Specker, Elizabeth A.
2008 L1/L2 Eye Movement Reading of Closed Captioning: A Multimodal Analysis of Multimodal Use. PhD diss. University of Arizona.
Vanderplank, Robert
1988 “The Value of Teletext Sub-titles in Language Learning.” ELT Journal 42 (4): 272–281.
1990 “Paying Attention to the Words: Practical and Theoretical Problems in Watching Television Programmes with Uni-lingual (CEEFAX) Sub-titles.” System 18 (2): 221–234.
2013 “ ‘Effects of’ and ‘Effects with’ Captions: How Exactly Does Watching a TV Programme with Same-Language Subtitles Make a Difference to Language Learners?” Language Teaching 1–16.
VanPatten, Bill
2014 “The Psycholinguistics of SLA.” In Research Methods in Second Language Psycholinguistics (Second Language Acquisition Research Series), ed. by Jill Jegerski, and Bill VanPatten, 1–19. New York: Routledge.
Winke, Paula, Susan Gass, and Tetyana Syderenko
2013 “Factors Influencing the Use of Captions by Foreign Language Learners: An Eye Tracking Study.” The Modern Language Journal 97 (1): 254–275.
Cited by
Cited by 14 other publications
Brescia-Zapata, Marta, Krzysztof Krejtz, Andrew T. Duchowski, Chris J. Hughes & Pilar Orero
2023. Subtitles in VR 360° video. Results from an eye-tracking experiment. Perspectives ► pp. 1 ff. 
Federici, Federico M. & Callum Walker
2018. Chapter 2. A mapping exercise. In Eye Tracking and Multidisciplinary Studies on Translation [Benjamins Translation Library, 143], ► pp. 11 ff.
Guillot, Marie-Noëlle
2020. Ocean’s Eleven Scene 12: the sample as methodological cogitation – rationale and data. Perspectives 28:6 ► pp. 816 ff.
Hermosa-Ramírez, Irene
2022. Physiological instruments meet mixed methods in Media Accessibility. Translation Spaces 11:1 ► pp. 38 ff.
Kruger, Haidee & Jan‐Louis Kruger
2017. Cognition and Reception. In The Handbook of Translation and Cognition, ► pp. 71 ff.
Kruger, Jan-Louis & Stephen Doherty
2018. Triangulation of online and offline measures of processing and reception in AVT. In Reception Studies and Audiovisual Translation [Benjamins Translation Library, 141], ► pp. 91 ff.
Kruger, Jan-Louis & Sixin Liao
Liao, Sixin, Lili Yu, Jan-Louis Kruger & Erik D. Reichle
2022. The impact of audio on the reading of intralingual versus interlingual subtitles: Evidence from eye movements. Applied Psycholinguistics 43:1 ► pp. 237 ff.
Pedersen, Jan
2020. Audiovisual Translation Norms and Guidelines. In The Palgrave Handbook of Audiovisual Translation and Media Accessibility [Palgrave Studies in Translating and Interpreting, ], ► pp. 417 ff.
Ragni, Valentina
2023. Reverse subtitles in foreign language learning. Translation and Translanguaging in Multilingual Contexts 9:2 ► pp. 261 ff.
Szarkowska, Agnieszka, Łukasz Dutka, Anna Szychowska & Olga Pilipczuk
2018. Chapter 9. Visual attention distribution in intralingual respeaking. In Eye Tracking and Multidisciplinary Studies on Translation [Benjamins Translation Library, 143], ► pp. 185 ff.
Walker, Callum
2019. A cognitive perspective on equivalent effect: using eye tracking to measure equivalence in source text and target text cognitive effects on readers. Perspectives 27:1 ► pp. 124 ff.
Walker, Callum
2021. The Cognitive Paradigm in Translation Studies. In An Eye-Tracking Study of Equivalent Effect in Translation, ► pp. 13 ff.
Xiao, Kairong & Wanyu Li
2023. Chapter 6. In the eye of the beholder. In Translation in Transition [American Translators Association Scholarly Monograph Series, XX], ► pp. 104 ff.
This list is based on CrossRef data as of 8 april 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.