Cover not available
Article published In:
Audiovisual Translation: Theoretical and methodological challenges
Edited by Yves Gambier and Sara Ramos Pinto
[Target 28:2] 2016
► pp. 276–287

Psycholinguistics and audiovisual translation

| Macquarie University (Australia) / North-West University (South Africa)
Psycholinguistic investigations of translated audiovisual products have been conducted since at least the 1980s. These mainly concerned the role of subtitles in the processing of language in the context of language acquisition, literacy, and education. This article provides an overview of some of the most productive lines of research from a psycholinguistic angle in audiovisual translation (AVT), focussing on studies that investigated the positive effects of subtitles on language performance, but also on a growing body of behavioural research on the cognitive processing of the language of subtitles. The article evaluates a number of methodologies in some of the most prominent studies on the processing of subtitles, primarily making use of eye tracking, and then provides some thoughts on future directions in psycholinguistic studies on the processing of the language of AVT.
Keywords: audiovisual translation (AVT), psycholinguistics, eye tracking, cognitive load, reading research
Article outline
  • 1.Introduction
  • 2.Performance and reception studies
  • 3.Physiological studies
  • 4.Eye tracking
  • 5.Conclusion: future psycholinguistic methodologies for AVT
  • References
Published online: 9 August 2016
DOI logo
https://doi.org/10.1075/target.28.2.08kru
References (27)
References
Bird, Stephen A., and John Williams. 2002. “The Effect of Bimodal Input on Implicit and Explicit Memory: An Investigation into the Benefits of Within-Language Subtitling.” Applied Psycholinguistics 23 (4): 509–533. DOI logoGoogle Scholar
Bisson, Marie-Josée, Walter Van Heuven, Kathy Conklin, and Richard Tunney. 2012. “Processing of Native and Foreign Language Subtitles in Films: An Eye Tracking Study.” Applied Psycholinguistics 351: 399–418. DOI logoGoogle Scholar
Danan, Martine. 2004. “Captioning and Subtitling: Undervalued Language Learning Strategies.” Meta 49 (1): 67–77. DOI logoGoogle Scholar
Diao, Yali, Paul Chandler, and John Sweller. 2007. “The Effect of Written Text on Comprehension of Spoken English as a Foreign Language.” The American Journal of Psychology 120 (2): 237–261.Google Scholar
d’Ydewalle, Géry, and Wim De Bruycker. 2007. “Eye Movements of Children and Adults While Reading Television Subtitles.” European Psychologist 121: 196–205. DOI logoGoogle Scholar
d’Ydewalle, Géry, and Ingrid Gielen. 1992. “Attention Allocation with Overlapping Sound, Image, and Text.” In Eye Movements and Visual Cognition: Scene Perception and Reading, ed. by Keith Rayner, 415–427. New York: Springer-Verlag. DOI logoGoogle Scholar
d’Ydewalle, Géry, Caroline Praet, Karl Verfaillie, and Johan van Rensbergen. 1991. “Watching Subtitled Television: Automatic Reading Behavior.” Communication Research 181: 650–665. DOI logoGoogle Scholar
Garza, Thomas J.. 1991. “Evaluating the Use of Captioned Video Materials in Advanced Foreign Language Learning.” Foreign Language Annuals 24 (3): 239–258. DOI logoGoogle Scholar
Ghia, Elisa. 2012. “The Impact of Translation Strategies on Subtitle Reading.” In Eye Tracking in Audiovisual Translation, ed. by Elisa Perego, 155–182. Roma: Aracne Editrice.Google Scholar
Just, Marcel Adam, and Patricia A. Carpenter. 1980. “A Theory of Reading: From Eye Fixations to Comprehension.” Psychological Review 87 (4): 329–354. DOI logoGoogle Scholar
Keating, Gregory D. 2014. “Eye Tracking with Text.” In Research Methods in Second Language Psycholinguistics, ed. by Jill Jegerski, and Bill VanPatten, 69–92. New York: Routledge.Google Scholar
Krejtz, Izabela, Agnieszka Szarkowska, and Krzysztof Krejtz. 2013. “The Effects of Shot Changes on Eye Movements in Subtitling.” Journal of Eye Movement Research 6 (5): 1–12.Google Scholar
Kruger, Jan-Louis. 2013. “Subtitles in the Classroom: Balancing the Benefits of Dual Coding with the Cost of Increased Cognitive Load.” Journal for Language Teaching 47 (1): 29–53.Google Scholar
Kruger, Jan-Louis, and Faans Steyn. 2014. “Subtitles and Eye Tracking: Reading and Performance.” Reading Research Quarterly 49 (1): 105–120. DOI logoGoogle Scholar
Kruger, Jan-Louis, Esté Hefer, and Gordon Matthew. 2013. “Measuring the Impact of Subtitles on Cognitive Load: Eye Tracking and Dynamic Audiovisual Texts.” Proceedings of the 2013 Conference on Eye Tracking South Africa, 62–66. New York: ACM. DOI logoGoogle Scholar
Markham, Peter L. 1993. “Captioned Television Videotapes: Effects of Visual Support on Second Language Comprehension.” Journal of Educational Technology Systems 211: 183–191. DOI logoGoogle Scholar
Markham Peter, L. 1999. “Captioned Video-Tapes and Second Language Listening Word Recognition.” Foreign Language Annals 32 (3): 321–328. DOI logoGoogle Scholar
Perego, Elisa, Fabio Del Missier, Marco Porta, and Mauro Mosconi. 2010. “The Cognitive Effectiveness of Subtitle Processing.” Media Psychology 13 (3): 243–272. DOI logoGoogle Scholar
Pulvermüller, Friedemann. 2009. “Word Processing in the Brain as Revealed by Neuro­physiological Imaging.” In The Oxford Handbook of Psycholinguistics, ed. by M. Gareth Gaskell, 119–139. Oxford: Oxford University Press.Google Scholar
Rajendran, Dhevi, Andrew Duchowski, Pilar Orero, Juan Martínez, and Pablo Romero-Fresco. 2013. “Effects of Text Chunking on Subtitling: A Quantitative and Qualitative Examination.” Perspectives: Studies in Translatology 21 (1): 5–31. DOI logoGoogle Scholar
Rayner, Keith. 1998. “Eye Movements in Reading and Information Processing: 20 Years of Research.” Psychological Bulletin 1241: 372–422. DOI logoGoogle Scholar
Specker, Elizabeth A. 2008. L1/L2 Eye Movement Reading of Closed Captioning: A Multimodal Analysis of Multimodal Use. PhD diss. University of Arizona.Google Scholar
Vanderplank, Robert. 1988. “The Value of Teletext Sub-titles in Language Learning.” ELT Journal 42 (4): 272–281. DOI logoGoogle Scholar
. 1990. “Paying Attention to the Words: Practical and Theoretical Problems in Watching Television Programmes with Uni-lingual (CEEFAX) Sub-titles.” System 18 (2): 221–234. DOI logoGoogle Scholar
. 2013. “‘Effects of’ and ‘Effects with’ Captions: How Exactly Does Watching a TV Programme with Same-Language Subtitles Make a Difference to Language Learners?” Language Teaching 1–16.Google Scholar
VanPatten, Bill. 2014. “The Psycholinguistics of SLA.” In Research Methods in Second Language Psycholinguistics (Second Language Acquisition Research Series), ed. by Jill Jegerski, and Bill VanPatten, 1–19. New York: Routledge.Google Scholar
Winke, Paula, Susan Gass, and Tetyana Syderenko. 2013. “Factors Influencing the Use of Captions by Foreign Language Learners: An Eye Tracking Study.” The Modern Language Journal 97 (1): 254–275. DOI logoGoogle Scholar
Cited by (14)

Cited by 14 other publications

Brescia-Zapata, Marta, Krzysztof Krejtz, Andrew T. Duchowski, Chris J. Hughes & Pilar Orero
2023. Subtitles in VR 360° video. Results from an eye-tracking experiment. Perspectives ► pp. 1 ff. DOI logo
Ragni, Valentina
2023. Reverse subtitles in foreign language learning. Translation and Translanguaging in Multilingual Contexts 9:2 ► pp. 261 ff. DOI logo
Xiao, Kairong & Wanyu Li
2023. Chapter 6. In the eye of the beholder. In Translation in Transition [American Translators Association Scholarly Monograph Series, XX], ► pp. 104 ff. DOI logo
Hermosa-Ramírez, Irene
2022. Physiological instruments meet mixed methods in Media Accessibility. Translation Spaces 11:1 ► pp. 38 ff. DOI logo
Kruger, Jan-Louis & Sixin Liao
2022. Establishing a theoretical framework for AVT research. Translation Spaces 11:1 ► pp. 12 ff. DOI logo
Liao, Sixin, Lili Yu, Jan-Louis Kruger & Erik D. Reichle
2022. The impact of audio on the reading of intralingual versus interlingual subtitles: Evidence from eye movements. Applied Psycholinguistics 43:1 ► pp. 237 ff. DOI logo
Guillot, Marie-Noëlle
2020. Ocean’s Eleven Scene 12: the sample as methodological cogitation – rationale and data. Perspectives 28:6 ► pp. 816 ff. DOI logo
Pedersen, Jan
2020. Audiovisual Translation Norms and Guidelines. In The Palgrave Handbook of Audiovisual Translation and Media Accessibility [Palgrave Studies in Translating and Interpreting, ], ► pp. 417 ff. DOI logo
Walker, Callum
2019. A cognitive perspective on equivalent effect: using eye tracking to measure equivalence in source text and target text cognitive effects on readers. Perspectives 27:1 ► pp. 124 ff. DOI logo
Walker, Callum
2021. The Cognitive Paradigm in Translation Studies. In An Eye-Tracking Study of Equivalent Effect in Translation, ► pp. 13 ff. DOI logo
Federici, Federico M. & Callum Walker
2018. Chapter 2. A mapping exercise. In Eye Tracking and Multidisciplinary Studies on Translation [Benjamins Translation Library, 143], ► pp. 11 ff. DOI logo
Kruger, Jan-Louis & Stephen Doherty
2018. Triangulation of online and offline measures of processing and reception in AVT. In Reception Studies and Audiovisual Translation [Benjamins Translation Library, 141], ► pp. 91 ff. DOI logo
Szarkowska, Agnieszka, Łukasz Dutka, Anna Szychowska & Olga Pilipczuk
2018. Chapter 9. Visual attention distribution in intralingual respeaking. In Eye Tracking and Multidisciplinary Studies on Translation [Benjamins Translation Library, 143], ► pp. 185 ff. DOI logo
Kruger, Haidee & Jan‐Louis Kruger
2017. Cognition and Reception. In The Handbook of Translation and Cognition, ► pp. 71 ff. DOI logo

This list is based on CrossRef data as of 27 september 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.