Part of
Reembedding Translation Process ResearchEdited by Ricardo Muñoz Martín
[Benjamins Translation Library 128] 2016
► pp. 171–194
Edited by Ricardo Muñoz Martín
[Benjamins Translation Library 128] 2016
► pp. 171–194
Towards a cognitive audiovisual translatology
subtitles and embodied cognition
This chapter presents the findings of a study to investigate the impact of subtitles on the processing of audiovisual texts in terms of levels of self-reported engagement with the text. It also presents a methodology for investigating the neural processing of subtitles using electroencephalography (EEG) and psychometrics. By establishing the beta coherence between prefrontal and posterior regions, we can determine whether the effect of subtitles can be captured through EEG, and how this is related to psychological immersion. We therefore use EEG to investigate the cognitive processing of subtitled film in combination with subjective psychometrics. Initial validation of this methodology shows that adding same language subtitles results in statistically significantly higher levels of immersion and enjoyment, lending support to a view that subtitles facilitate (dis)embodied cognition. We propose these as the first steps towards a cognitive audiovisual translatology.
References (61)
Aftanas, Ljubomir I., Natalia V. Lotova, vladimir I. Koshkarov, Victor P. Makhnev, Yuri N. Mordvintsev, and Serguei A. Popov. 1998. “Non-Linear Dynamic Complexity of the Human EEG during Evoked Emotions.” International Journal of Psychophysiology 28 (1): 63–76. 
Bilandzic, Helena, and Rick W. Busselle. 2011. “Enjoyment of Films as a Function of Narrative Experience, Perceived Realism and Transportability.” Communications 36 (1): 29–50.
Bos, Danny Oude. 2006. “EEG-based emotion recognition. The Influence of Visual and Auditory Stimuli.” Capita Selecta (MSc course), University of Twente.
Busselle, Rick W., and Helena Bilandzic. 2008a. “Measuring Narrative Engagement.” Media Psychology 12 (4): 321–347.
. 2008b. “Fictionality and Perceived Realism in Experiencing Stories: A Model of Narrative Comprehension and Engagement.” Communication Theory 18 (2): 255–280.
Carroll, Noel, and William P. Seeley. 2013. “Cognitivism, Psychology, and Neuroscience: Movies as Attentional Engines.” In Psychocinematics: Exploring cognition at the Movies, ed. by A.P. Shimamura, 53–75. Oxford: Oxford University Press.
Cho, Hyunyi, Lijiang Shen, and Kari Wilson. 2014. “Perceived Realism: Dimensions and Roles in Narrative Persuasion.” Communication Research 41 (6): 828–851.
Cohen, Jonathan. 2001. “Defining Identification: A Theoretical Look at the Identification of Audiences with Media Characters.” Mass Communication Society 4 (3): 245–264.
. 2006. “Audience Identification: A Theoretical Look at the Identification of Audiences with Characters.” In Psychology of entertainment, ed. by J. Bryant and P. Vorderer, 183–197. Mahwah, NJ: Lawrence Erlbaum.
D’Ydewalle, Géry, Caroline Praet, Karl Verfaillie, and Johan van Rensbergen. 1991. “Watching Subtitled Television: Automatic Reading Behavior.” Communication Research 18: 650–666.
Fozooni, Babak. 2006. “All Translators are Bastards!” South African Journal of Psychology 36 (2): 281–298.
Fryer, Louise, and Jonathan Freeman. 2012. “Cinematic Language and the Description of Film: Keeping AD Users in the Frame.” Perspectives: Studies in Translatology 21 (3): 412–426.
. 2013. “Visual Impairment and Presence: Measuring the Effect of Audio Description.” Proceedings of the
2013 Inputs-Outputs Conference: An Interdisciplinary Conference on Engagement in HCI and Performance. Brighton, United Kingdom, June 26–26.
Fryer, Louise, Linda Pring, and Jonathan Freeman. 2013. “Audio Drama and the Imagination: The Influence of Sound Effects on Presence in People with and without Sight.” Journal of Media Psychology: Theories, Methods, and Applications 25 (2): 65–71.
Gallese, Vittorio, and Michele Guerra. 2012. “Embodying Movies: Embodied Simulation and Film Studies.” Cinema: Journal of Philosophy and the Moving Image 3: 183–210.
Gallese, Vittorio. 2009. “Mirror Neurons, Embodied Simulation, and the Neural Basis of Social Identification.” Psychoanalytic Dialogue 19: 519–536.
García Mainar, Luis Miguel. 1993. “Auto-Focalisation in Film Narrative.” Atlantis 15 (1–2): 153–167.
Green, Melanie C., and Timothy C. Brock. 2000. “The Role of Transportation in the Persuasiveness of Public Narratives.” Journal of Personality and Social Psychology 79 (5): 701–721.
. 2013. “Transport Narrative Questionnaire.” Measurement Instrument Database for the Social Science. Retrieved from [URL].
Green, Melanie C., Timothy C. Brock, and Geoff F. Kaufman. 2004. “Understanding Media Enjoyment: The Role of Transportation into Narrative Worlds.” Communication theory 14 (4): 311–327.
Igartua Perosanz, Juan J. 2010. “Identification with Characters and Narrative Persuasion through Fictional Feature Films.” Communications. The European Journal of Communication Research 35 (4): 347–373.
Igartua Perosanz, Juan J., and Carlos Muñiz. 2008. “Identificación con los personajes y disfrute ante largometrajes de ficción. Una investigación empírica.” Comunicación y Sociedad 21 (1): 25–51.
Igartua Perosanz, Juan J., and Darío Páez Rovira. 1998. “Validez y fiabilidad de una escala de empatía e identificación con los personajes.” Psicothema 10 (2): 423–436.
Ivarsson, Jan, and Mary Carroll. 1998. Subtitling. Simrishamn, Sweden: TransEdit HB.
Jung, Tzyy-Ping, Scott Makeig, Colin Humphries, Te-Won Lee, Martin J. McKeown, Vicente Iragui, and Terrence J. Sejnowski. 2000. “Removing Electroencephalographic Artefacts by Blind Source Separation.” Psychophysiology, pp. 163–178.
Kim, T., and Biocca, F. 1997. “Telepresence via Television: Two Dimensions of Telepresence May Have Different Connections to Memory and Persuasion.” Journal of Computer-Mediated Communication 3 (2).
Koolstra, Cees, M., Allerd L. Peeters and Herman Spinhof. 2002. “The Pros and Cons of Dubbing and Subtitling.” European Journal of Communication 17 (3): 325–354.
Kruger, Jan-Louis. 2012. “Making Meaning in AVT: Eye Tracking and Viewer Construction of Narrative.” Perspectives: Studies in Translatology 20 (1): 67–86.
Levin, Daniel T., Alicia M. Hymel, and Lewis Baker. 2013. “Belief, Desire, Action and Other Stuff: Theory of Mind in Movies.” In Psychocinematics: Exploring Cognition at the Movies, ed. by A.P. Shimamura, 244–266. Oxford: Oxford University Press.
Lin, Yuan-Pin, Chi-Hong Wang, Tzyy-Ping, Jung, Tien-Lin Wu, Shyh-Kang Jeng, Jeng-Ren Duann, and Jyh-Horng Chen. 2010. “EEG-Based Emotion Recognition in Music Listening.” IEEE Transactions on Biomedical Engineering 57 (7): 1798–1806.
Lombard, Matthew, and Theresa Ditton. 1997. “At the Heart of It All: The Concept of Presence.” Journal of Computer-Mediated Communication 3 (2).
Lothe, Jakob. 2000. Narrative in Fiction and Film: An Introduction. Oxford: Oxford University Press.
Marian, Viorica, Henrike K. Blumenfeld, and Margarita Kaushanskaya. 2007. “The Language Experience and Proficiency Questionnaire (LEAP-Q): Assessing Language Profiles in Bilinguals and Multilinguals.” Journal of Speech Language and Hearing Research 50 (4): 940–967.
Mayer, Richard E. 2002. “Cognitive Theory and the Design of Multimedia Instruction: An Example of the Two-Way Street Between Cognition and Instruction.” New Directions for Teaching and Learning 89: 55–71.
McQuail, Denis, Jay G. Blumler, and J.R. Brown. 1972. The Television Audience: A Revised Perspective. In Sociology of Mass Communication, ed. by D. McQuail, 135–165. Middlesex: Penguin.
Miskovic, V., and L.A. Schmidt. 2010. “Cross-Regional Cortical Synchronization during Affective Image Viewing.” Brain Research 1362: 102–111.
Moyer-Gusé, Emily. 2008. “Toward a Theory of Entertainment Persuasion: Explaining the Persuasive Effects of Entertainment‐Education Messages.” Communication Theory 18 (3): 407–425.
Muñoz Martín, Ricardo. 2010. “On Paradigms and Cognitive Translatology.” In Translation and Cognition, ed. by M.S. Gregory Shreve and E. Angelone, 169–187. Amsterdam: John Benjamins.
Nie, Dan, Xiao-Wei Wang, Li-Chen Shi, and Bao-Liang Lu. 2011. “EEG-based Emotion Recognition during Watching Movies.” Proceedings of the
5th International IEEE EMBS Conference on Neural Engineering
, Cancun, Mexico, April 29–May 1, 667–670.
Perego, Elisa, Fabio Del Missier, and Sara Bottiroli. 2015. “Dubbing versus Subtitling in Young and Older Adults: Cognitive and Evaluative Aspects.” Perspectives: Studies in Translatology 23 (1): 1–21.
Perego, Elisa, Fabio Del Missier, Marco Porta, and Mauro Mosconi. 2010. “The Cognitive Effectiveness of Subtitle Processing.” Media Psychology 13 (3), 243–272.
Petrantonakis, Panagiotis C., and Leontios J. Hadjileontiadis. 2010. “Adaptive Extraction of Emotion-Related EEG Segments Using Multidimensional Directed Information in Time-Frequency Domain.”
32nd Annual International Conference of the IEEE EMBS
, Buenos Aires, Argentina, August 31–September 4 2010.
. 2011. “A Novel Emotion Elicitation Index Using Frontal Brain Asymmetry for Enhanced EEG-Based Emotion Recognition.” IEEE Transactions on Information Technology in Biomedicine 15 (5): 737–746.
Raney, Arthur A. 2002. “Moral Judgment as a Predictor of Enjoyment of Crime Drama.” Media Psychology 4 (4): 305–322.
Reiser, Eva M., Günter Schulter, Elizabeth M. Weiss, Andreas Fink, Christian Rominger, and Ilona Papousek. 2012. “Decrease of Prefrontal–Posterior EEG Coherence: Loose Control During Social–Emotional Stimulation.” Brain and Cognition 80: 144–154.
Schellberg, Dieter, Christoph Besthorn, Thomas Klos, and Theodor Gasser. 1990. “EEG Power and Coherence while Male Adults Watch Emotional Video Films.” International Journal of Psychophysiology 9: 279–291.
Sasamoto, Ryoko, and Stephen Doherty. 2015. “Towards the Optimal Use of Impact Captions on TV Programmes.” In Conflict and Communication, a Changing Asia in a Globalising World, ed. by M. O’Hagan and Q. Zhang, 210–247. Bremen: EHV Academic Press.
Shimamura, Arthur P. (ed). 2013. Psychocinematics: Exploring Cognition at the Movies. Oxford: Oxford University Press.
Soto Sanfiel, María T. 2015. “Engagement and Mobile Listening.” International Journal of Mobile Communication 13 (1): 29–50.
Soto Sanfiel, María T., Laura Aymerich Franch, and Francesc X. Ribes Guàrdia. 2010. “Impacto de la interactividad en la identificación con los personajes de las ficciones.” Psicothema 22 (4): 822–857.
Soto Sanfiel, María T., Laura Aymerich Franch, and Estrella Romero Triñanes. 2014. “Personality in Interaction: How the Big Five Relates to the Reception of Interactive Narratives”. Communication and Society 27 (3): 151–186.
Tal-Or, Nurit, and Jonathan Cohen. 2010. “Understanding Audience Involvement: Conceptualizing and Manipulating Identification and Transportation.” Poetics 38: 402–418.
Tversky, Barbara, and Bridgette Martin Hard. 2009. “Embodied and Disembodied Cognition: Spatial Perspective-Taking.” Cognition 110: 124–129.
Vorderer, Peter, Christoph Klimmt, and Ute Ritterfeld. 2004. “Enjoyment: At the Heart of Media Entertainment.” Communication Theory 14 (4): 388–408.
Voss, Christiane, 2008. “Fiktionale Immersion.” Montage AV 17 February 2008, [URL]
Weibel, David, Bartholomäus Wissmath, and Fred W. Mast. 2010. “Immersion in Mediated Environments: The Role of Personality Traits.” Cyberpsychology, Behavior, and Social Networking 13 (3): 251–256.
Wirth, Werner. 2006. “Involvement.” In Psychology of Entertainment, ed. by J. Bryant and P. Vorderer, 199–213. Mahwah, NJ: Lawrence Erlbaum.
Wissmath, Bartholomäus, David Weibel, and R. Groner. 2009. “Dubbing or subtitling? Effects on Presence, Transportation, Flow, and Enjoyment.” Journal of Media Psychology 21 (3): 114–125.
Wissmath, Bartholomäus, and David Weibel. 2012. “Translating Movies and the Sensation of ‘Being There’.” In Eye Tracking in Audiovisual Translation, ed. by E. Perego, 281–298. Roma: Aracne Editrice.
Witmer, Bob G., and Michael J. Singer. 1998. “Measuring Presence in Virtual Environments: A Presence Questionnaire.” Presence: Teleoperators and Virtual Environments 7: 225–240.
Cited by (22)
Cited by 22 other publications
Li, Jingjing
2024. Reception of audiovisual translation in China: a survey study. Perspectives ► pp. 1 ff. 
Tanrıkulu, Halid Bilal & Mahmut Ceran
2024. MEDYA ETKİLERİNE YÖNELİK METODOLOJİK BİR TARTIŞMA: KONVANSİYONEL YÖNTEMLER VE EEG. Moment Journal 11:1 ► pp. 109 ff.
de Lacerda Pataca, Caluã, Matthew Watkins, Roshan Peiris, Sooyeon Lee & Matt Huenerfauth
2023. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, ► pp. 1 ff.
Greco, Gian Maria, Anna Jankowska & Agnieszka Szarkowska
2022. Addressing methodological issues in experimental research in audiovisual translation. Translation Spaces 11:1 ► pp. 1 ff.
Van Hoecke, Senne M., Iris Schrijver & Isabelle S. Robert
2022. Preparing and comparing subtitles for quasi-experimental and experimental research in audiovisual translation
studies. Translation Spaces 11:1 ► pp. 113 ff.
Iturregui-Gallardo, Gonzalo & Anna Matamala
2021. Audio subtitling: dubbing and voice-over effects and their impact on user experience. Perspectives 29:1 ► pp. 64 ff.
Sun, Sanjun, Ricardo Muñoz Martín & Defeng Li
2021. Introduction: One More Step Forward—Cognitive Translation Studies at the Start of a New Decade. In Advances in Cognitive Translation Studies [New Frontiers in Translation Studies, ], ► pp. 1 ff.
Xiao, Kairong & Sandra L. Halverson
2021. Cognitive Translation and Interpreting Studies (CTIS). Cognitive Linguistic Studies 8:2 ► pp. 235 ff.
Deckert, Mikołaj
2020. Decision-Making: Putting AVT and MA into Perspective. In The Palgrave Handbook of Audiovisual Translation and Media Accessibility [Palgrave Studies in Translating and Interpreting, ], ► pp. 483 ff.
Massey, Gary & Peter Jud
2020. Translation Process Research in Audiovisual Translation. In The Palgrave Handbook of Audiovisual Translation and Media Accessibility [Palgrave Studies in Translating and Interpreting, ], ► pp. 359 ff.
Naranjo, Beatriz
Raffi, Francesca
2020. The Impact of Italian Dubbing on Viewers’ Immersive Experience: An Audience Reception Study. Online Journal of Communication and Media Technologies 10:3 ► pp. e202019 ff.
Naranjo Sánchez, Beatriz
2019. Immersed in the source text. Babel. Revue internationale de la traduction / International Journal of Translation 65:2 ► pp. 264 ff.
Doherty, Stephen
2018. Chapter 5. Analysing variable relationships and time-course data in eye-tracking studies of translation processes and products. In Eye Tracking and Multidisciplinary Studies on Translation [Benjamins Translation Library, 143], ► pp. 71 ff.
Doherty, Stephen & Jan-Louis Kruger
2018. Assessing Quality in Human- and Machine-Generated Subtitles and Captions. In Translation Quality Assessment [Machine Translation: Technologies and Applications, 1], ► pp. 179 ff.
Kruger, Jan-Louis & Stephen Doherty
2018. Triangulation of online and offline measures of processing and reception in AVT. In Reception Studies and Audiovisual Translation [Benjamins Translation Library, 141], ► pp. 91 ff.
Kruger, Jan-Louis, Stephen Doherty, Wendy Fox & Peter de Lissa
2018. Chapter 12. Multimodal measurement of cognitive load during subtitle processing. In Innovation and Expansion in Translation Process Research [American Translators Association Scholarly Monograph Series, XVIII], ► pp. 267 ff.
Rojo López, Ana María
Romero-Fresco, Pablo
2018. Reception studies in live and pre-recorded subtitles for the deaf and hard of hearing. In Reception Studies and Audiovisual Translation [Benjamins Translation Library, 141], ► pp. 199 ff.
Teixeira, Carlos S. C. & Sharon O'Brien
2018. Chapter 3. Overcoming methodological challenges of eye tracking in the translation workplace. In Eye Tracking and Multidisciplinary Studies on Translation [Benjamins Translation Library, 143], ► pp. 33 ff.
Kruger, Haidee & Jan‐Louis Kruger
2017. Cognition and Reception. In The Handbook of Translation and Cognition, ► pp. 71 ff.
Kruger, Jan-Louis, Stephen Doherty & María-T. Soto-Sanfiel
2017. Original language subtitles: Their effects on the native and foreign viewer. Comunicar 25:50 ► pp. 23 ff.
This list is based on CrossRef data as of 25 july 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.