Part of
Reembedding Translation Process ResearchEdited by Ricardo Muñoz Martín
[Benjamins Translation Library 128] 2016
► pp. 171–194
Edited by Ricardo Muñoz Martín
[Benjamins Translation Library 128] 2016
► pp. 171–194
Towards a cognitive audiovisual translatology
subtitles and embodied cognition
This chapter presents the findings of a study to investigate the impact of subtitles on the processing of audiovisual texts in terms of levels of self-reported engagement with the text. It also presents a methodology for investigating the neural processing of subtitles using electroencephalography (EEG) and psychometrics. By establishing the beta coherence between prefrontal and posterior regions, we can determine whether the effect of subtitles can be captured through EEG, and how this is related to psychological immersion. We therefore use EEG to investigate the cognitive processing of subtitled film in combination with subjective psychometrics. Initial validation of this methodology shows that adding same language subtitles results in statistically significantly higher levels of immersion and enjoyment, lending support to a view that subtitles facilitate (dis)embodied cognition. We propose these as the first steps towards a cognitive audiovisual translatology.
References
Aftanas, Ljubomir I., Natalia V. Lotova, vladimir I. Koshkarov, Victor P. Makhnev, Yuri N. Mordvintsev, and Serguei A. Popov
1998 “Non-Linear Dynamic Complexity of the Human EEG during Evoked Emotions.” International Journal of Psychophysiology 28 (1): 63–76. 
Bilandzic, Helena, and Rick W. Busselle
2011 “Enjoyment of Films as a Function of Narrative Experience, Perceived Realism and Transportability.” Communications 36 (1): 29–50.
Bos, Danny Oude
2006 “EEG-based emotion recognition. The Influence of Visual and Auditory Stimuli.” Capita Selecta (MSc course), University of Twente.
Busselle, Rick W., and Helena Bilandzic
2008a “Measuring Narrative Engagement.” Media Psychology 12 (4): 321–347.
2008b “Fictionality and Perceived Realism in Experiencing Stories: A Model of Narrative Comprehension and Engagement.” Communication Theory 18 (2): 255–280.
Carroll, Noel, and William P. Seeley
2013 “Cognitivism, Psychology, and Neuroscience: Movies as Attentional Engines.” In Psychocinematics: Exploring cognition at the Movies, ed. by A.P. Shimamura, 53–75. Oxford: Oxford University Press.
Cho, Hyunyi, Lijiang Shen, and Kari Wilson
2014 “Perceived Realism: Dimensions and Roles in Narrative Persuasion.” Communication Research 41 (6): 828–851.
Cohen, Jonathan
2001 “Defining Identification: A Theoretical Look at the Identification of Audiences with Media Characters.” Mass Communication Society 4 (3): 245–264.
2006 “Audience Identification: A Theoretical Look at the Identification of Audiences with Characters.” In Psychology of entertainment, ed. by J. Bryant and P. Vorderer, 183–197. Mahwah, NJ: Lawrence Erlbaum.
D’Ydewalle, Géry, Caroline Praet, Karl Verfaillie, and Johan van Rensbergen
1991 “Watching Subtitled Television: Automatic Reading Behavior.” Communication Research 18: 650–666.
Fozooni, Babak
2006 “All Translators are Bastards!” South African Journal of Psychology 36 (2): 281–298.
Fryer, Louise, and Jonathan Freeman
2012 “Cinematic Language and the Description of Film: Keeping AD Users in the Frame.” Perspectives: Studies in Translatology 21 (3): 412–426.
2013 “Visual Impairment and Presence: Measuring the Effect of Audio Description.” Proceedings of the
2013 Inputs-Outputs Conference: An Interdisciplinary Conference on Engagement in HCI and Performance. Brighton, United Kingdom, June 26–26.
Fryer, Louise, Linda Pring, and Jonathan Freeman
2013 “Audio Drama and the Imagination: The Influence of Sound Effects on Presence in People with and without Sight.” Journal of Media Psychology: Theories, Methods, and Applications 25 (2): 65–71.
Gallese, Vittorio, and Michele Guerra
2012 “Embodying Movies: Embodied Simulation and Film Studies.” Cinema: Journal of Philosophy and the Moving Image 3: 183–210.
Gallese, Vittorio
2009 “Mirror Neurons, Embodied Simulation, and the Neural Basis of Social Identification.” Psychoanalytic Dialogue 19: 519–536.
García Mainar, Luis Miguel
1993 “Auto-Focalisation in Film Narrative.” Atlantis 15 (1–2): 153–167.
Green, Melanie C., and Timothy C. Brock
2000 “The Role of Transportation in the Persuasiveness of Public Narratives.” Journal of Personality and Social Psychology 79 (5): 701–721.
2013 “Transport Narrative Questionnaire.” Measurement Instrument Database for the Social Science. Retrieved from [URL].
Green, Melanie C., Timothy C. Brock, and Geoff F. Kaufman
2004 “Understanding Media Enjoyment: The Role of Transportation into Narrative Worlds.” Communication theory 14 (4): 311–327.
Igartua Perosanz, Juan J
2010 “Identification with Characters and Narrative Persuasion through Fictional Feature Films.” Communications. The European Journal of Communication Research 35 (4): 347–373.
Igartua Perosanz, Juan J., and Carlos Muñiz
2008 “Identificación con los personajes y disfrute ante largometrajes de ficción. Una investigación empírica.” Comunicación y Sociedad 21 (1): 25–51.
Igartua Perosanz, Juan J., and Darío Páez Rovira
1998 “Validez y fiabilidad de una escala de empatía e identificación con los personajes.” Psicothema 10 (2): 423–436.
Ivarsson, Jan, and Mary Carroll
1998 Subtitling. Simrishamn, Sweden: TransEdit HB.
Jung, Tzyy-Ping, Scott Makeig, Colin Humphries, Te-Won Lee, Martin J. McKeown, Vicente Iragui, and Terrence J. Sejnowski
2000 “Removing Electroencephalographic Artefacts by Blind Source Separation.” Psychophysiology, pp. 163–178.
Kim, T., and Biocca, F
1997 “Telepresence via Television: Two Dimensions of Telepresence May Have Different Connections to Memory and Persuasion.” Journal of Computer-Mediated Communication 3 (2).
Koolstra, Cees, M., Allerd L. Peeters and Herman Spinhof
2002 “The Pros and Cons of Dubbing and Subtitling.” European Journal of Communication 17 (3): 325–354.
Kruger, Jan-Louis
2012 “Making Meaning in AVT: Eye Tracking and Viewer Construction of Narrative.” Perspectives: Studies in Translatology 20 (1): 67–86.
Levin, Daniel T., Alicia M. Hymel, and Lewis Baker
2013 “Belief, Desire, Action and Other Stuff: Theory of Mind in Movies.” In Psychocinematics: Exploring Cognition at the Movies, ed. by A.P. Shimamura, 244–266. Oxford: Oxford University Press.
Lin, Yuan-Pin, Chi-Hong Wang, Tzyy-Ping, Jung, Tien-Lin Wu, Shyh-Kang Jeng, Jeng-Ren Duann, and Jyh-Horng Chen
2010 “EEG-Based Emotion Recognition in Music Listening.” IEEE Transactions on Biomedical Engineering 57 (7): 1798–1806.
Lombard, Matthew, and Theresa Ditton
1997 “At the Heart of It All: The Concept of Presence.” Journal of Computer-Mediated Communication 3 (2).
Lothe, Jakob
2000 Narrative in Fiction and Film: An Introduction. Oxford: Oxford University Press.
Marian, Viorica, Henrike K. Blumenfeld, and Margarita Kaushanskaya
2007 “The Language Experience and Proficiency Questionnaire (LEAP-Q): Assessing Language Profiles in Bilinguals and Multilinguals.” Journal of Speech Language and Hearing Research 50 (4): 940–967.
Mayer, Richard E
2002 “Cognitive Theory and the Design of Multimedia Instruction: An Example of the Two-Way Street Between Cognition and Instruction.” New Directions for Teaching and Learning 89: 55–71.
McQuail, Denis, Jay G. Blumler, and J.R. Brown
1972 The Television Audience: A Revised Perspective. In Sociology of Mass Communication, ed. by D. McQuail, 135–165. Middlesex: Penguin.
Miskovic, V., and L.A. Schmidt
2010 “Cross-Regional Cortical Synchronization during Affective Image Viewing.” Brain Research 1362: 102–111.
Moyer-Gusé, Emily
2008 “Toward a Theory of Entertainment Persuasion: Explaining the Persuasive Effects of Entertainment‐Education Messages.” Communication Theory 18 (3): 407–425.
Muñoz Martín, Ricardo
2010 “On Paradigms and Cognitive Translatology.” In Translation and Cognition, ed. by M.S. Gregory Shreve and E. Angelone, 169–187. Amsterdam: John Benjamins.
Nie, Dan, Xiao-Wei Wang, Li-Chen Shi, and Bao-Liang Lu
2011 “EEG-based Emotion Recognition during Watching Movies.” Proceedings of the
5th International IEEE EMBS Conference on Neural Engineering
, Cancun, Mexico, April 29–May 1, 667–670.
Perego, Elisa, Fabio Del Missier, and Sara Bottiroli
2015 “Dubbing versus Subtitling in Young and Older Adults: Cognitive and Evaluative Aspects.” Perspectives: Studies in Translatology 23 (1): 1–21.
Perego, Elisa, Fabio Del Missier, Marco Porta, and Mauro Mosconi
2010 “The Cognitive Effectiveness of Subtitle Processing.” Media Psychology 13 (3), 243–272.
Petrantonakis, Panagiotis C., and Leontios J. Hadjileontiadis
2010 “Adaptive Extraction of Emotion-Related EEG Segments Using Multidimensional Directed Information in Time-Frequency Domain.”
32nd Annual International Conference of the IEEE EMBS
, Buenos Aires, Argentina, August 31–September 4 2010.
2011 “A Novel Emotion Elicitation Index Using Frontal Brain Asymmetry for Enhanced EEG-Based Emotion Recognition.” IEEE Transactions on Information Technology in Biomedicine 15 (5): 737–746.
Raney, Arthur A
2002 “Moral Judgment as a Predictor of Enjoyment of Crime Drama.” Media Psychology 4 (4): 305–322.
Reiser, Eva M., Günter Schulter, Elizabeth M. Weiss, Andreas Fink, Christian Rominger, and Ilona Papousek
2012 “Decrease of Prefrontal–Posterior EEG Coherence: Loose Control During Social–Emotional Stimulation.” Brain and Cognition 80: 144–154.
Schellberg, Dieter, Christoph Besthorn, Thomas Klos, and Theodor Gasser
1990 “EEG Power and Coherence while Male Adults Watch Emotional Video Films.” International Journal of Psychophysiology 9: 279–291.
Sasamoto, Ryoko, and Stephen Doherty
2015 “Towards the Optimal Use of Impact Captions on TV Programmes.” In Conflict and Communication, a Changing Asia in a Globalising World, ed. by M. O’Hagan and Q. Zhang, 210–247. Bremen: EHV Academic Press.
Shimamura, Arthur P
(ed) 2013 Psychocinematics: Exploring Cognition at the Movies. Oxford: Oxford University Press.
Soto Sanfiel, María T
2015 “Engagement and Mobile Listening.” International Journal of Mobile Communication 13 (1): 29–50.
Soto Sanfiel, María T., Laura Aymerich Franch, and Francesc X. Ribes Guàrdia
2010 “Impacto de la interactividad en la identificación con los personajes de las ficciones.” Psicothema 22 (4): 822–857.
Soto Sanfiel, María T., Laura Aymerich Franch, and Estrella Romero Triñanes
2014 “Personality in Interaction: How the Big Five Relates to the Reception of Interactive Narratives”. Communication and Society 27 (3): 151–186.
Tal-Or, Nurit, and Jonathan Cohen
2010 “Understanding Audience Involvement: Conceptualizing and Manipulating Identification and Transportation.” Poetics 38: 402–418.
Tversky, Barbara, and Bridgette Martin Hard
2009 “Embodied and Disembodied Cognition: Spatial Perspective-Taking.” Cognition 110: 124–129.
Vorderer, Peter, Christoph Klimmt, and Ute Ritterfeld
2004 “Enjoyment: At the Heart of Media Entertainment.” Communication Theory 14 (4): 388–408.
Voss, Christiane
Weibel, David, Bartholomäus Wissmath, and Fred W. Mast
2010 “Immersion in Mediated Environments: The Role of Personality Traits.” Cyberpsychology, Behavior, and Social Networking 13 (3): 251–256.
Wirth, Werner
2006 “Involvement.” In Psychology of Entertainment, ed. by J. Bryant and P. Vorderer, 199–213. Mahwah, NJ: Lawrence Erlbaum.
Wissmath, Bartholomäus, David Weibel, and R. Groner
2009 “Dubbing or subtitling? Effects on Presence, Transportation, Flow, and Enjoyment.” Journal of Media Psychology 21 (3): 114–125.
Wissmath, Bartholomäus, and David Weibel
2012 “Translating Movies and the Sensation of ‘Being There’.” In Eye Tracking in Audiovisual Translation, ed. by E. Perego, 281–298. Roma: Aracne Editrice.
Witmer, Bob G., and Michael J. Singer
1998 “Measuring Presence in Virtual Environments: A Presence Questionnaire.” Presence: Teleoperators and Virtual Environments 7: 225–240.
Cited by
Cited by 20 other publications
de Lacerda Pataca, Caluã, Matthew Watkins, Roshan Peiris, Sooyeon Lee & Matt Huenerfauth
2023. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, ► pp. 1 ff. 
Deckert, Mikołaj
2020. Decision-Making: Putting AVT and MA into Perspective. In The Palgrave Handbook of Audiovisual Translation and Media Accessibility [Palgrave Studies in Translating and Interpreting, ], ► pp. 483 ff.
Doherty, Stephen
2018. Chapter 5. Analysing variable relationships and time-course data in eye-tracking studies of translation processes and products. In Eye Tracking and Multidisciplinary Studies on Translation [Benjamins Translation Library, 143], ► pp. 71 ff.
Doherty, Stephen & Jan-Louis Kruger
2018. Assessing Quality in Human- and Machine-Generated Subtitles and Captions. In Translation Quality Assessment [Machine Translation: Technologies and Applications, 1], ► pp. 179 ff.
Greco, Gian Maria, Anna Jankowska & Agnieszka Szarkowska
2022. Addressing methodological issues in experimental research in audiovisual translation. Translation Spaces 11:1 ► pp. 1 ff.
Iturregui-Gallardo, Gonzalo & Anna Matamala
2021. Audio subtitling: dubbing and voice-over effects and their impact on user experience. Perspectives 29:1 ► pp. 64 ff.
Kruger, Haidee & Jan‐Louis Kruger
2017. Cognition and Reception. In The Handbook of Translation and Cognition, ► pp. 71 ff.
Kruger, Jan-Louis & Stephen Doherty
2018. Triangulation of online and offline measures of processing and reception in AVT. In Reception Studies and Audiovisual Translation [Benjamins Translation Library, 141], ► pp. 91 ff.
Kruger, Jan-Louis, Stephen Doherty, Wendy Fox & Peter de Lissa
2018. Chapter 12. Multimodal measurement of cognitive load during subtitle processing. In Innovation and Expansion in Translation Process Research [American Translators Association Scholarly Monograph Series, XVIII], ► pp. 267 ff.
Kruger, Jan-Louis , Stephen Doherty & María-T. Soto-Sanfiel
2017. Original language subtitles: Their effects on the native and foreign viewer. Comunicar 25:50 ► pp. 23 ff.
Massey, Gary & Peter Jud
2020. Translation Process Research in Audiovisual Translation. In The Palgrave Handbook of Audiovisual Translation and Media Accessibility [Palgrave Studies in Translating and Interpreting, ], ► pp. 359 ff.
Naranjo, Beatriz
Naranjo Sánchez, Beatriz
2019. Immersed in the source text. Babel. Revue internationale de la traduction / International Journal of Translation 65:2 ► pp. 264 ff.
Raffi, Francesca
2020. The Impact of Italian Dubbing on Viewers’ Immersive Experience: An Audience Reception Study. Online Journal of Communication and Media Technologies 10:3 ► pp. e202019 ff.
Rojo López, Ana María
Romero-Fresco, Pablo
2018. Reception studies in live and pre-recorded subtitles for the deaf and hard of hearing. In Reception Studies and Audiovisual Translation [Benjamins Translation Library, 141], ► pp. 199 ff.
Sun, Sanjun, Ricardo Muñoz Martín & Defeng Li
2021. Introduction: One More Step Forward—Cognitive Translation Studies at the Start of a New Decade. In Advances in Cognitive Translation Studies [New Frontiers in Translation Studies, ], ► pp. 1 ff.
Teixeira, Carlos S. C. & Sharon O'Brien
2018. Chapter 3. Overcoming methodological challenges of eye tracking in the translation workplace. In Eye Tracking and Multidisciplinary Studies on Translation [Benjamins Translation Library, 143], ► pp. 33 ff.
Van Hoecke, Senne M., Iris Schrijver & Isabelle S. Robert
2022. Preparing and comparing subtitles for quasi-experimental and experimental research in audiovisual translation
studies. Translation Spaces 11:1 ► pp. 113 ff.
Xiao, Kairong & Sandra L. Halverson
2021. Cognitive Translation and Interpreting Studies (CTIS). Cognitive Linguistic Studies 8:2 ► pp. 235 ff.
This list is based on CrossRef data as of 21 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.