Article In:
Translation and Interpreting at the Interface of Cognition and EmotionEdited by Ana María Rojo López and Purificación Meseguer Cutillas
[Translation, Cognition & Behavior 7:1] 2024
► pp. 23–48
Edited by Ana María Rojo López and Purificación Meseguer Cutillas
[Translation, Cognition & Behavior 7:1] 2024
► pp. 23–48
Measuring psychological immersion through cardiovascular response measures in subtitled films
This paper presents a conceptual replication of Kruger et al.’s (2017a) study, investigating the utility of heart rate (HR) and heart rate variability (HRV) indicators as objective measurements of psychological immersion in subtitled audiovisual products. The study employed a quasi-experimental design in ten participants viewing video clips with and without subtitles, while their physiological responses were monitored. Results suggest lower arousal levels in both conditions compared to baseline, with statistically significant differences observed in mean HR, mean RR, and RMSSD for the subtitles condition. While most inferential tests did not yield significant outcomes, descriptive trends indicate decreased arousal relative to the baseline. They also show increased arousal throughout the clip for most indicators and opposing trends for Mean HR, RMSSD and LF/HF between the two conditions. Methodological suggestions include using multiple data points for HR and HRV analysis, exploring ultra short-term measurements, selecting self-contained clips for viewing, and considering post-task baselines for a more accurate resting state representation. Despite limitations due to the small sample size, this study underscores the potential of HR and HRV as measures of immersion in translated audiovisual products, emphasising the need for larger sample sizes in future research to enhance statistical power and generalizability.
Keywords: subtitling, immersion, heart rate (HR), heart rate variability (HRV)
Article outline
- 1.Introduction
- 2.Psychological immersion
- 3.Materials and methods
- 3.1Participants
- 3.2Procedures
- 3.3Measures
- 3.4Analysis
- 4.Results and discussion
- 4.1Self-reports on immersion
- 4.2Descriptive results on HR and HRV
- 4.3HR and HRV intervals
- 4.4Scenes as a continuum
- 4.5English level of the participants
- 5.Discussion and suggestions
- 5.1Data points in audiovisual products
- 5.2HR and HRV ultra short-term measurements
- 5.3Materials
- 5.4Immersion and arousal
- 5.5Baseline
- 6.Conclusions
- Notes
- Author queries
-
References
This content is being prepared for publication; it may be subject to changes.
References (56)
Agrawal, Sarvesh, Adèle Simon, Søren Bech, Klaus Bærentsen, and Søren Forchhammer
2020 “Defining
Immersion: Literature Review and Implications for Research on Audiovisual
Experiences.” Journal of the Audio
Engineering
Society 68 (6): 404–417. 
Baek, Hyun J., Chul-Ho Cho, Jaegon Cho, and Jong-Min Woo
2015 “Reliability
of Ultra-Short-Term Analysis as a Surrogate of Standard 5-Min Analysis of
Heart Rate Variability.” Telemed J E
Health 211: 404–414.
Baños, Rosa, Cristina Botella, Mariano Alcañiz, Víctor Liaño, Belén Guerrer, and Beatriz Rey
2004 “Immersion
and Emotion: Their Impact on the Sense of
Presence.” Cyberpsychol.
Behav. 71: 734–741.
Castaldo, Rossana, Paolo Melillo, Umberto Bracale, Marco Caserta, Maria Triassi, and Leandro Pecchia
2015 “Acute
Mental Stress Assessment via Short Term HRV Analysis in Healthy Adults: A
Systematic Review with
Meta-Analysis.” Biomedical Signal Processing
and
Control 181: 370–377.
Chaume, Frederic
2018 “An
Overview of Audiovisual Translation: Four Methodological Turns in a Mature
Discipline.” Journal of Audiovisual
Translation 1 (1): 40–63.
Díaz Cintas, Jorge, and Agnieszka Szarkowska
2020 “Introduction:
Experimental Research in Audiovisual Translation — Cognition, Reception,
Production.” The Journal of Specialised
Translation (33): 3–16.
Di Lascio, Elena, Shkurta Gashi, and Silvia Santini
2018 “Unobtrusive
Assessment of Students’ Emotional Engagement during Lectures Using
Electrodermal Activity Sensors.” Proc. ACM
Interact. Mob. Wearable Ubiquitous
Technol. 2 (3): 1–21.
Fairclough, Stephen, Katie Ewing, and Jenna Roberts
2009 “Measuring
Task Engagement as an Input to Physiological
Computing.” Proceedings — 2009 3rd
International Conference on Affective Computing and Intelligent Interaction
and Workshops,
ACII, 1–9.
Flis, Gabriela, Adam Sikorski, and Agnieszka Szarkowska
2020 “Does
the Dubbing Effect Apply to Voice-Over? A Conceptual Replication Study on
Visual Attention and Immersion.” Journal of
Specialised
Translation (33): 41–69.
Fryer, Louise
2013 Putting
it into Words: The Impact of Visual Impairment on Perception, Experience and
Presence. PhD
thesis, Goldsmiths College University of London.
Fryer, Louise and Jonathan Freeman
2012 “Cinematic
Language and the Description of Film: Keeping AD Users in the
Frame.” Perspectives: Studies in
Translatology 21(3): 412–426.
Gambier, Yves, and Elena di Giovanni
Greco, Gian Maria, Anna Jankowska, and Agnieszka Szarkowska
2022 “Addressing
Methodological Issues in Experimental Research in Audiovisual
Translation.” Translation
Spaces 11 (1): 1–11.
Green, Melanie, and Timothy Brock
2013 Transport
narrative questionnaire. Measurement
Instrument Database for the Social
Science. Retrieved from [URL]
Iturregui-Gallardo, Gonzalo, and Anna Matamala
2021 “Audio
Subtitling: Dubbing and Voice-Over Effects and Their Impact on User
Experience.” Perspectives 29 (1): 64–83.
Jamovi
project
Jankowska, Anna, Joanna Pilarczyk, Kinga Wołoszyn, and Michał Kuniecki
2022 “Enough
is Enough: How Much Intonation is Needed in the Vocal Delivery of Audio
Description?” Perspectives 31 (4): 705–723.
Kim, Taeyong, and Frank Biocca
1997 “Telepresence
via Television: Two Dimensions of Telepresence May Have Different
Connections to Memory and
Persuasion.” Journal of Computer-Mediated
Communication 3 (2).
Kim, Hye-Geum, Eun-Jin Cheon, Dai-Seg Bai, Young Lee, and Bon-Hoon Koo
2018 “Stress
and Heart Rate Variability: A Meta-Analysis and Review of the
Literature.” Psychiatry
Investigation 15 (3): 235–245.
Kruger, Haidee, and Jan-Louis. Kruger
2017 “Cognition
and
Reception.” In The
Handbook of Translation and Cognition, edited
by Fabio Alves, and Arnt Jakobsen, 71–89. John Wiley & Sons.
Kruger, Jan-Louis, María Soto-Sanfiel, Stephen Doherty, and Ronny Ibrahim
2016 “Towards
a Cognitive Audiovisual Translatology: Subtitles and Embodied
Cognition”. In Reembedding
Translation Process Research, edited
by Ricardo Muñoz, 171–194. London: John Benjamins.
Kruger, Jan-Louis, Stephen Doherty, and Ronny Ibrahim
2017a “Electroencephalographic
Beta Coherence as an Objective Measure of Psychological Immersion in
Film.” Rivista Internazionale di Tecnica
della
Traduzione 191, 99–111.
Kruger, Jan-Louis, Stephen Doherty, and María Soto-Sanfiel
2017b “Original
Language Subtitles: Their Effects on the Native and Foreign
Viewer.” Comunicar 501, 23–32.
Laborde, Sylvain, Emma Mosley, and Julian Thayer
2017 “Heart
Rate Variability and Cardiac Vagal Tone in Psychophysiological Research —
Recommendations for Experiment Planning, Data Analysis, and Data
Reporting.” In Frontiers
in
Psychology 81: 213.
Lemma, Anna, Bethany Kok, and Tania Singer
2015 “Is
Meditation Always Relaxing? Investigating Heart Rate, Heart Rate
Variability, Experienced Effort and Likeability during Training of Three
Types of Meditation.” International Journal
of
Psychophysiology 97 (1): 38–45.
Lemhöfer, Kristin, and Mirjam Broersma
2012 “Introducing
LexTALE: A Quick and Valid Lexical Test for Advanced Learners of
English.” Behavior Research
Methods 44 (2), 325–343.
Lipponen, Jukka, and Mika Tarvainen
2019 “A
Robust Algorithm for Heart Rate Variability Time Series Artefact Correction
Using Novel Beat Classification.” Journal of
Medical Engineering &
Technology 43(3): 173–181.
Lohani, Monika, Brennan Payne, and David Strayer
2019 “A
Review of Psychophysiological Measures to Assess Cognitive States in
Real-World Driving.” Frontiers in Human
Neuroscience. 131: 57.
Loudon, Gareth, and Gina Deininger
2016 “The
Physiological Response During Divergent
Thinking.” Journal of Behavioral and Brain
Science 6 (1): 28–37.
McCarthy, Cameron, Nikhilesh Pradhan, Calum Redpath, and Andy Adler
2016 “Validation
of the Empatica E4 wristband.” 2016 IEEEEMBS
International Student Conference
(ISC), Ottawa, ON, Canada, 1–4.
Moses, Ziev, Linda Luecken, and Eason, James
2007 “Measuring
Task-Related Changes in Heart Rate
Variability.” 29th Annual International
Conference of the IEEE Engineering in Medicine and Biology
Society: 644–647. Lyon, France: IEEE.
Munoz M. Loreto, Arie van Roon, Harriëtte Riese, Chris Thio, Emma Oostenbroek, Iris Westrik, Eco de Geus, Ron Gansevoort, Joop Lefrandt, Ilja Nolte, and Harold Sneider
2015 “Validity
of (Ultra-) Short Recordings for Heart Rate Variability
Measurements.” PLoS
One 10 (9).
Nussinovitch, Udi, Keren Elishkevitz, Keren Katz, Moshe Nussinovitch, Shlomo Segev, Benjamin Volovitz, and Naomi Nussinovitch
2011 “Reliability
of Ultra-Short ECG Indices for Heart Rate
Variability.” Annals of Noninvasive
Electrocardiology 16 (2):117–22.
Olalla-Soler, Christian, Nicoletta Spinolo, and Ricardo Muñoz
2023 “Under
Pressure? A Study of Heart Rate and Heart-Rate Variability Using the
SmarTerp CAI tool.” HERMES — Journal of
Language and Communication in
Business (63): 119–142.
Orero, Pilar, Stephen Doherty, Jan-Louis Kruger, Anna Matamala, Jan Pedersen, Elisa Perego, Pablo Romero Fresco, Sara Rovira-Esteva, Olga Soler, and Agnieszka Szarkowska
2018 “Conducting
Experimental Research in Audiovisual Translation (AVT): A Position
Paper.” Journal of specialised
translation 301: 105–126. [URL]
Price, Mathew, Natasha Mehta, Erin Tone, and Page Anderson
2011 “Does
Engagement with Exposure Yield Better Outcomes? Components of Presence as a
Predictor of Treatment Response for Virtual Reality Exposure Therapy for
Social Phobia.” Journal of Anxiety
Disorder. 251: 763–770.
Raffi, Francesca
2020 “The
Impact of Italian Dubbing on Viewers’ Immersive Experience: An Audience
Reception Study.” Online Journal of
Communication and Media
Technologies 10 (3).
Ramos Caro, Marina
2015 “The
Emotional Experience of Films: Does Audio Description Make a
Difference?” Translator 21 (1): 68–94.
2016 “Testing
Audio Narration: The Emotional Impact of Language in Audio
Description.” Perspectives 24 (4): 606–634.
Ramos Caro, Marina, Laura Espín López, and Ana Rojo López
2021 “The
Psychophysiological Impact of Audio Described
Porn.” Onomázein Revista de Lingüística
Filología y
Traducción (8): 104–127.
Rojo Lopez, Ana, Marina Ramos Caro, and Laura Espin Lopez
2021 “Audio
Described vs. Audiovisual Porn: Cortisol, Heart Rate and Engagement in
Visually Impaired vs. Sighted
Participants.” Frontiers in
Psychology 121: 661452.
Rojo López, Ana, and Paweł Korpal
2020 “Through
Your Skin to Your Heart and Brain: A Critical Evaluation of Physiological
Methods in Cognitive Translation and Interpreting
Studies.” Linguistica Antverpiensia, New
Series — Themes in Translation
Studies 191: 191–217.
Rojo López, Ana, Paula Cifuentes-Férez, and Laura Espín López
2021 “The
Influence of Time Pressure on Translation Trainees’ Performance: Testing the
Relationship Between Self-Esteem, Salivary Cortisol and Subjective Stress
Response.” PLOS
ONE 16 (9): e0257727.
Romero Fresco, Pablo
2020 “The
Dubbing Effect: An Eye-Tracking Study on how Viewers Make Dubbing
Work.” The Journal of Specialised
Translation (33): 17–40.
Salahuddin, Lizawati, Jaegeol Cho, Myeong G. Jeong, and Desok Kim
2007 “Ultra
Short-Term Analysis of Heart Rate Variability for Monitoring Mental Stress
in Mobile Settings.” 29th Annual
International Conference of the IEEE Engineering in Medicine and Biology
Society: 4656–4659.
Schuurmans, Angela, Peter de Looff, Karin S. Nijhof, Catarina Rosada, Ron H. J. Scholte, Arne Popma, and Roy Otten
2020 “Validity
of the Empatica E4 wristband to Measure Heart Rate Variability
(HRV)Pparameters: A Comparison to Electrocardiography
(ECG).” Journal of Medical
Systems 44 (11): 190.
Shaffer, Fred, and J. P. Ginsberg
2017 “An
Overview of Heart Rate Variability Metrics and
Norms.” Frontiers in Public
Health 51: 258.
Shaffer, Fredric, Steven Shearman, and Zachary Meehan
2016 “The
Promise Of Ultra-Short-Term (UST) Heart Rate Variability
Measurements”. Biofeedback 441: 229–33.
Spinolo, Nicoletta, Christian Olalla-Soler, and Ricardo Muñoz
2022 “Finding
a Way Into an Interpreter’s Heart: Methodological Considerations on
Heart-Rate Variability Building on an Exploratory
Study.” Interpreters
Newsletter 271, 63–87.
Szarkowska, Agnieszka, and Julianna Boczkowska
2020 “Colour
Coding Subtitles in Multilingual Films — A Reception
Study.” Perspectives 30 (3): 520–536.
Tal-Or, Nurit, and Jonathan Cohen
2010 “Understanding
Audience Involvement: Conceptualizing and Manipulating Identification and
Transportation.” Poetics 381: 402–418.
Task Force of The European
Society of Cardiology and The North American Society of Pacing and
Electrophysiology
(1996) “Heart
Rate Variability.” Eur Heart
J 17/281.
Veltman, J. A. and Anthony Gaillard
1993 “Indices
of Mental Workload in a Complex Task
Environment.” Neuropsychobiology 28(1–2): 72–75.
Walczak, Agnieszka, and Louis Fryer
2017 “Creative
Description: The Impact of Audio Description Style on Presence in Visually
Impaired Audiences.” British Journal of
Visual
Impairment 35 (1): 6–17.
2017 “Vocal
Delivery of Audio Description by Genre: Measuring Users’
Presence.” Perspectives 261: 69–83.
Wissmath, Bartholomäus, David Weibel, and Rudolf Groner
2009 “Dubbing
or Subtitling?: Effects on Spatial Presence, Transportation, Flow, and
Enjoyment.” Journal of Media
Psychology 21 (3): 114–125. [URL].