Cover not available
Part of
Reception Studies and Audiovisual Translation
Edited by Elena Di Giovanni and Yves Gambier
[Benjamins Translation Library 141] 2018
► pp. 91–109

Chapter 5
Triangulation of online and offline measures of processing and reception in AVT

This chapter reviews methodologies for investigating the cognitive processing of audiovisual translation products using the triangulation of physiological, psychometric and performance measures. It also provides an analysis of the strengths and limitations of these measures to offer a synthesis of what has been achieved in this field and what remains to be developed. Such triangulation faces many challenges and rewards related to conducting interdisciplinary research that combines tightly-controlled experimental research with real-world, ecologically valid studies. Hence, the discussion emphasises the importance of both borrowing methods from other disciplines, and employing them effectively to add value to the study of film. We call for increased quality and integrity in AVT research to resolve questions on the cognitive processing and reception of AVT products.
Keywords: AVT, eye tracking, triangulation, EEG, post-hoc measures
Article outline
  • 1.Introduction
  • 2.Offline measures of reception and processing
    • 2.1Comprehension, recall and learning
    • 2.2Cognitive load
    • 2.3Immersion
  • 3.Online measures of reception and processing
    • 3.1Eye tracking
    • 3.2Electroencephalography
  • 4.Looking ahead
  • References
Published online: 15 June 2018
DOI logo
https://doi.org/10.1075/btl.141.06kru
References
Akahori, Wataru, Tatsunori Hirai, Shunya Kawamura, and Shigeo Morishima
2016, June. “Region-of-interest-based subtitle placement using eye-tracking data of multiple viewers”. In Proceedings of the Association for Computing Machinery International Conference on Interactive Experiences for TV and Online Video. Association for Computing Machinery, pp 123–128. doi:DOI logoGoogle Scholar
Antonenko, Pavlo, Fred Paas, and Roland Grabner
2010 “Using Electroencephalography to Measure Cognitive Load.” Educational Psychology Review 22: 425–438. doi:DOI logoGoogle Scholar
APA Publications and Communications Board Working Group on Journal Article Reporting Standards
2008 “Reporting standards for research in psychology: Why do we need them? What might they be?” The American Psychologist 63 (9): 839–851. DOI logoGoogle Scholar
Bird, Stephen and John Williams
2002 “The effect of bimodal input on implicit and explicit memory: An investigation into the benefits of within-language subtitling.” Applied Psycholinguistics 23 (4): 509–533. doi:DOI logoGoogle Scholar
Bisson, Marie-Josée, Walter Van Heuven, Kathy Conklin, and Richard Tunney
2014 “Processing of native and foreign language subtitles in films: An eye tracking study”. Applied Psycholinguistics 35 (02): 399–418. doi:DOI logoGoogle Scholar
Bos, Danny Oude
2006 “EEG-based emotion recognition.” The Influence of Visual and Auditory Stimuli 1–17.Google Scholar
Caffrey, Colm
2008a “Viewer perception of visual nonverbal cues in subtitled TV Anime”. European Journal of English Studies 12 (2): 163–178. doi:DOI logoGoogle Scholar
2008b “Using pupillometric, fixation-based and subjective measures to measure the processing effort experienced when viewing subtitled TV anime with pop-up gloss”. Copenhagen Studies in Language 36:125–144.Google Scholar
2009 Relevant abuse? Investigating the effects of an abusive subtitling procedure on the perception of TV anime using eye tracker and questionnaire (Doctoral dissertation, Dublin City University).Google Scholar
2012 “Using an eye–tracking tool to measure the effects of experimental subtitling procedures on viewer perception of subtitled AV content”. In Eye Tracking in Audiovisual Translation, E. Perego (ed), 223–258. Rome: Aracne.Google Scholar
Cambra, Cristina, Olivier Penacchio, Nuria Silvestre, and Aurora Leal
2014 “Visual attention to subtitles when viewing a cartoon by deaf and hearing children: An eye-tracking pilot study”. Perspectives: Studies in Translatology 22(4): 607–617. doi:DOI logoGoogle Scholar
Cho, Hyunyi., Lijang Shen and Kari Wilson
2014 Perceived realism: Dimensions and roles in narrative persuasion. Communication Research, 41(6), 828–851. doi:DOI logoGoogle Scholar
Cohen, Jonathan
2001 “Defining Identification: A Theoretical Look at the Identification of Audiences with Media Characters”. Mass Communication Society, 4(3): 245–264. doi: DOI logoGoogle Scholar
Creswell, John and Clark, V. Plano
2007 Designing and Conducting Mixed Methods Research. Thousand Oaks, California: Sage Publications.Google Scholar
Danan, Martine
2004 “Captioning and Subtitling: Undervalued Language Learning Strategies.” Meta: Translators’ Journal 49 (1): 67–77. doi:DOI logoGoogle Scholar
Doherty, Stephen
2016 “The impact of translation technologies on the process and product of translation”. International Journal of Communication 10: 947–969.Google Scholar
2017 “Issues in human and automatic translation quality assessment”. In Human Issues in Translation Technology, D. Kenny (ed), 131–186. London/New York: Routledge.Google Scholar
Doherty, Stephen and Sharon O’Brien
2014 “Assessing the usability of raw machine translated output: A user-centered study using eye tracking”. International Journal of Human-Computer Interaction 30 (1): 40–51. DOI logoGoogle Scholar
Doherty, Stephen and Jan-Louis Kruger
2018 “The development of eye tracking in empirical research on subtitling and captioning”. In Seeing into Screens: EyeTtracking and the Moving Image. J. Sita, T. Dwyer, S. Redmond, and C. Perkins. (eds). London: Bloomsbury. Chapter 3. pp 46–64.Google Scholar
Doherty, Stephen, Sharon O’Brien, and Michael Carl
2010 “Eye tracking as an MT evaluation technique”. Machine Translation 24 (1): 1–13. DOI logoGoogle Scholar
D’Ydewalle, Géry and Wim De Bruycker
2007 “Eye movements of children and adults while reading television subtitles”. European Psychologist 12 (3): 196–205. doi:DOI logoGoogle Scholar
D’Ydewalle, Géry, Caroline Praet, Karl Verfaillie, and Johan Van Rensbergen
1991 “Watching subtitled television automatic reading behaviour”. Communication Research 18(5): 650–666. DOI logoGoogle Scholar
D’Ydewalle, Géry, Johan Van Rensbergen, and Joris Pollet
1987 “Reading a message when the same message is available auditorily in another language: The case of subtitling”. In Eye movements from Physiology to Cognition, J. K. O’Regan and A. Levy-Schoen (eds), 313–321. Amsterdam: Elsevier. DOI logoGoogle Scholar
Fernández, Anna, Anna Matamala, and Anna Vilaró
2014 “The reception of subtitled colloquial language in Catalan: an eye-tracking exploratory study”. International Journal of Applied Linguistics 11: 63–80.Google Scholar
Fox, Wendy
2016 “Integrated titles: An improved viewing experience? In Eye tracking and Applied Linguistics, S. Hansen-Schirra and S. Grucza (eds), 5–30. Berlin: Language Science Press.Google Scholar
Fryer, Louise and Jonathan Freeman
2012 “Cinematic language and the description of film: Keeping AD users in the frame”. Perspectives: Studies in Translatology 21(3): 412–426. DOI logoGoogle Scholar
2013 “Visual impairment and presence: Measuring the effect of audio description”. In Proceedings of the 2013 Inputs-Outputs Conference: An Interdisciplinary Conference on Engagement in HCI and Performance. Brighton.DOI logo
2014 “Can you feel what I’m saying? The impact of verbal information on emotion elicitation and presence in people with a visual impairment. In Challenging Presence: Proceedings of the 15th International Conference on Presence, A. Felnhoferand O. D. Kothgassner (eds), 99–107. Wien: facultas.wuv.Google Scholar
Fryer, Louise, Linda Pring, and Jonathan Freeman
2013 “Audio drama and the imagination: The influence of sound effects on presence in people with and without sight”. Journal of Media Psychology: Theories, Methods, and Applications 25 (2): 65–71. doi:DOI logoGoogle Scholar
Garza, Thomas
1991 “Evaluating the use of captioned video materials in advanced foreign language Learning.” Foreign Language Annals 24 (3): 239–258. doi:DOI logoGoogle Scholar
1996 “The Message is the medium: Using video materials to facilitate foreign language performance.” Texas Papers in Foreign Language Education 2 (2): 1–20.Google Scholar
Gernsbacher, Morton Ann. A.
2015 “Video captions benefit everyone. Policy Insights from the Behavioral and Brain Sciences 2 (1): 195–202. DOI logoGoogle Scholar
Ghia, Elisa
2012 The impact of translation strategies on subtitle reading. In Eye tracking in audiovisual translation, ed by E. Perego, 155–182. Roma: Aracne EditriceGoogle Scholar
Green, Melanie and Timothy C. Brock
2000 The Role of Transportation in the Persuasiveness of Public Narratives. Journal of Personality and Social Psychology 79 (5): 701–721. DOI logoGoogle Scholar
Green, Melanie, Timothy C, Brock and Geoff Kaufman
2004 Understanding media enjoyment: The role of transportation into narrative worlds. Communication Theory 14 (4): 311–327. DOI logoGoogle Scholar
Hart, Sandra and Staveland, L. E. Lowell
1988 Development of a multi-dimensional workload rating scale: Results of empirical and theoretical research. In Human mental workload, P. A. Hancock and N. Meshkati (eds), 139–183. Amsterdam: Elsevier. DOI logoGoogle Scholar
Hefer, Esté
2011 Reading second language subtitles: a case study of South African viewers reading in their native language and L2-English (Doctoral dissertation, North-West University).Google Scholar
2013 “Reading second language subtitles: A case study of Afrikaans viewers reading in Afrikaans and English”. Perspectives: Studies in Translatology 21 (1): 22–41. DOI logoGoogle Scholar
Jensema, Carl, Ramalinga Sarma Danturthi, and Robert Burch
2000a “Time spent viewing captions on television programs”. American Annals of the Deaf 145 (5): 464–468. DOI logoGoogle Scholar
Jensema, Carl, Sameh El Sharkawy, Ramalinga Sarma Danturthi, Robert Burch, and D. Hsu
2000b “Eye movement patterns of captioned television viewers”. American Annals of the Deaf 145 (3): 275–285. DOI logoGoogle Scholar
Kim, Taeyong and Frank Biocca
1997 “Telepresence via television: Two dimensions of telepresence may have different connections to memory and persuasion”. Journal of Computer-Mediated Communication, 3(2). DOI logoGoogle Scholar
Kothari, Brij
2000 “Same language subtitling on Indian television: Harnessing the power of popular culture for literacy.” In Redeveloping Communication for Social Change: Theory, Practice, and Power, K. G. Wilkins (ed), 135–146. New York, USA: Rowman & Littlefield.Google Scholar
Krejtz, Izabela, Agnieszka Szarkowska, and Krzysztof Krejtz
2013 “The effects of shot changes on eye movements in subtitling”. Journal of Eye Movement Research 6 (5): 1–12.Google Scholar
Krejtz, Izabela, Agnieszka Szarkowska, and Maria Łogińska
2015 “Reading function and content words in subtitled videos”. Journal of Deaf Studies and Deaf Education, 21(2): 222–232. doi:DOI logoGoogle Scholar
Kruger, Jan-Louis
2010 “Audio narration: Re-narrativising film”. Perspectives: Studies in Translatology 18 (3): 231–249. DOI logoGoogle Scholar
2012 “Making meaning in AVT: Eye tracking and viewer construction of narrative”. Perspectives: Studies in Translatology 20 (1): 67–86. DOI logoGoogle Scholar
2013 “Subtitles in the classroom: Balancing the benefits of dual coding with the cost of increased cognitive load”. Journal for Language Teaching 47 (1): 29–53.Google Scholar
2016 “Psycholinguistics and audiovisual translation”. Target 28 (2): 276–287.DOI logoGoogle Scholar
Forthcoming). “Eye tracking in audiovisual translation research”. In The Routledge Handbook of Audiovisual Translation Studies, L. Perez-Gonzalez ed London/New York Routledge
Kruger, Jan-Louis and Stephen Doherty
2016 “Measuring cognitive load in the presence of educational video: Towards a multimodal methodology”. Australasian Journal of Educational Technology 32 (4): 19–31.Google Scholar
Kruger, Jan-Louis, Stephen Doherty, Wendy Fox, and Pete de Lissa
2017 “Multimodal measurement of cognitive load during subtitle processing: Same-language subtitles for foreign language viewers”. In Innovation and Expansion in Translation Process Research, I. Lacruz and R. Jääskeläinen (eds). Amsterdam/Philadelphia: John Benjamins. Chapter 12. pp.267–294. doi:DOI logoGoogle Scholar
Kruger, Jan-Louis, Esté Hefer, and Gordon Matthew
2013 Measuring the impact of subtitles on cognitive load: Eye tracking and dynamic audiovisual texts. In Proceedings of the 2013 Conference on Eye Tracking South Africa. Association for Computing Machinery (pp.62–66).
Kruger, Jan-Louis, Maria Soto-Sanfiel, and Stephen Doherty
2017 “Original language subtitles: Their effects on the native and foreign viewer”. Comunicar 25 (50): 23–32.DOI logoGoogle Scholar
Kruger, Jan-Louis, Maria Soto-Sanfiel, Stephen Doherty, and Ronny Ibrahim
2016 “Towards a cognitive audiovisual translatology: Subtitles and embodied cognition”. In Reembedding Translation Process Research, R. Muñoz (ed), 171–194. Amsterdam/Philadelphia: John Benjamins.DOI logoGoogle Scholar
Kruger, Jan-Louis, and Faans Steyn
2014 “Subtitles and eye tracking: Reading and performance”. Reading Research Quarterly 49 (1): 105–120. doi:DOI logoGoogle Scholar
Kruger, Jan-Louis, Agnieszka Szarkowska, and Izabella Krejtz
2015 “Subtitles on the moving image: An overview of eye tracking studies”. Refractory: A Journal of Entertainment Media 25: no page numbers.Google Scholar
Künzli, Alexander and Maureen Ehrensberger-Dow
2011 Innovative subtitling: A reception study. In Methods and Strategies of Process Research: Integrative Approaches in Translation Studies, C. Alvstad, A. Hild, and E. Tiselius, 187–200. Amsterdam/Philadelphia: John Benjamins. DOI logoGoogle Scholar
Leppink, Jimmie, Fred Paas, Cees Van der Vleuten, Tamara Van Gog, and Jeroen Van Merrienboer
2013 “Development of an instrument for measuring different types of cognitive load”. Behavior Research Methods 45: 1058–1072. doi:DOI logoGoogle Scholar
2014 “Effects of pairs of problems and examples on task performance and different types of cognitive load”. Learning and Instruction 30: 32–42. doi:DOI logoGoogle Scholar
Lin, Yuan-Pin, Chi-Hong Wang, Tzyy-Ping Jungm Tien-Lin Wu, Shyh-Kang Jeng, Jeng-Ren Duann, and Jyh-Horng Chen
2010 “EEG-Based emotion recognition in music listening.” IEEE Transactions on Biomedical Engineering 57(7): 1798–1806. doi:DOI logoGoogle Scholar
Linebarger, Deborah, Jessica Piotrowski, and Charles Greenwood
2010 “Onscreen print: The role of captions as a supplemental literacy tool”. Journal of Research in Reading 33 (2): 148–167. doi:DOI logoGoogle Scholar
Mäkisalo, Juha Lån, Tersia Gowases, and Sami Pietinen
2013 “Using eye tracking to study the effect of badly synchronized subtitles on the gaze paths of television viewers. New Voices in Translation Studies 10: 72–86.Google Scholar
Markham, Paul
1999 “Captioned video-tapes and second language listening word recognition”. Foreign Language Annals 32 (3): 321–328. doi:DOI logoGoogle Scholar
Maszerowska, Anna
2013 “Language without words: Light and contrast in audio description”. The Journal of Specialised Translation 20: 165–180.Google Scholar
Moran, Siobhan
2012 “The effect of linguistic variation on subtitle reception”. In Eye Tracking in Audiovisual Translation, E. Perego (ed), 183–222. Roma: Aracne.Google Scholar
Nie, Dan, Xiao-Wei Wang, Li-Chen Shi, and Bao-Liang Lu.
2011 “EEG-based Emotion Recognition during Watching Movies.” Proceedings of the 5th International IEEE EMBS Conference on Neural Engineering, Cancun, Mexico, April 29–May 1, 667–670. doi:DOI logo
Nilsson, Niels Christian, Rolf Nordahl, and Stefania Serafin
2016 Immersion revisited: A review of existing definitions of immersion and their relation to different theories of presence. Human Technology 12(2): 108–134, doi:DOI logoGoogle Scholar
Paas, Fred, Jeroen Van Merriënboer, and Jos Adam
1994 “Measurement of cognitive load in instructional research”. Perceptual and Motor Skills 79 (1): 419–430, doi:DOI logoGoogle Scholar
Paas, Fred, Juhani Tuovinen, Huib Tabbers, and Pascal Van Gerven
2003 “Cognitive load measurement as a means to advance cognitive load theory”. Educational Psychologist 38 (1): 63–71. doi:DOI logoGoogle Scholar
Perego, Elisa, Fabio Del Missier, Marco Porta, and Mauro Mosconi
2010 “The cognitive effectiveness of subtitle processing”. Media Psychology 13 (3): 243–272. doi:DOI logoGoogle Scholar
Petrantonakis, Panagiotis C. and Leontios J. Hadjileontiadis
2010 “Adaptive extraction of emotion-related EEG segments using multidimensional directed information in time-frequency domain.” 32nd Annual International Conference of the IEEE EMBS, Buenos Aires, Argentine, August 31–September 4 2010. doi:DOI logo
Petrantonakis, Panagiotis C. and Leontios J. Hadjileontiadis
2011 “A novel emotion elicitation index using frontal brain asymmetry for enhanced EEG-based emotion recognition”. IEEE Transactions on Information Technology in Biomedicine 15(5): 737–746. doi: DOI logoGoogle Scholar
Plass, Jan, Roxana Moreno, and Roland Brünken
2010 Cognitive load theory. London: Cambridge University Press. DOI logoGoogle Scholar
Rajendran, Dhevi, J., Andrew T. Duchowski, Pilar Orero, Juan Martínez, and Pablo Romero-Fresco
2013 “Effects of text chunking on subtitling: A quantitative and qualitative examination”. Perspectives: Studies in Translatology 21 (1): 5–21. doi:DOI logoGoogle Scholar
Reiser, Eva M., Günter Schulter, Elizabeth M. Weiss, Andreas Fink, Christian Rominger, and Ilona Papousek
2012 “Decrease of prefrontal–posterior EEG coherence: Loose control during social–emotional stimulation.” Brain and Cognition 80: 144–154. doi: DOI logoGoogle Scholar
Romero-Fresco, Pablo
2009 “More haste less speed: Edited versus verbatim respoken subtitles”. Vigo International Journal of Applied Linguistics 6 (1): 109–133.Google Scholar
2010 “Standing on quicksand: Hearing viewers’ comprehension and reading patterns of respoken subtitles for the news”. In New Insights into Audiovisual Translation and Media Accessibility, J. Diaz-Cintas, A. Matamala, and J. Neves (eds), 175–194. Amsterdam: Rodopi.Google Scholar
Romero-Fresco, Pablo and Louise Fryer
2013 “Could audio-described films benefit from audio introductions? An audience response study”. Journal of Visual Impairment and Blindness 107( 4): 287.Google Scholar
Sasamoto, Ryoko and Stephen Doherty
2016 “Towards the optimal use of impact captions on TV programmes”. In Conflict and communication: A changing Asia in a globalising world, M. O’Hagan, and Q. Zhang (eds), 210–247. Hauppauge, NY: Nova Science Publishers.Google Scholar
Sasamoto, Ryoko, Minako O’Hagan, and Stephen Doherty
2016 “Telop, affect and media design: A multimodal analysis of a Japanese TV programme”. Television and New Media: no page numbers. doi:DOI logoGoogle Scholar
Shimamura, Arthur
2013 Psychocinematics: Exploring cognition at the movies. Oxford: Oxford University Press. DOI logoGoogle Scholar
Specker, Elizabeth
2008 “L1/L2 eye movement reading of closed captioning: A multimodal analysis of multimodal use”. (Doctoral dissertation, University of Arizona).Google Scholar
Sweller, John, Paul Ayres, and Slava Kalyuga
2011 Cognitive load theory. New York, NY: Springer. doi: DOI logoGoogle Scholar
Szarkowska, Agnieszka, Izabela Krejtz, Zuzanna Klyszejko, and Anna Wieczorek
2011 “Verbatim, standard, or edited? Reading patterns of different captioning styles among deaf, hard of hearing, and hearing viewers”. American Annals of the Deaf 156 (4): 363–378. doi: DOI:DOI logoGoogle Scholar
Szarkowska, Agnieszka, Izabela Krejtz, Olga Pilipczuk, Łukasz Dutka, and Jan-Louis Kruger
2016 “The effects of text editing and subtitle presentation rate on the comprehension and reading patterns of interlingual and intralingual subtitles among deaf, hard of hearing and hearing viewers”. Across Languages and Cultures 17 (2): 183–204.DOI logoGoogle Scholar
Tal-Or, Nurit and Jonathan Cohen
2010 “Understanding audience involvement: Conceptualizing and manipulating identification and transportation”. Poetics 38 (4): 402–418. doi: DOI logoGoogle Scholar
Vanderplank, Robert
1988 “The value of teletext sub-titles in language learning”. ELT Journal 42 (4): 272–281. doi:DOI logoGoogle Scholar
1990 “Paying attention to the words: Practical and theoretical problems in watching television programmes with uni-lingual (CEEFAX) sub-titles”. System 18 (2): 221–234. doi:DOI logoGoogle Scholar
2010 “Déjà vu? A decade of research on language laboratories, television and video in language learning”. Language Teaching 43 (1): 1–37. doi:DOI logoGoogle Scholar
Weibel, David, Bartholomäus Wissmath, and Fred Mast
2010 Immersion in mediated environments: The role of personality traits. Cyberpsychology, Behavior, and Social Networking 13 (3): 251–256. doi:DOI logoGoogle Scholar
Winke, Paula, Susan Gass, and Tetyana Sydorenko
2013 “Factors influencing the use of captions by foreign language learners: An eye-tracking study”. The Modern Language Journal 97 (1): 254–275. doi:DOI logoGoogle Scholar
Wissmath, Bartholomäus and David Weibel
2012 “Translating movies and the sensation of being there”. In Eye Tracking in Audiovisual Translation, E. Perego (ed), 281–298. Roma: Aracne.Google Scholar
Wissmath, Bartholomäus, David Weibel, and Rudolf Groner
2009 “Dubbing or subtitling? Effects on spatial presence, transportation, flow, and enjoyment”. Journal of Media Psychology 21 (3): 114–125. doi:DOI logoGoogle Scholar
Cited by

Cited by 8 other publications

Chen, Yi-Chiao
2023. Foreignness as a border-crossing challenge. FORUM. Revue internationale d’interprétation et de traduction / International Journal of Interpretation and Translation 21:1 ► pp. 1 ff. DOI logo
Doherty, Stephen
2021. Introduction. Translation, Cognition & Behavior 4:2 ► pp. 244 ff. DOI logo
Federici, Federico M. & Callum Walker
2018. Chapter 2. A mapping exercise. In Eye Tracking and Multidisciplinary Studies on Translation [Benjamins Translation Library, 143], ► pp. 11 ff. DOI logo
Raffi, Francesca
2020. The Impact of Italian Dubbing on Viewers’ Immersive Experience: An Audience Reception Study. Online Journal of Communication and Media Technologies 10:3 ► pp. e202019 ff. DOI logo
Raffi, Francesca
2023. Accessibility and reception studies at the Macerata Opera Festival. Translation and Translanguaging in Multilingual Contexts 9:3 ► pp. 398 ff. DOI logo
Silva, Breno B., David Orrego-Carmona & Agnieszka Szarkowska
2022. Using linear mixed models to analyze data from eye-tracking research on subtitling. Translation Spaces 11:1 ► pp. 60 ff. DOI logo
Szarkowska, Agnieszka & Julianna Boczkowska
2022. Colour coding subtitles in multilingual films – a reception study. Perspectives 30:3 ► pp. 520 ff. DOI logo
Wu, Zhiwei & Zhuojia Chen
2021. A systematic review of experimental research in audiovisual translation 1992–2020. Translation, Cognition & Behavior 4:2 ► pp. 281 ff. DOI logo

This list is based on CrossRef data as of 18 november 2023. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.