Establishing a theoretical framework for AVT research: The importance of cognitive models

Kruger, Jan-Louis; Liao, Sixin

doi:10.1075/ts.21024.kru

Article published In:

Methodological Issues in Experimental Research in Audiovisual Translation and Media Accessibility
Edited by Gian Maria Greco, Anna Jankowska and Agnieszka Szarkowska
[Translation Spaces 11:1] 2022
► pp. 12–37

Establishing a theoretical framework for AVT research

The importance of cognitive models

Jan-Louis Kruger | Macquarie University | North-West University

Sixin Liao | Macquarie University

Empirical research on the cognitive processing of audiovisual translation (AVT) products has been thriving over the past decade. While the use of cutting-edge experimental tools such as eye trackers has drawn increasing scholarly attention and accelerated the progress in understanding the complex mental processes involved in the reception of multimodal AVT products, relatively less attention has been devoted to the importance of establishing a theoretical framework or cognitive model that can explain and predict the behaviours observed in empirical experiments. By reviewing numerous theories or cognitive models relevant to AVT research in explaining how different perceptual and cognitive systems operate for understanding multimodal products, this paper calls for engagement with these theoretical frameworks and models to work towards a robust model that can generate testable hypotheses for the integration and interaction of multiple sources of information involved in the processing of AVT or other multimodal products.

Keywords: audiovisual translation (AVT), cognitive models, multimodal integrated language framework (MILF), subtitles

Article outline

Introduction
Memory and cognition
- Modal model – Atkinson and Shiffrin (1968)
- Model of working memory – Baddeley and Hitch (1974); Baddeley (2000, 2010, 2012)
Multimodality
- Cognitive theory of multimedia learning (CTML) – Mayer (2014)
  - Dual-channel assumption
  - Limited capacity assumption
  - Active processing assumption
- Integrated model of text and picture comprehension (ITPC) – Schnotz and Bannert (2003)
- Scene perception and event comprehension theory (SPECT) – Loschky et al. (2020)
  - Front-end cognitive processes
  - Back-end cognitive processes
  - Dynamic visual narratives
Models in reading research
- The E-Z Reader model – Reichle et al. (2012)
- The Über-Reader model – Reichle (2021)
- Multimodal integrated-language framework (MILF) – Liao et al. (2021a)
Concluding remarks
Acknowledgement
Notes
References

Published online: 19 July 2022

https://doi.org/10.1075/ts.21024.kru

References

Allen, Richard J., Graham J. Hitch, Judit Mate, and Alan D. Baddeley

2012 “Feature Binding and Attention in Working Memory: A Resolution of Previous Contradictory Findings.” Quarterly Journal of Experimental Psychology 65 (12): 2369–83.

Ans, Bernard, Serge Carbonnel, and Sylviane Valdois

1998 “A Connectionist Multiple-Trace Memory Model for Polysyllabic Word Reading.” Psychological Review 105 (4): 678–723.

Atkinson, Richard. C., and Richard M. Shiffrin

1968 “Human Memory: A Proposed System and Its Control Processes.” In The Psychology of Learning and Motivation: Advances in Research and Theory, edited by Spence, Kenneth W., and Janet Taylor Spence, 89–195. New York: Academic Press.

Baddeley, Alan David

2000 “The Episodic Buffer: A New Component of Working Memory?” Trends in Cognitive Sciences 4 (11): 417–23.

2010 “Working memory.” Current biology 20 (4), 136–140.

2012 “Working Memory: Theories, Models, and Controversies.” Annual Review of Psychology 63 (1): 1–29.

Baddeley, Alan David, and Graham J. Hitch

1974 “Working memory.” In The Psychology of Learning and Motivation: Advances in Research and Theory, edited by Bower, Gordon H., 47–89. New York: Academic Press.

Baddeley, Alan David, Richard J. Allen, and Graham J. Hitch

2011 “Binding in Visual Working Memory: The Role of the Episodic Buffer.” Neuropsychologia 49 (6): 1393–1400.

Chung, Jing-mei

1999 “The Effects of Using Video Texts Supported with Advance Organizers and Captions on Chinese College Students’ Listening Comprehension: An Empirical Study.” Foreign Language Annals 32 (3): 295–308.

Danan, Martine

2004 “Captioning and Subtitling: Undervalued Language Learning Strategies.” Meta 49 (1): 67–77.

de Pedro Ricoy, Raquel

2012 “Multimodality in Translation: Steps towards Socially Useful Research.” Journal Multimodal Communication 1 (2): 181–203.

Díaz Cintas, Jorge, and Agnieszka Szarkowska

2020 “Introduction: Experimental Research in Audiovisual Translation–Cognition, Reception, Production.” The Journal of Specialised Translation, 331, 3–16.

Forstmann, Birte U., Eric-Jan Wagenmakers, Tom Eichele, Scott Brown, and John T. Serences

2011 “Reciprocal Relations between Cognitive Neuroscience and Formal Cognitive Models: Opposites Attract?” Trends in Cognitive Sciences 15 (6): 272–79.

Gernsbacher, Morton Ann

2015 “Video Captions Benefit Everyone.” Edited by Susan T. Fiske. Policy Insights from the Behavioral and Brain Sciences 2 (1): 195–202.

Hintzman, Douglas L.

1984 “MINERVA 2: A Simulation Model of Human Memory.” Behavior Research Methods, Instruments, & Computers 16 (2): 96–101.

Kaindl, Klaus

2013 “Multimodality and Translation.” In The Routledge Handbook of Translation Studies, edited by Millán, Carmen, and Francesca Bartrina, 257–269. London: Routledge.

Kalyuga, Slava, and John Sweller

2014 “The Redundancy Principle in Multimedia Learning.” In The Cambridge Handbook of Multimedia Learning, edited by Mayer, Richard E., 247–262. New York: Cambridge University Press.

Ketola, Anne

2016 “Towards a Multimodally Oriented Theory of Translation: A Cognitive Framework for the Translation of Illustrated Technical Texts.” Translation Studies 9 (1): 67–81.

Kruger, Jan-Louis

2016 “Psycholinguistics and Audiovisual Translation.” Target. International Journal of Translation Studies 28 (2): 276–87.

2019 “Eye tracking in Audiovisual Translation Research.” In The Routledge Handbook of Audiovisual Translation, edited by Pérez-González, Luis, 350–366. London: Routledge, Taylor and Francis Group.

2020 “Translation, Multimodality and Cognition.” In The Routledge Handbook of Translation and Cognition, edited by Alves, Fabio, and Arnt Lykke Jakobsen, 433–444. London, UK: Routledge.

Kruger, Jan-Louis, and Faans Steyn

2014 “Subtitles and Eye Tracking: Reading and Performance.” Reading Research Quarterly 49 (1): 105–20.

Kruger, Jan-Louis, Natalia Wisniewska, and Sixin Liao

2021 “Why Subtitle Speed Matters: Evidence from Word Skipping and Rereading.” Applied Psycholinguistics 43 (1): 211–36.

Liao, Sixin, and Jan-Louis Kruger

2022 “Cognitive Processing of Subtitles: Charting the Future by Mapping the Past.” In The Routledge Handbook of Translation, Interpreting, and Bilingualism, edited by Ferreira, Aline, and John W. Schwieter.

Liao, Sixin, Lili Yu, Jan-Louis Kruger, and Erik D. Reichle

2021a “The impact of Audio on the Reading of Inter- vs. Intra-lingual Subtitles: Evidence from Eye Movements.” Applied Psycholinguistics 43 (1): 237–69.

Liao, Sixin, Lili Yu, Erik D. Reichle, and Jan-Louis Kruger

2021b “Using Eye Movements to Study the Reading of Subtitles in Video.” Scientific Studies of Reading 25(5): 417–435.

Loschky, Lester C., Adam M. Larson, Tim J. Smith, and Joseph P. Magliano

2020 “The Scene Perception & Event Comprehension Theory (SPECT) Applied to Visual Narratives.” Topics in Cognitive Science 12 (1): 311–51.

Lum, Jarrad A. G., Gina Conti-Ramsden, Debra Page, and Michael T. Ullman

2012 “Working, Declarative and Procedural Memory in Specific Language Impairment.” Cortex 48 (9): 1138–54.

Mayer, Richard E.

2014 The Cambridge Handbook of Multimedia Learning. New York: Cambridge University Press.

Paivio, Allan

1986 Mental Representations: A Dual Coding Approach. Oxford, England: Oxford University Press.

1991 “Dual Coding Theory: Retrospect and Current Status.” Canadian Journal of Psychology 45 (3): 255–87.

2007 Mind and Its Evolution: A Dual Coding Theoretical Approach. Mahwah, NJ: Erlbaum.

Perego, Elisa, Fabio Del Missier, Marco Porta, and Mauro Mosconi

2010 “The Cognitive Effectiveness of Subtitle Processing.” Media Psychology 13 (3): 243–72.

Pérez-González, Luis

2014 “Multimodality in Translation and Interpreting Studies: Theoretical and Methodological Perspectives.” In A Companion to Translation Studies, edited by Bermann, Sandra, and Catherine Porter, 119–131. Malden, MA/Oxford, England: Wiley-Blackwell.

Prezenski, Sabine, André Brechmann, Susann Wolff, and Nele Russwinkel

2017 “A Cognitive Modeling Approach to Strategy Formation in Dynamic Decision Making.” Frontiers in Psychology 81 (August): 1–18.

Reichle, Erik D.

2021 Computational Models of Reading: A Handbook. Oxford: Oxford University Press.

Reichle, Erik D., Alexander Pollatsek, and Keith Rayner

2012 “Using E-Z Reader to Simulate Eye Movements in Nonreading Tasks: A Unified Framework for Understanding the Eye–Mind Link.” Psychological Review 119 (1): 155–85.

Reichle, Erik D., Lili Yu, Sixin Liao, and Jan-Louis Kruger

2021 “Using Simulations to Understand the Reading of Rapidly Displayed Subtitles.” Proceedings of the 43rd Annual Conference of the Cognitive Science Society.

Schnotz, Wolfgang

2005 “Integrated Model of Text and Picture Comprehension.” In The Cambridge Handbook of Multimedia Learning, edited by Mayer, Richard E., 49–70. New York: Cambridge University Press.

Schnotz, Wolfgang, and Maria Bannert

2003 “Construction and Interference in Learning from Multiple Representation.” Learning and Instruction 13 (2): 141–56.

Schnotz, Wolfgang

2014 “Integrated model of text and picture comprehension.” In The Cambridge Handbook of Multimedia Learning (2nd ed), edited by Mayer, Richard E., 72–103.

Schnotz, Wolfgang, and Inga Wagner

2018 “Construction and Elaboration of Mental Models through Strategic Conjoint Processing of Text and Pictures.” Journal of Educational Psychology 110 (6): 850–63.

Seeber, Kilian G.

2017 Multimodal processing in simultaneous interpreting. In The Handbook of Translation and Cognition, edited by Schwieter, John and Aline Ferreira, 461–475. Joh Wiley & Sons.

Squire, Larry R., and Stuart M. Zola

1996 “Structure and Function of Declarative and Nondeclarative Memory Systems.” Proceedings of the National Academy of Sciences 93 (24): 13515–22.

Szarkowska, Agnieszka, and Piotr Wasylczyk

2018 “Five Things You Wanted to Know about Audiovisual Translation Research, but Were Afraid to Ask.” Journal of Audiovisual Translation 1 (1): 8–25.

Szarkowska, Agnieszka, Breno Silva, and David Orrego-Carmona

2021 “Effects of Subtitle Speed on Proportional Reading Time.” Translation, Cognition & Behavior.

Taylor, Christopher

2016 “The Multimodal Approach in Audiovisual Translation.” Target. International Journal of Translation Studies 28 (2): 222–36.

Tuominen, Tiina, Catalina Jiménez Hurtado, and Anne Ketola

2018 “Why Methods Matter: Approaching Multimodality in Translation Research.” Linguistica Antverpiensia, New Series: Themes in Translation Studies, 171, 1–21.

Ullman, Michael T.

2001 “A Neurocognitive Perspective on Language: The Declarative/Procedural Model.” Nature Reviews Neuroscience 2 (10): 717–26.

2004 “Contributions of Memory Circuits to Language: The Declarative/Procedural Model.” Cognition 92 (1–2): 231–70.

Zhong, Ling, Brianne P. Noud, Harriet Pruitt, Steven C. Marcrum, and Erin M. Picou

2021 “Effects of Text Supplementation on Speech Intelligibility for Listeners with Normal and Impaired Hearing: A Systematic Review with Implications for Telecommunication.” International Journal of Audiology, June, 1–11.