Cover not available
Article published In:
Methodological Issues in Experimental Research in Audiovisual Translation and Media Accessibility
Edited by Gian Maria Greco, Anna Jankowska and Agnieszka Szarkowska
[Translation Spaces 11:1] 2022
► pp. 12–37

Establishing a theoretical framework for AVT research

The importance of cognitive models

ORCID logo | Macquarie University | North-West University
ORCID logo | Macquarie University
Empirical research on the cognitive processing of audiovisual translation (AVT) products has been thriving over the past decade. While the use of cutting-edge experimental tools such as eye trackers has drawn increasing scholarly attention and accelerated the progress in understanding the complex mental processes involved in the reception of multimodal AVT products, relatively less attention has been devoted to the importance of establishing a theoretical framework or cognitive model that can explain and predict the behaviours observed in empirical experiments. By reviewing numerous theories or cognitive models relevant to AVT research in explaining how different perceptual and cognitive systems operate for understanding multimodal products, this paper calls for engagement with these theoretical frameworks and models to work towards a robust model that can generate testable hypotheses for the integration and interaction of multiple sources of information involved in the processing of AVT or other multimodal products.
Keywords: audiovisual translation (AVT), cognitive models, multimodal integrated language framework (MILF), subtitles
Article outline
  • Introduction
  • Memory and cognition
    • Modal model – Atkinson and Shiffrin (1968)
    • Model of working memory – Baddeley and Hitch (1974); Baddeley (2000, 2010, 2012)
  • Multimodality
    • Cognitive theory of multimedia learning (CTML) – Mayer (2014)
      • Dual-channel assumption
      • Limited capacity assumption
      • Active processing assumption
    • Integrated model of text and picture comprehension (ITPC) – Schnotz and Bannert (2003)
    • Scene perception and event comprehension theory (SPECT) – Loschky et al. (2020)
      • Front-end cognitive processes
      • Back-end cognitive processes
      • Dynamic visual narratives
  • Models in reading research
    • The E-Z Reader model – Reichle et al. (2012)
    • The Über-Reader model – Reichle (2021)
    • Multimodal integrated-language framework (MILF) – Liao et al. (2021a)
  • Concluding remarks
  • Acknowledgement
  • Notes
  • References
Published online: 19 July 2022
DOI logo
https://doi.org/10.1075/ts.21024.kru
References (51)
References
Allen, Richard J., Graham J. Hitch, Judit Mate, and Alan D. Baddeley. 2012. “Feature Binding and Attention in Working Memory: A Resolution of Previous Contradictory Findings.” Quarterly Journal of Experimental Psychology 65 (12): 2369–83. DOI logoGoogle Scholar
Ans, Bernard, Serge Carbonnel, and Sylviane Valdois. 1998. “A Connectionist Multiple-Trace Memory Model for Polysyllabic Word Reading.” Psychological Review 105 (4): 678–723. DOI logoGoogle Scholar
Atkinson, Richard. C., and Richard M. Shiffrin. 1968. “Human Memory: A Proposed System and Its Control Processes.” In The Psychology of Learning and Motivation: Advances in Research and Theory, edited by Spence, Kenneth W., and Janet Taylor Spence, 89–195. New York: Academic Press.Google Scholar
Baddeley, Alan David. 2000. “The Episodic Buffer: A New Component of Working Memory?” Trends in Cognitive Sciences 4 (11): 417–23. DOI logoGoogle Scholar
. 2010. “Working memory.” Current biology 20 (4), 136–140. DOI logoGoogle Scholar
. 2012. “Working Memory: Theories, Models, and Controversies.” Annual Review of Psychology 63 (1): 1–29. DOI logoGoogle Scholar
Baddeley, Alan David, and Graham J. Hitch. 1974. “Working memory.” In The Psychology of Learning and Motivation: Advances in Research and Theory, edited by Bower, Gordon H., 47–89. New York: Academic Press.Google Scholar
Baddeley, Alan David, Richard J. Allen, and Graham J. Hitch. 2011. “Binding in Visual Working Memory: The Role of the Episodic Buffer.” Neuropsychologia 49 (6): 1393–1400. DOI logoGoogle Scholar
Chung, Jing-mei. 1999. “The Effects of Using Video Texts Supported with Advance Organizers and Captions on Chinese College Students’ Listening Comprehension: An Empirical Study.” Foreign Language Annals 32 (3): 295–308. DOI logoGoogle Scholar
Danan, Martine. 2004. “Captioning and Subtitling: Undervalued Language Learning Strategies.” Meta 49 (1): 67–77. DOI logoGoogle Scholar
de Pedro Ricoy, Raquel. 2012. “Multimodality in Translation: Steps towards Socially Useful Research.” Journal Multimodal Communication 1 (2): 181–203. DOI logoGoogle Scholar
Díaz Cintas, Jorge, and Agnieszka Szarkowska. 2020. “Introduction: Experimental Research in Audiovisual Translation–Cognition, Reception, Production.” The Journal of Specialised Translation, 331, 3–16.Google Scholar
Forstmann, Birte U., Eric-Jan Wagenmakers, Tom Eichele, Scott Brown, and John T. Serences. 2011. “Reciprocal Relations between Cognitive Neuroscience and Formal Cognitive Models: Opposites Attract?” Trends in Cognitive Sciences 15 (6): 272–79. DOI logoGoogle Scholar
Gernsbacher, Morton Ann. 2015. “Video Captions Benefit Everyone.” Edited by Susan T. Fiske. Policy Insights from the Behavioral and Brain Sciences 2 (1): 195–202. DOI logoGoogle Scholar
Hintzman, Douglas L. 1984. “MINERVA 2: A Simulation Model of Human Memory.” Behavior Research Methods, Instruments, & Computers 16 (2): 96–101. DOI logoGoogle Scholar
Kaindl, Klaus. 2013. “Multimodality and Translation.” In The Routledge Handbook of Translation Studies, edited by Millán, Carmen, and Francesca Bartrina, 257–269. London: Routledge.Google Scholar
Kalyuga, Slava, and John Sweller. 2014. “The Redundancy Principle in Multimedia Learning.” In The Cambridge Handbook of Multimedia Learning, edited by Mayer, Richard E., 247–262. New York: Cambridge University Press. DOI logoGoogle Scholar
Ketola, Anne. 2016. “Towards a Multimodally Oriented Theory of Translation: A Cognitive Framework for the Translation of Illustrated Technical Texts.” Translation Studies 9 (1): 67–81. DOI logoGoogle Scholar
Kruger, Jan-Louis. 2016. “Psycholinguistics and Audiovisual Translation.” Target. International Journal of Translation Studies 28 (2): 276–87. DOI logoGoogle Scholar
. 2019. “Eye tracking in Audiovisual Translation Research.” In The Routledge Handbook of Audiovisual Translation, edited by Pérez-González, Luis, 350–366. London: Routledge, Taylor and Francis Group.Google Scholar
. 2020. “Translation, Multimodality and Cognition.” In The Routledge Handbook of Translation and Cognition, edited by Alves, Fabio, and Arnt Lykke Jakobsen, 433–444. London, UK: Routledge. DOI logoGoogle Scholar
Kruger, Jan-Louis, and Faans Steyn. 2014. “Subtitles and Eye Tracking: Reading and Performance.” Reading Research Quarterly 49 (1): 105–20. DOI logoGoogle Scholar
Kruger, Jan-Louis, Natalia Wisniewska, and Sixin Liao. 2021. “Why Subtitle Speed Matters: Evidence from Word Skipping and Rereading.” Applied Psycholinguistics 43 (1): 211–36. DOI logoGoogle Scholar
Liao, Sixin, and Jan-Louis Kruger. 2022. “Cognitive Processing of Subtitles: Charting the Future by Mapping the Past.” In The Routledge Handbook of Translation, Interpreting, and Bilingualism, edited by Ferreira, Aline, and John W. Schwieter.Google Scholar
Liao, Sixin, Lili Yu, Jan-Louis Kruger, and Erik D. Reichle. 2021a. “The impact of Audio on the Reading of Inter- vs. Intra-lingual Subtitles: Evidence from Eye Movements.” Applied Psycholinguistics 43 (1): 237–69. DOI logoGoogle Scholar
Liao, Sixin, Lili Yu, Erik D. Reichle, and Jan-Louis Kruger. 2021b. “Using Eye Movements to Study the Reading of Subtitles in Video.” Scientific Studies of Reading 25(5): 417–435. DOI logoGoogle Scholar
Loschky, Lester C., Adam M. Larson, Tim J. Smith, and Joseph P. Magliano. 2020. “The Scene Perception & Event Comprehension Theory (SPECT) Applied to Visual Narratives.” Topics in Cognitive Science 12 (1): 311–51. DOI logoGoogle Scholar
Lum, Jarrad A. G., Gina Conti-Ramsden, Debra Page, and Michael T. Ullman. 2012. “Working, Declarative and Procedural Memory in Specific Language Impairment.” Cortex 48 (9): 1138–54. DOI logoGoogle Scholar
Mayer, Richard E. 2014. The Cambridge Handbook of Multimedia Learning. New York: Cambridge University Press. DOI logoGoogle Scholar
Paivio, Allan. 1986. Mental Representations: A Dual Coding Approach. Oxford, England: Oxford University Press.Google Scholar
. 1991. “Dual Coding Theory: Retrospect and Current Status.” Canadian Journal of Psychology 45 (3): 255–87. DOI logoGoogle Scholar
. 2007. Mind and Its Evolution: A Dual Coding Theoretical Approach. Mahwah, NJ: Erlbaum.Google Scholar
Perego, Elisa, Fabio Del Missier, Marco Porta, and Mauro Mosconi. 2010. “The Cognitive Effectiveness of Subtitle Processing.” Media Psychology 13 (3): 243–72. DOI logoGoogle Scholar
Pérez-González, Luis. 2014. “Multimodality in Translation and Interpreting Studies: Theoretical and Methodological Perspectives.” In A Companion to Translation Studies, edited by Bermann, Sandra, and Catherine Porter, 119–131. Malden, MA/Oxford, England: Wiley-Blackwell. DOI logoGoogle Scholar
Prezenski, Sabine, André Brechmann, Susann Wolff, and Nele Russwinkel. 2017. “A Cognitive Modeling Approach to Strategy Formation in Dynamic Decision Making.” Frontiers in Psychology 81 (August): 1–18. DOI logoGoogle Scholar
Reichle, Erik D. 2021. Computational Models of Reading: A Handbook. Oxford: Oxford University Press. DOI logoGoogle Scholar
Reichle, Erik D., Alexander Pollatsek, and Keith Rayner. 2012. “Using E-Z Reader to Simulate Eye Movements in Nonreading Tasks: A Unified Framework for Understanding the Eye–Mind Link.” Psychological Review 119 (1): 155–85. DOI logoGoogle Scholar
Reichle, Erik D., Lili Yu, Sixin Liao, and Jan-Louis Kruger. 2021. “Using Simulations to Understand the Reading of Rapidly Displayed Subtitles.” Proceedings of the 43rd Annual Conference of the Cognitive Science Society.Google Scholar
Schnotz, Wolfgang. 2005. “Integrated Model of Text and Picture Comprehension.” In The Cambridge Handbook of Multimedia Learning, edited by Mayer, Richard E., 49–70. New York: Cambridge University Press. DOI logoGoogle Scholar
Schnotz, Wolfgang, and Maria Bannert. 2003. “Construction and Interference in Learning from Multiple Representation.” Learning and Instruction 13 (2): 141–56. DOI logoGoogle Scholar
Schnotz, Wolfgang. 2014. “Integrated model of text and picture comprehension.” In The Cambridge Handbook of Multimedia Learning (2nd ed), edited by Mayer, Richard E., 72–103. DOI logoGoogle Scholar
Schnotz, Wolfgang, and Inga Wagner. 2018. “Construction and Elaboration of Mental Models through Strategic Conjoint Processing of Text and Pictures.” Journal of Educational Psychology 110 (6): 850–63. DOI logoGoogle Scholar
Seeber, Kilian G. 2017. Multimodal processing in simultaneous interpreting. In The Handbook of Translation and Cognition, edited by Schwieter, John and Aline Ferreira, 461–475. Joh Wiley & Sons. DOI logoGoogle Scholar
Squire, Larry R., and Stuart M. Zola. 1996. “Structure and Function of Declarative and Nondeclarative Memory Systems.” Proceedings of the National Academy of Sciences 93 (24): 13515–22. DOI logoGoogle Scholar
Szarkowska, Agnieszka, and Piotr Wasylczyk. 2018. “Five Things You Wanted to Know about Audiovisual Translation Research, but Were Afraid to Ask.” Journal of Audiovisual Translation 1 (1): 8–25. DOI logoGoogle Scholar
Szarkowska, Agnieszka, Breno Silva, and David Orrego-Carmona. 2021. “Effects of Subtitle Speed on Proportional Reading Time.” Translation, Cognition & Behavior. DOI logoGoogle Scholar
Taylor, Christopher. 2016. “The Multimodal Approach in Audiovisual Translation.” Target. International Journal of Translation Studies 28 (2): 222–36. DOI logoGoogle Scholar
Tuominen, Tiina, Catalina Jiménez Hurtado, and Anne Ketola. 2018. “Why Methods Matter: Approaching Multimodality in Translation Research.” Linguistica Antverpiensia, New Series: Themes in Translation Studies, 171, 1–21.Google Scholar
Ullman, Michael T. 2001. “A Neurocognitive Perspective on Language: The Declarative/Procedural Model.” Nature Reviews Neuroscience 2 (10): 717–26. DOI logoGoogle Scholar
2004. “Contributions of Memory Circuits to Language: The Declarative/Procedural Model.” Cognition 92 (1–2): 231–70. DOI logoGoogle Scholar
Zhong, Ling, Brianne P. Noud, Harriet Pruitt, Steven C. Marcrum, and Erin M. Picou. 2021. “Effects of Text Supplementation on Speech Intelligibility for Listeners with Normal and Impaired Hearing: A Systematic Review with Implications for Telecommunication.” International Journal of Audiology, June, 1–11. DOI logoGoogle Scholar
Cited by (1)

Cited by one other publication

Bolaños García-Escribano, Alejandro
2024. Cloud subtitling in research-led education. Babel. Revue internationale de la traduction / International Journal of Translation / Revista Internacional de Traducción 70:4 ► pp. 484 ff. DOI logo

This list is based on CrossRef data as of 28 september 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.