Article published in:
Language, Interaction and Acquisition
Vol. 13:1 (2022) ► pp. 125–143

Audiovisual input in L2 learning

| University of Barcelona

This article provides an overview of research showing the advantages of audiovisual input in enhancing second/foreign language learning. It reviews the theoretical frameworks that explain the processing of audiovisual input and its language learning outcomes, presents the multimodal complexity of audiovisual input, and discusses its various components, focusing largely on research findings that indicate the potential of subtitles and captions for enhancing language learning. The paper goes on to address the learning that occurs through audiovisual input and issues related to processing and attention, examined primarily through eye-tracking data. Lastly, it discusses learner-specific factors, namely the role of working memory and aptitude, and learner perceptions of the viewing experience.

Keywords: captions, subtitles, multimodal input, cognitive factors, learner perceptions
Article outline
  • 1.Audiovisual input
    • 1.1A type of multimodal input
    • 1.2Research on audiovisual input
      • 1.2.1Captions and subtitles
      • 1.2.2Imagery
  • 2.Learning through audiovisual input
    • 2.1Incidental and deliberate learning through audiovisual input
    • 2.2Audiovisual input processing and attention
  • 3.Learner factors
    • 3.1Cognitive individual differences
    • 3.2Learners’ attitudes to learning through viewing
    • 3.3Learners’ perception of learning
    • 3.4Perceived focus of attention in multimodal input
  • 4.Conclusions
  • Acknowledgements
  • References
Published online: 24 August 2022
https://doi.org/10.1075/lia.22001.mun
References
Ayres, P., & Sweller, J.
(2014) The split-attention principle in multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 206–226). Cambridge University Press. DOI logoGoogle Scholar
Baddeley, A.
(2012) Working memory: Theories, models and controversies. Annual Review of Psychology, 63 1, 1–29. DOI logoGoogle Scholar
Barón, J., & Muñoz, C.
in press). Exploring the effects of audiovisual material and age in pragmatic learning. Revista Española de Lingüística Aplicada.
Benson, P.
(2011) Language learning and teaching beyond the classroom: An introduction to the field. In P. Benson, & H. Reinders (Eds.), Beyond the language classroom (pp. 7–16). Palgrave Macmillan. DOI logoGoogle Scholar
Bianchi, F., & Ciabattoni, T.
(2008) Captions and subtitles in EFL learning: An investigative study in a comprehensive computer environment. In A. Baldry, M. Pavesi, & C. Taylor Torsello, (Eds.), From didactas to Ecolingua (pp. 69–80). Edizioni Universita di Trieste.Google Scholar
Birulés-Muntané, J., & Soto-Faraco, S.
(2016) Watching subtitled films can help learning foreign languages. PloS One, 11 (6).e0158409. DOI logoGoogle Scholar
Bisson, M. J., Van Heuven, W. J., Conklin, K., & Tunney, R. J.
(2012) Processing of native and foreign language subtitles in films: An eye tracking study. Applied Psycholinguistics, 35 (2), 399–418. DOI logoGoogle Scholar
Chai, J., & Erlam, R.
(2008) The effect and the influence of the use of video and captions on second language learning. New Zealand Studies in Applied Linguistics, 14 1, 25–44.Google Scholar
Charles, T. J., & Trenkic, D.
(2015) Speech segmentation in a second language: The role of bimodal input. In Y. Gambier, A. Caimi, & C. Mariotti (Eds.), Subtitles and language learning: Principles, strategies and practical experiences (pp. 173–198). Peter Lang.Google Scholar
Cintrón-Valentín, M., García-Amaya, L., & Ellis, N. C.
(2019) Captioning and grammar learning in the L2 Spanish classroom. The Language Learning Journal, 47 (4), 1–21. DOI logoGoogle Scholar
Collins, L., & Muñoz, C.
(2016) The foreign language classroom: Current perspectives and future considerations. The Modern Language Journal, 100 1, 133–147. DOI logoGoogle Scholar
Danan, M.
(2004) Captioning and subtitling: Undervalued language learning strategies. Meta: Journal des traducteurs/Meta: Translators’ Journal, 49 (1), 67–77. DOI logoGoogle Scholar
Dressman, M.
(2020) Multimodality and language learning. In M. Dressman, & R. W. Sadler (Eds.), The handbook of informal language learning (pp. 39–55). Wiley Blackwell.Google Scholar
Duchowski, A. T.
(2002) A breadth-first survey of eye tracking applications. Behavior Research Methods, Instruments, & Computers, 34 1, 455–470. DOI logoGoogle Scholar
Erlam, R.
(2005) Language aptitude and its relationship to instructional effectiveness in second language acquisition. Language Teaching Research, 9 1, 147–171. DOI logoGoogle Scholar
Finger-Bou, R.
(2021) The effects of regular and enhanced captions on incidental vocabulary acquisition. [Unpublished Master’s Dissertation, University of Barcelona].
Gass, S., Winke, P., Isbell, D. R., & Ahn, J.
(2019) How captions help people learn languages. A working-memory, eye-tracking study. Language Learning & Technology, 23 (2), 84–104.Google Scholar
Gesa, F.
(2019) L1 / L2 subtitled TV series and EFL learning: A study on vocabulary acquisition and content comprehension at different proficiency levels. [Doctoral dissertation, Universitat de Barcelona]. UB Campus Repository. http://​hdl​.handle​.net​/2445​/149505
Godfroid, A., Boers, F., & Housen, A.
(2013) An eye for words: Gauging the role of attention in incidental L2 vocabulary acquisition by means of eye-tracking. Studies in Second Language Acquisition, 35 (3), 483–517. https://​www​.jstor​.org​/stable​/26336219. DOI logo
Hubbard, P.
(2020) Leveraging technology to integrate informal language learning within classroom settings. In M. Dressman, & R. W. Sadler (Eds.), The handbook of informal language learning (pp. 405–419). Wiley Blackwell.Google Scholar
Hulstijn, J. H.
(2003) Incidental and intentional learning. In C. J. Doughty, & M. H. Long (Eds.), The handbook of second language acquisition (pp. 349–381). Blackwell. DOI logoGoogle Scholar
Jewitt, C.
(2013) Multimodal methods for researching digital technologies. In S. Price, C. Jewitt, & B. Brown (Eds.), The SAGE handbook of digital technology research (pp. 250–265). Sage Publications Limited. DOI logoGoogle Scholar
Jordan, G., & Long, M.
in press). English language teaching: The way it is now, and the way it could be. Cambridge Scholars.
Kam, E. F., Liu, Y.-T., & Tseng, W.-T.
(2020) Effects of modality preference and working memory capacity on captioned videos in enhancing L2 listening outcomes. ReCALL, 32 (2): 213–230. DOI logoGoogle Scholar
Krashen, S. D.
(1985) The input hypothesis: Issues and implications. Longman.Google Scholar
Kruger, J-L., Hefer-Jordaan, E., & Matthew, G.
(2013) Measuring the impact of subtitles on cognitive load: Eye tracking and dynamic audiovisual texts. In Proceedings of the 2013 Conference on Eye Tracking South Africa (pp. 62–66). Association for Computing Machinery. DOI logoGoogle Scholar
Kruger, J. L., & Steyn, F.
(2014) Subtitles and eye tracking: Reading and performance. Reading Research Quarterly, 49 (1), 105–120. DOI logoGoogle Scholar
Kusyk, M., & Sockett, G.
(2012) From informal resource usage to incidental language acquisition: Language uptake from online television viewing in English. Asp, la revue dun GERAS, 62 1, 45–65. DOI logoGoogle Scholar
Lee, M., & Révész, A.
(2020) Promoting grammatical development through captions and textual enhancement in multimodal input-based tasks. Studies in Second Language Acquisition, 42 (3), 625–651. DOI logoGoogle Scholar
Leow, R., & Martin, A.
(2017) Enhancing the input to promote salience of the L2. In S. M. Gass, P. Spinner, & J. Behney (Eds.), Salience in Second Language Acquisition (pp. 167–186). Routledge. DOI logoGoogle Scholar
Long, M. H.
(2020) Optimal input for language learning: Genuine, simplified, elaborated or modified elaborated? Language Teaching, 53 (2): 169–182. DOI logoGoogle Scholar
Majuddin, E.
(2020) Incidental and intentional acquisition of multiword expressions from audio-visual input: The effects of typographically enhanced captions and repetition [Doctoral dissertation, Victoria University of Wellington].
Mariotti, C.
(2015) A survey on stakeholders’ perceptions of subtitles as a means to promote foreign language learning. In Y. Gambier, A. Caimi, & C. Mariotti (Eds.), Subtitles and language learning. Principles, strategies and practical experiences (pp. 83–102). Peter Lang.Google Scholar
Mayer, R. E.
(2009) Multimedia learning. Cambridge University Press. DOI logoGoogle Scholar
Mayer, R. E., Lee, H., & Peebles, A.
(2014) Multimedia learning in a second language: A cognitive load perspective. Applied Cognitive Psychology, 28 (5), 653–660. DOI logoGoogle Scholar
Mayer, R. E., & Moreno, R.
(2003) Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38 (1), 43–52. DOI logoGoogle Scholar
Mitterer, H., & McQueen, J.
(2009) Foreign subtitles help but native-language subtitles harm foreign speech perception. PloS One, 4 (11): e7785. DOI logoGoogle Scholar
Montero Perez, M.
(2020) Incidental vocabulary learning through viewing video. The role of vocabulary knowledge and working memory. Studies in Second Language Acquisition, 42 (4), 749–773. DOI logoGoogle Scholar
Montero Perez, M., Van Den Noortgate, W., & Desmet, P.
(2013) Captioned video for L2 listening and vocabulary learning. A meta-analysis. System, 41 (3), 720–739. DOI logoGoogle Scholar
Montero Perez, M., Peters, E., Clarebout, G., & Desmet, P.
(2014) Effects of captioning on video comprehension and incidental vocabulary learning. Language Learning & Technology, 18 (1), 118–141.Google Scholar
Montero Perez, M., Peters, E., & Desmet, P.
(2015) Enhancing vocabulary learning through captioned video: An eye-tracking study. The Modern Language Journal, 99 (2), 308–328. DOI logoGoogle Scholar
(2018) Vocabulary learning through viewing video: The effect of two enhancement techniques. Computer Assisted Language Learning, 31 1, 1–26. DOI logoGoogle Scholar
Muñoz, C.
(2008) Symmetries and asymmetries of age effects in naturalistic and instructed L2 learning. Applied Linguistics, 24 (4), 578–596. DOI logoGoogle Scholar
(2017) The role of age and proficiency in subtitle reading. An eye-tracking study. System, 67 1, 77–86. DOI logoGoogle Scholar
Paivio, A.
(1991) Dual coding theory: Retrospect and current status. Canadian Journal of Psychology/Revue canadienne de psychologie, 45 (3), 255–87. DOI logoGoogle Scholar
Pattemore, A., & Muñoz, C.
(2020) Learning L2 constructions from captioned audio-visual exposure: The effect of learner-related factors. System, 93 1, 1–13. DOI logoGoogle Scholar
Pattemore, A., Suárez, M. M., & Muñoz, C.
(2020) Exploring L2 TV preferences and perceptions: Feeling of learning and viewing mode. Short Papers. Eurocall 2020.Google Scholar
(2021, July 1–3). Learning L2 grammar constructions through audio-visual input: The role of aptitude. [Paper presentation] 30th EuroSLA Conference, University of Barcelona, Spain.
Pavesi, M., & Ghia, E.
(2020) Informal contact with English. A case study of Italian postgraduate students. Edizioni ETS.Google Scholar
Pellicer-Sánchez, A., & Boers, F.
(2019) Pedagogical approaches to the teaching and learning of formulaic language. In A. Siyanova-Chanturia, & A. Pellicer-Sánchez (Eds.), Understanding formulaic language: A second language acquisition perspective (pp. 153–173). Routledge.Google Scholar
Peters, E.
(2019) The effect of imagery and on-screen text on foreign language vocabulary learning from audiovisual input. TESOL Quarterly, 53 (4), 1008–1032. DOI logoGoogle Scholar
Peters, E., & Webb, S.
(2018) Incidental vocabulary acquisition through viewing L2 television and factors that affect learning. Studies in Second Language Acquisition, 40 1, 551–577. DOI logoGoogle Scholar
Peters, E., Heynen, E., & Puimege, E.
(2016) Learning vocabulary through audiovisual input: The differential effect of L1 subtitles and captions. System, 63 1, 134–148. DOI logoGoogle Scholar
Prensky, M.
(2001) Digital natives, digital immigrants. On the horizon, 9 (5), 1–6. DOI logoGoogle Scholar
Pujadas, G.
(2019) Language learning through extensive TV viewing. A study with adolescent EFL learners [Doctoral dissertation, University of Barcelona]. UB Campus Repository. http://​diposit​.ub​.edu​/dspace​/handle​/2445​/146118
Pujadas, G., & Muñoz, C.
(2017, April 19–21). Learning through subtitles. Learners’ preferences and task perception. [Paper presentation] International Conference on Task-Based Language Teaching, Barcelona, Spain.
(2018, October 26–28). What words do we learn better through TV series? The effects of age, proficiency and type of instruction. [Paper presentation] Second Language Research Forum, Montreal, Canada.
(2019) Extensive viewing of captioned and subtitled TV series: A study of L2 vocabulary learning by adolescents, The Language Learning Journal, 47 (4), 479–496. DOI logoGoogle Scholar
(2020) Examining adolescent EFL learners’ TV viewing comprehension through captions and subtitles. Studies in Second Language Acquisition, 42 (3), 551–575. DOI logoGoogle Scholar
(2021, July 1–3). Repeating or seeing? Frequency and imagery in vocabulary learning through TV viewing. [Paper presentation] 30th EuroSLA Conference, University of Barcelona., Spain.
Rayner, K., Slattery, T. J., & Bélanger, N. N.
(2010) Eye movements, the perceptual span, and reading speed. Psychonomic Bulletin & Review, 17 (6), 834-e839. DOI logoGoogle Scholar
Rodgers, M. P. H.
(2013) English language learning through viewing television: An investigation of comprehension, incidental vocabulary acquisition, lexical coverage, attitudes and captions [Doctoral dissertation, Victoria University of Wellington].
(2018) The images in television programs and the potential for learning unknown words. ITL – International Journal of Applied Linguistics, 169 (1), 191–211. DOI logoGoogle Scholar
Rodgers, M. P. H., & Webb, S.
(2020) Incidental vocabulary learning through viewing television. ITL – International Journal of Applied Linguistics, 171 (2), 191–220. DOI logoGoogle Scholar
Sergeeva, M.
(2021) Subtitled audiovisual materials in the FL classroom: Teachers’ perspective. [Master’s Thesis, University of Barcelona].
Suárez, M., & Gesa, F.
(2019) Learning vocabulary with the support of sustained exposure to captioned video: Do proficiency and aptitude make a difference? The Language Learning Journal, 47 (4), 497–517. DOI logoGoogle Scholar
Suzuki, Y.
(2021) Probing the construct validity of LLAMA_D as a measure of implicit learning aptitude. Incidental instructions, confidence ratings, and reaction time. Studies in Second Language Acquisition, 43 (3), 663–676. DOI logoGoogle Scholar
Sweller, J.
(1988) Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. DOI logoGoogle Scholar
(1999) Instructional design in technical areas. ACER Press.Google Scholar
Sydorenko, T.
(2010) Modality of input and vocabulary acquisition. Language Learning & Technology, 14 (2), 50–73.Google Scholar
Uchihara, T., Webb, S., & Yanagisawa, A.
(2019) The effects of repetition on incidental vocabulary learning: A meta-analysis of correlational studies. Language Learning, 69 (3), 559–599. DOI logoGoogle Scholar
Vandergrift, L.
(2007) Recent developments in second language and foreign language listening comprehension research. Language Teaching, 40 (3), 191–210. DOI logoGoogle Scholar
Vanderplank, R.
(1988) The value of teletext sub-titles in language learning. ELT Journal, 42 (4), 272–281. DOI logoGoogle Scholar
(2016) Captioned media in foreign language learning and teaching: Subtitles for the deaf and hard-of-hearing as tools for language learning. Palgrave Macmillan. DOI logoGoogle Scholar
Van Lommel, S., Laenen, A., & D’Ydewalle, G.
(2006) Foreign-grammar acquisition while watching subtitled television programmes. British Journal of Educational Psychology, 76 1, 243–258. DOI logoGoogle Scholar
Webb, S., & Rodgers, M. P. H.
(2009) Vocabulary demands of television programs. Language Learning, 59 (2), 335–366. DOI logoGoogle Scholar
Wiley, J., Sanchez, C. A., & Jaeger, A. J.
(2014) The individual differences in working memory capacity principle in multimedia learning. In R. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed.) (pp. 598–619). Cambridge University Press. DOI logoGoogle Scholar
Winke, P., Gass, S., & Syodorenko, T.
(2010) The effects of captioning videos used for foreign language listening activities. Language Learning & Technology, 14 (1), 65–86.Google Scholar
Winke, P., Sydorenko, T., & Gass, S.
(2013) Factors influencing the use of captions by foreign language learners, an eye-tracking study. The Modern Language Journal, 97 1, 254–275. DOI logoGoogle Scholar
Wisniewska, N., & Mora, J. C.
(2020) Can captioned video benefit second language pronunciation? Studies in Second Language Acquisition, 42 (3), 599–624. DOI logoGoogle Scholar