Article published In:
ITL - International Journal of Applied LinguisticsVol. 173:1 (2022) ► pp.2–17
Multimodal reading and second language learning
Most of the texts that second language learners engage with include both text (written and/or spoken) and images. The use of images accompanying texts is believed to support reading comprehension and facilitate learning. Despite their widespread use, very little is known about how the presentation of multiple input sources affects the attentional demands and the underlying cognitive processes involved. This paper provides a review of research on multimodal reading, with a focus on attentional processing. It first introduces the relevant theoretical frameworks and empirical evidence provided in support of the use of pictures in reading. It then reviews studies that have looked at the processing of text and pictures in first and second language contexts. Based on this review, main gaps in research and future research directions are identified. The discussion provided in this paper aims at advancing research on multimodal reading in a second language. Achieving a better understanding of the underlying cognitive processes in multimodal reading is crucial to inform pedagogical practices and to develop theoretical accounts of second language multimodal reading.
Keywords: multimodality, second language learning, reading, eye-tracking, cognitive processes, multimedia learning, imagery
Article outline
- Introduction
- Multimodality in reading
- Multimodal reading in second language learning
- Cognitive processing of multimodal texts
- Research gaps and future directions
- Purposes of reading
- Relationship between attentional allocation and learning
- Image-Text relationship
- Conclusion
-
References
References
Abraham, P., & Farías, M.
(2017) Reading
with eyes wide open: Reflections on the impact of multimodal texts on second language
reading. Íkala, Revista de Lenguaje y
Cultura,
22
(1), 57–70. 
Alemdag, E., & Cagiltay, K.
(2018) A
systematic review of eye tracking research on multimedia learning. Computers &
Education,
125
1, 413–428.
Bezemer, J., & Kress, G.
(2008) Writing
in multimodal texts: A social semiotic account of designs for learning. Written
Communication,
25
(2), 166–195.
Bisson, M-J., W. Van Heuven, K. Conklin & R. Tunney
(2015) The
role of verbal and pictorial information in multimodal incidental acquisition of foreign language
vocabulary. Quarterly Journal of Experimental
Psychology
68
1, 1306–1326.
Boerma, I. E., Mol, S. E., & Jolles, J.
(2016) Reading
pictures for story comprehension requires mental imagery skills. Frontiers in
Psychology,
7
(1630), 1–10.
Buck, G.
(2001) Assessing
listening. New York, NY: Cambridge University Press.
Center, Y., Freeman, L., Robertson, G., & Outhred, L.
(1999) The
effect of visual imagery training on the reading and listening comprehension of low listening comprehenders in year
2. Journal of Research in
Reading
22
1, 241–256.
Chang, Y., & Choi, S.
(2014) Effects
of seductive details evidenced by gaze duration. Neurobiology of Learning and
Memory,
109
1, 131–138.
Chen, S. C., Hsiao, M. S., & She, H. C.
(2015) The
effects of static versus dynamic 3D representations on 10th grade students’ atomic orbital mental model construction: Evidence
from eye movement behaviors. Computers in Human
Behavior,
53
1, 169–180.
Clark, J. M., & Paivio, A.
(1991) Dual
coding theory and education. Educational Psychology
Review,
3
1, 149–210.
Eitel, A., & Scheiter, K.
(2015) Picture
or text first? Explaining sequence effects when learning with pictures and text. Educational
Psychology
Review,
27
1, 153–180.
Elley, W. B. & F. Mangubhai
(1983) The
impact of reading on second language learning. Reading Research
Quarterly
19
(1), 53–67.
Evans, M. A. & J. Saint-Aubin
(2005) What
children are looking at during shared storybook reading: Evidence from eye movement
monitoring. Psychological
Science
16
1, 913–920.
Grabe, W.
(2006) Areas
of research that influence L2 reading instruction. In Usó-Juan, E. & Martínez-Flor, A. (Eds.), Current
trends in the development and teaching of the four language
skills (pp. 279–301). New York: Mouton de Gruyter.
(2009) Reading
in a second language: Moving from theory to
practice. Cambridge: Cambridge University Press.
Godfroid, A., Boers, F., & Housen, A.
(2013) An
eye for words: Gauging the role of attention in incidental L2 vocabulary acquisition by means of
eye-tracking. Studies in Second Language
Acquisition,
35
(3), 483–517.
Guijarro, A. J. M., & Sanz, M. J. P.
(2009) In interaction of image and verbal text in a picture book. A multimodal and systemic functional study. In Ventola, E., & Guijarro, A. J. M. (Eds.), The
world told and the world
shown (pp. 107–123). London: Palgrave Macmillan.
Hannus, M., & Hyönä, J.
(1999) Utilization of illustrations during learning of science textbook passages among low- and high-ability children. Contemporary Educational
Psychology,
24
(2), 95–123.
Hill, D.
(2013) Graded
readers. ELT
Journal
67
(1), 85–125.
Holsanova, J.
(2014) Reception
of multimodality: Applying eye tracking methodology in multimodal
research. In C. Jewitt (Ed.), The
Routledge Handbook of Multimodal
Analysis (pp. 287–298). New York: Routledge.
Johnson, C. I., & Mayer, R. E.
(2012) An
eye movement analysis of the spatial contiguity effect in multimedia learning. Journal of
Experimental Psychology:
Applied,
18
1, 178–191.
Justice, L. M., Skibbe, L., Canning, A., & Lankford, C.
(2005) Pre-schoolers,
print and storybooks: An observational study using eye movement analysis. Journal of Research
in
Reading,
28
1, 229–243.
Kress, G.
(2000) Multimodality:
A social semiotic approach to communication. London: Routledge.
Kress, G. & van Leeuwen, T.
(2021) Reading
images: The grammar of visual
design. New York: Routledge.
Lee, H., & Mayer, R.
(2018) Fostering
learning from instructional video in a second language. Applied Cognitive
Psychology,
32
1, 648–654.
Lee, M., & Révész, A.
(2018) Promoting
grammatical development through textually enhanced captions: An eye-tracking study. The Modern
Language
Journal,
102
1, 557–577.
Makransky, G., Terkildsen, T. S., & Mayer, R. E.
(2019) Role
of subjective and objective measures of cognitive processing during learning in explaining the spatial contiguity
effect. Learning and
Instruction,
61
1, 23–34.
Mason, L., Pluchino, P., Tornatora, M. C., & Ariasi, N.
(2013a) An
eye-tracking study of learning science text with concrete and abstract illustrations. Journal
of Experimental
Education,
81
1, 356–384.
Mason, L., Tornatora, M. C., & Pluchino, P.
(2013b) Do
fourth graders integrate text and picture in processing and learning from an illustrated science text? Evidence from
eye-movement patterns. Computers and
Education,
60
1, 95–109.
(2015) Integrative
processing of verbal and graphical information during re-reading predicts learning from illustrated text: An eye movement
study. Reading and
Writing,
28
1, 851–872.
Mayer, R. E.
(2001) Multimedia
learning. Cambridge: Cambridge University Press.
(2009) Multimedia
learning (2nd
ed.). Cambridge: Cambridge University Press.
(2014a) Introduction
to multimedia learning. In R. E. Mayer (Ed.), The
Cambridge handbook of multimedia learning (2nd
ed.) (pp. 1–24). Cambridge: Cambridge University Press.
Mayer, R.
(2014b) Cognitive
theory of multimedia learning. In R. Mayer (Ed.), The
Cambridge handbook of multimedia
learning (pp. 43–71). Cambridge: Cambridge University Press.
Mayer, R. E.
(2021) Multimedia
learning (3rd
ed.). Cambridge: Cambridge University Press.
Mayer, R. E., & Anderson, R. B.
(1991) Animations
need narrations: An experimental test of a dual-coding hypothesis. Journal of Educational
Psychology,
83
(4), 484–490.
Mayer, R. E., Howarth, J. T., Kaplan, M., & Hanna, S.
(2018) Applying
the segmenting principle to online geography slideshow lessons. Educational Technology Research
and
Development,
66
(3), 563–577.
Mayer, R. E., & Moreno, R.
(1998) A
split-attention effect in multimedia learning: Evidence for dual processing systems in working
memory. Journal of Educational
Psychology,
90
(2), 312–320.
Mills, K., & Unsworth, L.
(2017) Multimodal
literacy. Oxford research encyclopedia of education. Retrieved 22 Jul.
2021, from: [URL]
Montero Perez, M.
(2020) Multimodal
input in SLA research. Studies in Second Language
Acquisition,
42
(3), 653–663.
Montero Perez, M., Van Den Noortgate, W., & Desmet, P.
(2013) Captioned
video for L2 listening and vocabulary learning: A
meta-analysis. System,
41
1, 720–739.
Montero Perez, M., Peters, E., & Desmet, P.
(2015) Enhancing
vocabulary learning through captioned video: an eye-tracking study. The Modern Language
Journal,
99
(2), 308–28.
Moreno, R., & Mayer, R. E.
(1999) Multimedia-supported
metaphors for meaning making in mathematics. Cognition and
Instruction,
17
(3), 215–248.
(2002a) Learning
science in virtual reality multimedia environments: Role of methods and media. Journal of
Educational
Psychology,
94
(3), 598–610.
(2002b) Verbal
redundancy in multimedia learning: When reading helps listening. Journal of Educational
Psychology,
94
(1), 156–163.
Omaggio, A. C.
(1979) Pictures
and second language comprehension: Do they help? Foreign Language
Annals,
12
1, 107–116.
Paivio, A.
(1986) Mental
representations: A dual coding approach. Oxford: Oxford University Press.
(2006) Mind
and its evolution: A dual coding approach. Lawrence Erlbaum.
Pellicer-Sánchez, A.
(2016) Incidental
L2 vocabulary acquisition from and while reading: An eye-tracking study. Studies in Second
Language
Acquisition,
38
1, 97–130.
Pellicer-Sánchez, A., Conklin, K., Rodgers, M. P., & Parente, F.
(2021), The Effect of Auditory Input on Multimodal Reading Comprehension: An Examination of Adult Readers’ Eye Movements. The Modern Language Journal, 105(4), 936–956:
Pellicer-Sánchez, A., Tragant, E., Conklin, K., Rodgers, M., Llanes, A., & Serrano, R.
(2018) L2
reading and reading-while-listening in multimodal learning conditions: An eye-tracking
study. ELT Research
Papers,
18
(01), 1–28. London: British Council.
Pellicer-Sánchez, A., Tragant, E., Conklin, K., Rodgers, M., Serrano, R., & Llanes, A.
(2020) Young
learners’ processing of multimodal input and its impact on reading comprehension: An eye-tracking
study. Studies in Second Language
Acquisition,
42
(3), 577–598.
Peters, E., & Muñoz, C.
(2020) Introduction
to the special issue: Language learning from multimodal input. Studies in Second Language
Acquisition,
42
(3), 489–497.
Plass, J. L., Chun, D. M., Mayer, R. E., & Leutner, D.
(1998) Supporting
visual and verbal learning preferences in a second language multimedia learning
environment. Journal of Educational
Psychology,
90
1, 25–36.
Puimège, E., Montero Perez, M., & Peters, E.
(2021) Promoting
L2 acquisition of multiword units through textually enhanced audiovisual input: An eye-tracking
study. Second Language Research. Online
First.
Rodgers, M. P. H., & Webb, S.
(2017) The
effects of captions on EFL learners’ comprehension of English-language television
programs. Calico
Journal,
34
1, 20–38.
Rodgers, M. P. H. & Webb, S.
(2019) Incidental
vocabulary learning through viewing television. ITL-International Journal of Applied
Linguistics,
171
(2), 191–220.
Roy-Charland, A., Saint-Aubin, J., & Evans, M. A.
(2007) Eye
movements in shared book reading with children from kindergarten to grade 4. Reading and
Writing,
20
1, 909–931.
Schmidt-Weigand, F.
(2011) Does
animation amplify the modality effect–or is there any modality effect at all? Zeitschrift für
Padagogische
Psychologie,
25
(4), 245–256.
Schmidt-Weigand, F., Kohnert, A., & Glowalla, U.
(2010) A
closer look at split visual attention in system-and self-paced instruction in multimedia
learning. Learning and
Instruction,
20
(2), 100–110.
Schnotz, W., & Baadte, C.
(2008) Domain
learning versus language learning with multimedia. In M. Farías & K. Obilinovic (Eds.), Aprendizaje
multimodal/Multimodal
learning (pp. 21–49). Santiago de Chile: Publifahu USACH.
Seburn, T.
(2017) Learner-sourced
visuals for deeper text engagement and conceptual
comprehension. In K. Donaghy & D. Xerri (Eds.), The
image in English language
teaching (pp. 79–88). Malta: Gutenberg Press.
Serrano, R., & Pellicer-Sánchez, A.
(2019) Young
L2 learners’ online processing of information in a graded reader during reading-only and reading-while-listening conditions: A
study of eye movements. Applied Linguistics Review. Advance online
publication.
Takacs, Z. K., & Bus, A. G.
(2016) Benefits
of motion in animated storybooks for children’s visual attention and story comprehension. An eye-tracking
study. Frontiers in
Psychology,
7
1, 1–12.
Tang, G.
(1992) The
effect of graphic representation of knowledge structures on ESL reading comprehension. Studies
in Second Language
Acquisition,
14
1, 177–195.
Tragant, E., & Pellicer-Sánchez, A.
(2019) Young
learners’ engagement with multimodal exposure: An eye tracking
study. System,
80
1, 212–223.
Unsworth, L.
(2006) Towards
a metalanguage for multiliteracies education: Describing the meaning – making resources of language-image
interaction. English
Teaching,
5
(1), 55–76.
(2014) Multimodal
reading comprehension: Curriculum expectations and large-scale literacy testing
practices. Pedagogies: An International
Journal,
9
(1), 26–44.
Winke, P., Gass, S., & Sydorenko, T.
(2010) The
effects of captioning videos used for foreign language listening activities. Language Learning
&
Technology,
14
1, 65–86.
Wright, A.
(2010) Pictures
for language learning. Cambridge: Cambridge University Press.