Article published In:
Theory in Applied Linguistics Research: Critical approaches to production, performance and participationEdited by Theresa Lillis
[AILA Review 28] 2015
► pp. 49–71
Edited by Theresa Lillis
[AILA Review 28] 2015
► pp. 49–71
Semiotic work
Applied Linguistics and a social semiotic account of Multimodality
This article imagines a tussle between Multimodality, focused on ‘modes’, and Applied Linguistics (AL), based on ‘language’. A Social Semiotic approach to MM treats speech and writing as modes with distinct affordances, and, as all modes, treats them as ‘partial’ means of communication. The implications of partiality confound long-held assumptions of the sufficiency of ‘language’ for all communicational needs: an assumption shared by AL. Given MM’s plurality of modes and the diversity of audiences, design moves into focus, with a shift from competent performance to apt design. Principles of composition — e.g. linearity versus modularity — become crucial, raising the question at the heart of this paper: how do AL and MM deal with the shape of the contemporary semiotic landscape?
Keywords: social semiotics, multimodality, modularity, audience, writing, linearity, design, affordances
Available under the Creative Commons Attribution-NonCommercial (CC BY-NC) 4.0 license.
For any use beyond this license, please contact the publisher at [email protected].
References
Coulthard, M
1992 Forensic Discourse Analysis. Advances in Spoken Discourse Analysis. London: Routledge,
Ellis, R
2008 The Study of Second Language Acquisition. Oxford: OUP.
Fisiak, J
1981 Contrastive Linguistics and the Language Teacher [Language Teaching Methodology Series]. Oxford: Pergamon Press. 
Hartmann, R.R.K
(ed) 2003 Lexicography. Critical Concepts. London: Routledge.
König, E. & Gast, V
2009 Understanding English-German Contrasts. Berlin: Erich Schmidt.
Kress, G
2010 Multimodality. A Social Semiotic Approach to Contemporary Communication. London: Routledge.
Kress, G. & van Leeuwen, T
2001 Multimodal Discourse. London: Edward Arnold.
Kress, G., Jewitt, C., Ogborn, J. & Tsatsarelis, C
2001 Multimodal Teaching and Learning. The Rhetorics of the Science Classroom. London: Continuum.
Lado, R
1957 Second Language Acquisition. Ann Arbor MI: University of Michigan.
Here I list key works which have informed my discussion of Multimodality in this paper
Barthes, R
1973 Mythologies. St. Albans: Paladin.
Bauman, Z
2000 Liquid Modernity. Cambridge: Polity Press.
Bezemer, J. & Kress, G
2015 The textbook in a changing multimodal landscape. In Language in Multimodal Contexts, N. Klug & H. Stöckl (eds). Berlin: De Gruyter.
Bezemer, J., Cope, A., Kress, G. & Kneebone, R
2014 Holding the scalpel: Achieving surgical care in a learning environment. Journal of Contemporary Ethnography 43(1): 38–63.
Bezemer, J. & Kress, G
2014 Touch: A resource for making meaning. Australian Journal of Language and Literacy 37(2): 77–85.
Bezemer, J., Kress, G., Cope, A. & Kneebone, R
2012 Learning in the operating theatre: A social semiotic perspective. In Work-based Learning in Clinical Settings: Insights from Socio-cultural Perspectives, V. Cook, C. Daly & M. Newman (eds), 125–141. Abingdon: Radcliffe.
Bezemer, J
2008 Displaying orientation in the classroom: Students’ multimodal responses to teacher instructions. Linguistics and Education 19(2): 166–178.
Bezemer, J. & Kress, G
2008 Writing in multimodal texts: A social semiotic account of designs for learning. Written Communication 25(2): 166–195.
Franks, A
1996 Drama education, the body and representation (or, the mystery of the missing bodies). Research in Drama Education: The Journal of Applied Theatre and Performance 11: 105–119.
Jewitt, C
(ed.) 2014 The Routledge Handbook of Multimodal Analysis. London: Routledge.
Kress, G
1971 Sentence complexity in contrastive linguistics. In (ERIC) Papers in Contrastive Linguistics, G. Nickel (ed.). Cambridge: CUP.
1997 Before Writing: Rethinking the Paths to Literacy. London: Routledge.
2000 Text as the punctuation of semiosis: Pulling at some of the threads. In Intertextuality and the Media. From Genre to Everyday Life, U.H. Meinhof & J. Smith (eds), 132–153. Manchester: Manchester University Press.
Kress, G. & van Leeuwen, T
2006 Reading Images: A Grammar of Visual Design. London: Routledge.
Kress, G., Jewitt, C., Bourne, J., Franks, A., Hardcastle, J., Jones, K. & Reid, E
2005 English in Urban Classrooms: Multimodal Perspectives on Teaching and Learning. London: RoutledgeFalmer.
Lindstrand, F. & Insulander, E
2014 Setting the ground for engagement: Multimodal perspectives on exhibition design. Designs for Learning 5(1-2): 30–49.
Mavers, D
2011 Children’s Drawing and Writing: The Remarkable in the Unremarkable. London: Routledge.
New London Group
1996 A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review 66(1): 60–92.
Newfield, D
2014 Transformation, transduction and the transmodal moment. In The Routledge Handbook of Multimodal Analysis, C. Jewitt (ed.), 100–113. London: Routledge.
Pelletier, C
2007 Learning through design: Subjectivity and meaning in young people’s computer game production work. PhD dissertation, University of London.
Stein, P
2008 Multimodal Pedagogies in diverse Classrooms. Representation, Rights and Resources. New York NY: Routledge.
Unsworth, L
(ed.) 2008 Multimodal Semiotics: Functional Analysis in Contexts of Education. London: Continuum.
Wulf, C. et al.
2010 Ritual and Identity. The Staging and Performing of Rituals in the Lives of Young People. London: Tufnell.
Yandell, J
2013 The Social Construction of Meaning. Reading Literature in Urban English Classrooms. London: Routledge.
Cited by
Cited by 53 other publications
Arnesen, Thomas, Eyvind Elstad, Gavriel Salomon & Lars Vavik
2016. Educational Technology and Polycontextual Bridging. In Educational Technology and Polycontextual Bridging, ► pp. 3 ff. 
Bateman, John A
2022. Multimodality, where next?–Some meta-methodological considerations. Multimodality & Society 2:1 ► pp. 41 ff.
Canale, Germán
2019. Laptops and Textbooks as Curriculum Artifacts: Audience, Authorization and Ideologies in the Classroom. In Technology, Multimodality and Learning, ► pp. 137 ff.
Canale, Germán
2019. Toward a Multimodal Socio-Semiotic Account of Learning. In Technology, Multimodality and Learning, ► pp. 41 ff.
Chen, Yixiong & Csilla Weninger
2024. The potential of eye tracking data to strengthen CDA’ explanatory power: the case of multimodal critical discourse analysis of advertising persuasion. Critical Discourse Studies ► pp. 1 ff.
Chen, Yixiong, Csilla Weninger & Fei Victor Lim
2024. Chapter 4. Multimodality. In Less Frequently Used Research Methodologies in Applied Linguistics [Research Methods in Applied Linguistics, 6], ► pp. 49 ff.
De Costa, Peter I., Magda Tigchelaar & Yaqiong Cui
de Souza Júnior, Jaime
2021. Transmediatisation of the Covid-19 crisis in Brazil: The emergence of (bio-/geo-)political repertoires of (re-)interpretation. Humanities and Social Sciences Communications 8:1
Dias, Reinildes & Daniela Gomes de Araújo Nóbrega
2022. MULTILITERACIES AND LEARNING: AN ANALYSIS OF AN ENGLISH HIGH SCHOOL TEXTBOOK FOR THE BRAZILIAN CONTEXT. Open Minds International Journal 3:1 ► pp. 80 ff.
Ding, Ai-Chu Elisha, Krista Glazewski & Faridah Pawan
2022. Language teachers and multimodal instructional reflections during video-based online learning tasks. Technology, Pedagogy and Education 31:3 ► pp. 293 ff.
Forzani, Elena E. & Chu N. Ly
2022. Beyond Multimodality to Multiplicity: Developing More Equitable and Relevant Literacy Learning Spaces for Young Children. The Reading Teacher 75:5 ► pp. 611 ff.
Govender, Navan
2023. Critical Literacies and the Conditions of Decolonial Possibility. In Young People Shaping Democratic Politics, ► pp. 235 ff.
Govender, Navan & Jennifer Farrar
2023. Introduction to critical literacies & social media. English in Education 57:4 ► pp. 252 ff.
Govender, Navan N.
2020. Critical Transmodal Pedagogies: Student Teachers Play with Genre Conventions. Multimodal Communication 9:1
Grapin, Scott E. & Lorena Llosa
2020. Toward an integrative framework for understanding multimodal L2 writing in the content areas. Journal of Second Language Writing 47 ► pp. 100711 ff.
Gruba, Paul
2020. What Does Language Testing Have to Offer to Multimodal Listening?. In Another Generation of Fundamental Considerations in Language Assessment, ► pp. 43 ff.
Guberman, Ainat
2023. Nurturing Writing of Narrative and Expository Texts at the Preschool Level. In Development of Writing Skills in Children in Diverse Cultural Contexts, ► pp. 243 ff.
Hiippala, Tuomo
2017. An overview of research within the Genre and Multimodality framework. Discourse, Context & Media 20 ► pp. 276 ff.
Ho, Wing Yee Jenifer
2019. Self-Directed Language Learning: A Semiotic Analysis of a Language Learning App. In Virtual Sites as Learning Spaces, ► pp. 295 ff.
Ho, Wing Yee Jenifer
2022. ‘Coming here you should speak Chinese’: the multimodal construction of interculturality in YouTube videos. Language and Intercultural Communication 22:6 ► pp. 662 ff.
Katsampoxaki-Hodgetts, Kallia
2024. Graphical abstracts’ pedagogical implications: Skills & challenges in visual remediation. English for Specific Purposes 73 ► pp. 141 ff.
Kendrick, Maureen
2022. Multimodality and Multilingualism. In Multilingualism and Education, ► pp. 168 ff.
Li, Luyao & Jing Huang
Ly, Chu N. & Elena E. Forzani
2023. Let's Learn from Them: Using the Integrative Multimodal Literacy Assessment Tool to Support Instruction for Young Children. The Reading Teacher 77:3 ► pp. 407 ff.
Mabrook, Radwa
2021. Between Journalist Authorship and User Agency: Exploring the Concept of Objectivity in VR Journalism. Journalism Studies 22:2 ► pp. 209 ff.
Martin, J. R.
2024. Intradisciplinarity: can one theory do it all?. Frontiers in Communication 8
Matsumoto, Yumi & Suresh Canagarajah
2020. The use of gesture, gesture hold, and gaze in trouble-in-talk among multilingual interlocutors in an English as a lingua franca context. Journal of Pragmatics 169 ► pp. 245 ff.
May, Laura, Thomas Crisp, Gary E. Bingham, Renée S. Schwartz, Mario T. Pickens & Kate Woodbridge
2020. The Durable, Dynamic Nature of Genre and Science: A Purpose‐Driven Typology of Science Trade Books. Reading Research Quarterly 55:3 ► pp. 399 ff.
Pantaleo, Sylvia
2019. Metalepsis in Elementary Students’ Multimodal Narrative Representations.
Research in the Teaching of English
54:1 ► pp. 8 ff.
Pantaleo, Sylvia
2021. The designing of ocean threat comics by elementary students. Multimodal Communication 10:3 ► pp. 229 ff.
Pantaleo, Sylvia
2023. Kindergarten Children’s Talk About Illustration Techniques in an Almost Wordless Picturebook. Early Childhood Education Journal
Parkwell, Corina
2019. Emoji as social semiotic resources for meaning-making in discourse: Mapping the functions of the toilet emoji in Cher’s tweets about Donald Trump. Discourse, Context & Media 30 ► pp. 100307 ff.
Pascual, Daniel, Ramón Plo-Alastrué & Isabel Corona
2023. Digital Scholarly Practices in Scientific Communication: Paths and Goals in Research Dissemination. In Digital Scientific Communication, ► pp. 3 ff.
Pereira, Íris Susana Pires
2023. Focalization and ideological meanings in a digital multimodal narrative: implications for the construction of the pedagogy of multiliteracies. Multimodal Communication 12:2 ► pp. 123 ff.
Pires Pereira, Íris Susana
2022. A multiliteracies approach to online reading to learn: a case study. Pedagogies: An International Journal 17:2 ► pp. 119 ff.
Pereira, Íris Susana Pires, Maitê Gil & Clecio Bunzen
2023. Wordless Picturebooks as Resources for the Construction of the Pedagogy of Multiliteracies. The Case of Migrants, by Issa Watanabe. Children's Literature in Education
Pires Pereira, Íris Susana & Ângela Campos
2023. Constructing the pedagogy of multiliteracies. The role of focalisation in the development of critical analysis of multimodal narratives. Language and Education 37:1 ► pp. 105 ff.
Porto, Melina
2023. Multimodal artistic recreations of the ecological university in the Global South. Pedagogy, Culture & Society ► pp. 1 ff.
Quintero, Edgar Nunes, Catarina Lelis & Bernardo Meza Guzman
2024. The Strange and Not So Strange About Disney’s Strange World: A Visceral and Social Semiotic Analysis. In Advances in Design and Digital Communication IV [Springer Series in Design and Innovation, 35], ► pp. 345 ff.
Ravelli, Louise J.
2019. Multimodality and the register of disciplinary History. Language, Context and Text. The Social Semiotics Forum 1:2 ► pp. 341 ff.
Santos, Marcelo Sousa
2017. A OFERTA DO LETRAMENTO E A GARANTIA DE FUTUROS SOCIAIS: ANÁLISE DAS POLÍTICAS DE LETRAMENTO DA UNESCO E DE SUAS IDEOLOGIAS LINGUÍSTICAS. Trabalhos em Linguística Aplicada 56:2 ► pp. 641 ff.
Schur, Yaron & Ainat Guberman
2024. Conceptual Change of ‘Teaching’ among Experienced Teachers after Studying Attentive Teaching. Education Sciences 14:3 ► pp. 231 ff.
Sergeev, Dmitrii
2023. Latina American and Caribbean Single Mothers in Anglophone Children’s Picturebooks. Children's Literature in Education
Tan, Lynde, Ching Sing Chai, Feng Deng, Chun Ping Zheng & Nur Arifah Drajati
2019. Examining pre-service teachers’ knowledge of teaching multimodal literacies: a validation of a TPACK survey. Educational Media International 56:4 ► pp. 285 ff.
Tan, Lynde & Tingjia Wang
2023. Fostering Student Agency and Teacher Leadership Through Augmented Reality-Mediated Learning. In Using Innovative Literacies to Develop Leadership and Agency [Advances in Media, Entertainment, and the Arts, ], ► pp. 100 ff.
Thurlow, Crispin
2024. Working beside/s words: A case study in the partiality and provinciality of language. Multimodality & Society 4:1 ► pp. 11 ff.
Wei, Li
2020. System remembers Gunther Kress: Gunther Kress’ contribution to the study of language, technology, and learning. System 93 ► pp. 102289 ff.
Wei, Li & Wing Yee (Jenifer) Ho
2018. Language LearningSans Frontiers: A Translanguaging View. Annual Review of Applied Linguistics 38 ► pp. 33 ff.
Wei, Li & Angel M. Y. Lin
2019. Translanguaging classroom discourse: pushing limits, breaking boundaries. Classroom Discourse 10:3-4 ► pp. 209 ff.
Wiśniewska, Danuta
2021. Linguistic landscape, murals and language learning. In Angewandte Linguistik – Neue Herausforderungen und Konzepte, ► pp. 429 ff.
Wong, May
2019. Social Semiotics: Setting the Scene. In Multimodal Communication, ► pp. 1 ff.
Zhu, Jianbin & Yingying Gu
2022. The Effects of Multimodal Communication Classroom on the Learning Performance of EFL Learners from the Perspective of Translanguaging. International Journal of Education and Humanities 4:3 ► pp. 223 ff.
Ávila-López, Javier & Fernando D. Rubio-Alcalá
2023. Multiliteracies and multilingualism in adult migrants. European Journal of Language Policy 15:2 ► pp. 167 ff.
This list is based on CrossRef data as of 28 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.