Article published In:
Theory in Applied Linguistics Research: Critical approaches to production, performance and participationEdited by Theresa Lillis
[AILA Review 28] 2015
► pp. 49–71
Edited by Theresa Lillis
[AILA Review 28] 2015
► pp. 49–71
Semiotic work
Applied Linguistics and a social semiotic account of Multimodality
This article imagines a tussle between Multimodality, focused on ‘modes’, and Applied Linguistics (AL), based on ‘language’. A Social Semiotic approach to MM treats speech and writing as modes with distinct affordances, and, as all modes, treats them as ‘partial’ means of communication. The implications of partiality confound long-held assumptions of the sufficiency of ‘language’ for all communicational needs: an assumption shared by AL. Given MM’s plurality of modes and the diversity of audiences, design moves into focus, with a shift from competent performance to apt design. Principles of composition — e.g. linearity versus modularity — become crucial, raising the question at the heart of this paper: how do AL and MM deal with the shape of the contemporary semiotic landscape?
Keywords: social semiotics, multimodality, modularity, audience, writing, linearity, design, affordances
Available under the Creative Commons Attribution-NonCommercial (CC BY-NC) 4.0 license.
For any use beyond this license, please contact the publisher at [email protected].
References (34)
Coulthard, M. 1992. Forensic Discourse Analysis. Advances in Spoken Discourse Analysis. London: Routledge,
Ellis, R. 2008. The Study of Second Language Acquisition. Oxford: OUP.
Fisiak, J. 1981. Contrastive Linguistics and the Language Teacher [Language Teaching Methodology Series]. Oxford: Pergamon Press. 
Hartmann, R.R.K. (ed) 2003. Lexicography. Critical Concepts. London: Routledge.
König, E. & Gast, V. 2009. Understanding English-German Contrasts. Berlin: Erich Schmidt.
Kress, G. 2010. Multimodality. A Social Semiotic Approach to Contemporary Communication. London: Routledge.
Kress, G. & van Leeuwen, T. 2001. Multimodal Discourse. London: Edward Arnold.
Kress, G., Jewitt, C., Ogborn, J. & Tsatsarelis, C. 2001. Multimodal Teaching and Learning. The Rhetorics of the Science Classroom. London: Continuum.
Lado, R. 1957. Second Language Acquisition. Ann Arbor MI: University of Michigan.
Here I list key works which have informed my discussion of Multimodality in this paper.
Barthes, R. 1973. Mythologies. St. Albans: Paladin.
Bauman, Z. 2000. Liquid Modernity. Cambridge: Polity Press.
Bezemer, J. & Kress, G. 2015. The textbook in a changing multimodal landscape. In Language in Multimodal Contexts, N. Klug & H. Stöckl (eds). Berlin: De Gruyter.
Bezemer, J., Cope, A., Kress, G. & Kneebone, R. 2014. Holding the scalpel: Achieving surgical care in a learning environment. Journal of Contemporary Ethnography 43(1): 38–63.
Bezemer, J. & Kress, G. 2014. Touch: A resource for making meaning. Australian Journal of Language and Literacy 37(2): 77–85.
Bezemer, J., Kress, G., Cope, A. & Kneebone, R. 2012. Learning in the operating theatre: A social semiotic perspective. In Work-based Learning in Clinical Settings: Insights from Socio-cultural Perspectives, V. Cook, C. Daly & M. Newman (eds), 125–141. Abingdon: Radcliffe.
Bezemer, J. 2008. Displaying orientation in the classroom: Students’ multimodal responses to teacher instructions. Linguistics and Education 19(2): 166–178.
Bezemer, J. & Kress, G. 2008. Writing in multimodal texts: A social semiotic account of designs for learning. Written Communication 25(2): 166–195.
Franks, A. 1996. Drama education, the body and representation (or, the mystery of the missing bodies). Research in Drama Education: The Journal of Applied Theatre and Performance 11: 105–119.
Jewitt, C. (ed.) 2014. The Routledge Handbook of Multimodal Analysis. London: Routledge.
Kress, G. 1971. Sentence complexity in contrastive linguistics. In (ERIC) Papers in Contrastive Linguistics, G. Nickel (ed.). Cambridge: CUP.
. 1997. Before Writing: Rethinking the Paths to Literacy. London: Routledge.
. 2000. Text as the punctuation of semiosis: Pulling at some of the threads. In Intertextuality and the Media. From Genre to Everyday Life, U.H. Meinhof & J. Smith (eds), 132–153. Manchester: Manchester University Press.
Kress, G. & van Leeuwen, T. 2006. Reading Images: A Grammar of Visual Design. London: Routledge.
Kress, G., Jewitt, C., Bourne, J., Franks, A., Hardcastle, J., Jones, K. & Reid, E. 2005. English in Urban Classrooms: Multimodal Perspectives on Teaching and Learning. London: RoutledgeFalmer.
Lindstrand, F. & Insulander, E. 2014. Setting the ground for engagement: Multimodal perspectives on exhibition design. Designs for Learning 5(1-2): 30–49.
Mavers, D. 2011. Children’s Drawing and Writing: The Remarkable in the Unremarkable. London: Routledge.
New London Group. 1996. A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review 66(1): 60–92.
Newfield, D. 2014. Transformation, transduction and the transmodal moment. In The Routledge Handbook of Multimodal Analysis, C. Jewitt (ed.), 100–113. London: Routledge.
Pelletier, C. 2007. Learning through design: Subjectivity and meaning in young people’s computer game production work. PhD dissertation, University of London.
Stein, P. 2008. Multimodal Pedagogies in diverse Classrooms. Representation, Rights and Resources. New York NY: Routledge.
Unsworth, L. (ed.) 2008. Multimodal Semiotics: Functional Analysis in Contexts of Education. London: Continuum.
Wulf, C. et al. 2010. Ritual and Identity. The Staging and Performing of Rituals in the Lives of Young People. London: Tufnell.
Yandell, J. 2013. The Social Construction of Meaning. Reading Literature in Urban English Classrooms. London: Routledge.
Cited by (61)
Cited by 61 other publications
Chen, Yixiong & Csilla Weninger
2024. The potential of eye tracking data to strengthen CDA’ explanatory power: the case of multimodal critical discourse analysis of advertising persuasion. Critical Discourse Studies ► pp. 1 ff. 
Chen, Yixiong, Csilla Weninger & Fei Victor Lim
2024. Chapter 4. Multimodality. In Less Frequently Used Research Methodologies in Applied Linguistics [Research Methods in Applied Linguistics, 6], ► pp. 49 ff.
Govender, Navan, Saima Salehjee & Clinton D. van der Merwe
2024. Doing critical disciplinary literacies in teacher education: A pedagogical framework. African Journal of Teacher Education and Development 3:1
Katsampoxaki-Hodgetts, Kallia
2024. Graphical abstracts’ pedagogical implications: Skills & challenges in visual remediation. English for Specific Purposes 73 ► pp. 141 ff.
Legel, Mascha, Gloria Soto, Nicola Grove, Annalu Waller, Stijn Deckers, Hans van Balkom, Ronald Spanjers, Christopher Norrie & Bert Steenbergen
2024. A tutorial: self-created film as a semiotic resource in AAC. Augmentative and Alternative Communication ► pp. 1 ff.
Martin, J. R.
2024. Intradisciplinarity: can one theory do it all?. Frontiers in Communication 8
Paiva, Francis Arthuso & Vicente de Lima Neto
2024. Análise sociossemiótica multimodal do grid e do layout de memes: transformações e reproduções em disputa. Revista Brasileira de Linguística Aplicada 24:4
Quintero, Edgar Nunes, Catarina Lelis & Bernardo Meza Guzman
2024. The Strange and Not So Strange About Disney’s Strange World: A Visceral and Social Semiotic Analysis. In Advances in Design and Digital Communication IV [Springer Series in Design and Innovation, 35], ► pp. 345 ff.
Rahardi, R Kunjana, Wahyudi Rahmat, Refa Lina Tiawati & Yuliana Setyaningsih
Schur, Yaron & Ainat Guberman
2024. Conceptual Change of ‘Teaching’ among Experienced Teachers after Studying Attentive Teaching. Education Sciences 14:3 ► pp. 231 ff.
Thurlow, Crispin
2024. Working beside/s words: A case study in the partiality and provinciality of language. Multimodality & Society 4:1 ► pp. 11 ff.
Wagner, Marie L.
2024. Deconstructing Identity: Empowering Student Diversity Through Critical Literacy for Enhanced Transitions. TEACHING Exceptional Children
Wyse, Dominic & Charlotte Hacking
2024. Decoding, reading and writing: the double helix theory of teaching. Literacy 58:3 ► pp. 256 ff.
Xiong, Yuan & Leonardo O. Munalim
2024. Relational Processes and Social Projections in Facebook Selfie-Quotation Juxtaposition: An Exploratory Study. Human Affairs
Govender, Navan
2023. Critical Literacies and the Conditions of Decolonial Possibility. In Young People Shaping Democratic Politics, ► pp. 235 ff.
Govender, Navan & Jennifer Farrar
2023. Introduction to critical literacies & social media. English in Education 57:4 ► pp. 252 ff.
Guberman, Ainat
2023. Nurturing Writing of Narrative and Expository Texts at the Preschool Level. In Development of Writing Skills in Children in Diverse Cultural Contexts, ► pp. 243 ff.
Ly, Chu N. & Elena E. Forzani
2023. Let's Learn from Them: Using the Integrative Multimodal Literacy Assessment Tool to Support Instruction for Young Children. The Reading Teacher 77:3 ► pp. 407 ff.
Pascual, Daniel, Ramón Plo-Alastrué & Isabel Corona
2023. Digital Scholarly Practices in Scientific Communication: Paths and Goals in Research Dissemination. In Digital Scientific Communication, ► pp. 3 ff.
Pereira, Íris Susana Pires
2023. Focalization and ideological meanings in a digital multimodal narrative: implications for the construction of the pedagogy of multiliteracies. Multimodal Communication 12:2 ► pp. 123 ff.
Pires Pereira, Íris Susana
2022. A multiliteracies approach to online reading to learn: a case study. Pedagogies: An International Journal 17:2 ► pp. 119 ff.
Pereira, Íris Susana Pires, Maitê Gil & Clecio Bunzen
2023. Wordless Picturebooks as Resources for the Construction of the Pedagogy of Multiliteracies. The Case of Migrants, by Issa Watanabe. Children's Literature in Education
Pires Pereira, Íris Susana & Ângela Campos
2023. Constructing the pedagogy of multiliteracies. The role of focalisation in the development of critical analysis of multimodal narratives. Language and Education 37:1 ► pp. 105 ff.
Porto, Melina
2023. Multimodal artistic recreations of the ecological university in the Global South. Pedagogy, Culture & Society ► pp. 1 ff.
Porto, Melina
2024. How Can Language Education Contribute to Securing a Livable Planet?. TESOL Quarterly
Sergeev, Dmitrii
2023. Latina American and Caribbean Single Mothers in Anglophone Children’s Picturebooks. Children's Literature in Education
Tan, Lynde & Tingjia Wang
2023. Fostering Student Agency and Teacher Leadership Through Augmented Reality-Mediated Learning. In Using Innovative Literacies to Develop Leadership and Agency [Advances in Media, Entertainment, and the Arts, ], ► pp. 100 ff.
Ávila-López, Javier & Fernando D. Rubio-Alcalá
2023. Multiliteracies and multilingualism in adult migrants. European Journal of Language Policy 15:2 ► pp. 167 ff.
Bateman, John A
2022. Multimodality, where next?–Some meta-methodological considerations. Multimodality & Society 2:1 ► pp. 41 ff.
Dias, Reinildes & Daniela Gomes de Araújo Nóbrega
2022. MULTILITERACIES AND LEARNING: AN ANALYSIS OF AN ENGLISH HIGH SCHOOL TEXTBOOK FOR THE BRAZILIAN CONTEXT. Open Minds International Journal 3:1 ► pp. 80 ff.
Ding, Ai-Chu Elisha, Krista Glazewski & Faridah Pawan
2022. Language teachers and multimodal instructional reflections during video-based online learning tasks. Technology, Pedagogy and Education 31:3 ► pp. 293 ff.
Forzani, Elena E. & Chu N. Ly
2022. Beyond Multimodality to Multiplicity: Developing More Equitable and Relevant Literacy Learning Spaces for Young Children. The Reading Teacher 75:5 ► pp. 611 ff.
Kendrick, Maureen
2022. Multimodality and Multilingualism. In Multilingualism and Education, ► pp. 168 ff.
Zhu, Jianbin & Yingying Gu
2022. The Effects of Multimodal Communication Classroom on the Learning Performance of EFL Learners from the Perspective of Translanguaging. International Journal of Education and Humanities 4:3 ► pp. 223 ff.
de Souza Júnior, Jaime
2021. Transmediatisation of the Covid-19 crisis in Brazil: The emergence of (bio-/geo-)political repertoires of (re-)interpretation. Humanities and Social Sciences Communications 8:1
Li, Luyao & Jing Huang
Mabrook, Radwa
2021. Between Journalist Authorship and User Agency: Exploring the Concept of Objectivity in VR Journalism. Journalism Studies 22:2 ► pp. 209 ff.
Wiśniewska, Danuta
2021. Linguistic landscape, murals and language learning. In Angewandte Linguistik – Neue Herausforderungen und Konzepte, ► pp. 429 ff.
Govender, Navan N.
2020. Critical Transmodal Pedagogies: Student Teachers Play with Genre Conventions. Multimodal Communication 9:1
Grapin, Scott E. & Lorena Llosa
2020. Toward an integrative framework for understanding multimodal L2 writing in the content areas. Journal of Second Language Writing 47 ► pp. 100711 ff.
Gruba, Paul
2020. What Does Language Testing Have to Offer to Multimodal Listening?. In Another Generation of Fundamental Considerations in Language Assessment, ► pp. 43 ff.
Matsumoto, Yumi & Suresh Canagarajah
2020. The use of gesture, gesture hold, and gaze in trouble-in-talk among multilingual interlocutors in an English as a lingua franca context. Journal of Pragmatics 169 ► pp. 245 ff.
May, Laura, Thomas Crisp, Gary E. Bingham, Renée S. Schwartz, Mario T. Pickens & Kate Woodbridge
2020. The Durable, Dynamic Nature of Genre and Science: A Purpose‐Driven Typology of Science Trade Books. Reading Research Quarterly 55:3 ► pp. 399 ff.
Wei, Li
2020. System remembers Gunther Kress: Gunther Kress’ contribution to the study of language, technology, and learning. System 93 ► pp. 102289 ff.
Canale, Germán
2019. Laptops and Textbooks as Curriculum Artifacts: Audience, Authorization and Ideologies in the Classroom. In Technology, Multimodality and Learning, ► pp. 137 ff.
Canale, Germán
2019. Toward a Multimodal Socio-Semiotic Account of Learning. In Technology, Multimodality and Learning, ► pp. 41 ff.
Ho, Wing Yee Jenifer
2019. Self-Directed Language Learning: A Semiotic Analysis of a Language Learning App. In Virtual Sites as Learning Spaces, ► pp. 295 ff.
Ho, Wing Yee Jenifer
2022. ‘Coming here you should speak Chinese’: the multimodal construction of interculturality in YouTube videos. Language and Intercultural Communication 22:6 ► pp. 662 ff.
Pantaleo, Sylvia
2019. Metalepsis in Elementary Students’ Multimodal Narrative Representations.
Research in the Teaching of English
54:1 ► pp. 8 ff.
Pantaleo, Sylvia
2021. The designing of ocean threat comics by elementary students. Multimodal Communication 10:3 ► pp. 229 ff.
Pantaleo, Sylvia
2024. Kindergarten Children’s Talk About Illustration Techniques in an Almost Wordless Picturebook. Early Childhood Education Journal 52:7 ► pp. 1281 ff.
Parkwell, Corina
2019. Emoji as social semiotic resources for meaning-making in discourse: Mapping the functions of the toilet emoji in Cher’s tweets about Donald Trump. Discourse, Context & Media 30 ► pp. 100307 ff.
Ravelli, Louise J.
2019. Multimodality and the register of disciplinary History. Language, Context and Text. The Social Semiotics Forum 1:2 ► pp. 341 ff.
Tan, Lynde, Ching Sing Chai, Feng Deng, Chun Ping Zheng & Nur Arifah Drajati
2019. Examining pre-service teachers’ knowledge of teaching multimodal literacies: a validation of a TPACK survey. Educational Media International 56:4 ► pp. 285 ff.
Wei, Li & Angel M. Y. Lin
2019. Translanguaging classroom discourse: pushing limits, breaking boundaries. Classroom Discourse 10:3-4 ► pp. 209 ff.
Wong, May
2019. Social Semiotics: Setting the Scene. In Multimodal Communication, ► pp. 1 ff.
Wei, Li & Wing Yee (Jenifer) Ho
2018. Language LearningSans Frontiers: A Translanguaging View. Annual Review of Applied Linguistics 38 ► pp. 33 ff.
Hiippala, Tuomo
2017. An overview of research within the Genre and Multimodality framework. Discourse, Context & Media 20 ► pp. 276 ff.
Santos, Marcelo Sousa
2017. A OFERTA DO LETRAMENTO E A GARANTIA DE FUTUROS SOCIAIS: ANÁLISE DAS POLÍTICAS DE LETRAMENTO DA UNESCO E DE SUAS IDEOLOGIAS LINGUÍSTICAS. Trabalhos em Linguística Aplicada 56:2 ► pp. 641 ff.
Arnesen, Thomas, Eyvind Elstad, Gavriel Salomon & Lars Vavik
2016. Educational Technology and Polycontextual Bridging. In Educational Technology and Polycontextual Bridging, ► pp. 3 ff.
De Costa, Peter I., Magda Tigchelaar & Yaqiong Cui
This list is based on CrossRef data as of 9 october 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.