Chapter published in:
Dynamism in Metaphor and Beyond
Edited by Herbert L. Colston, Teenie Matlock and Gerard J. Steen
[Metaphor in Language, Cognition, and Communication 9] 2022
► pp. 95–108

Multimodal body, multimodal mind, multimodal communication

| Institute Professor and Professor of Cognitive Science | Case Western Reserve University

This chapter reviews prospects for improving the scientific study of communication by developing big data infrastructure and computational, technical, and statistical tools. Such big data systems are typically free of the methodological threats to validity and underpowered datasets common in lab experiments.

Keywords: multimodality, big data, infrastructure, computational tools, communication
Article outline
  • An illumination
  • The foundation of communication is multimodal
  • To know a communicative system is to know a relational network of form-meaning pairs and how they blend
  • Communication science and multimodal big data
  • An infrastructure for research on multimodal communication
  • Examples
  • Conclusion
  • References
Published online: 09 September 2022
https://doi.org/10.1075/milcc.9.05tur
References
Alexander, James
(2011) Blending in mathematics. Semiotica 2011(187), 1–48. ISSN (Online) 1613-3692, ISSN (Print) 0037-1998, CrossrefGoogle Scholar
Craddock, R. Cameron, Daniel S. Margulies, Pierre Bellec, B. Nolan Nichols, Sarael Alcauter, Fernando A. Barrios, Yves Burnod, et al.
(2016) Brainhack: a collaborative workshop for the open neuroscience community. GigaScience 5:16. CrossrefGoogle Scholar
Davies, Mark
(2015) The importance of robust corpora in providing more realistic descriptions of variation in English grammar. Linguistics Vanguard 1(1). 305–312. CrossrefGoogle Scholar
Fauconnier, Gilles and Mark Turner
(1996) Blending as a Central Process of Grammar. In Conceptual Structure, Discourse, and Language. Edited by Adele Goldberg. Stanford: Center for the Study of Language and Information (CSLI), 113–130 [distributed by Cambridge University Press]Google Scholar
(2002) The Way We Think: Conceptual Blending and the Mind’s Hidden Complexities. Basic Books.Google Scholar
Hampe, Beate, Irene Mittelberg, Peter Uhrig, and Mark Turner
(2017)  There-constructions ‘in the wild.’ Presentation at the 2017 International Conference on Multimodal Communication. 9–11 July. University of Osnabrück.
Hinnell, Jennifer
(2018) The multimodal marking of aspect: The case of five periphrastic auxiliary constructions in North American English. Cognitive Linguistics, 29(4), 773–806. CrossrefGoogle Scholar
Hoffmann, Thomas
(2017) From constructions to Construction Grammar. In: Barbara Dancygier, ed. The Cambridge Handbook of Cognitive Linguistics. Cambridge: Cambridge University Press, 284–309. CrossrefGoogle Scholar
Hoffmann, Thomas & Graeme Trousdale
(2013) The Oxford Handbook of Construction Grammar. Oxford University Press. CrossrefGoogle Scholar
Israel, Michael
(1996) The Way Constructions Grow. In Adele Goldberg, ed. Conceptual Structure, Discourse and Language. Stanford: CSLI, 217–230.Google Scholar
Joo, Jungseock, Francis F. Steen & Song-Chun Zhu
(2015) Automated facial trait judgment and election outcome prediction: Social dimensions of face. Proceedings of the IEEE International Conference on Computer Vision, 3712–3720. Los Alamitos, CA: Institute of Electrical and Electronics Engineers. CrossrefGoogle Scholar
Kay, Paul, and Charles J. Fillmore
(1999) Grammatical constructions and linguistic generalizations: The What’s X doing Y? construction. Language 75: 1–33. CrossrefGoogle Scholar
Li, Weixin, Jungseock Joo, Hang Qi, & Song-Chun Zhu
(2017) Joint image-text news topic detection and tracking by multimodal topic and-or graph. IEEE Transactions on Multimedia 19(2). 367–381. CrossrefGoogle Scholar
Murphy, G.
(1996) On metaphoric representations. Cognition, 60, 173–204. CrossrefGoogle Scholar
Nesset, T., A. Endresen, L. Janda, A. Makarova, F. Steen & M. Turner
(2013) How ‘here and ‘now in Russian and English establish joint attention in TV news broadcasts. Russian Linguistics 37. 229–251. CrossrefGoogle Scholar
Pagán Cánovas, C. & J. Valenzuela
(2017) Timelines and multimodal constructions: Facing new challenges. Linguistic Vanguard 3(s1). CrossrefGoogle Scholar
Stamenković, Dušan
(2015) “The Effects of Animated Visual Stimuli on the Process of Conceptual Blending in Riddle Solving.” Facta Universitatis. Vol. 13, No 1 2015, pp. 11–19. Available at SSRN: https://​ssrn​.com​/abstract​=2496010
Stec, Kashmiri & Eve Sweetser
(2013) Borobudur and Chartres: Religious spaces as performative real-space blends. In R. Caballero & J. E. Diaz Vera (Eds.), Sensuous cognition Berlin: Mouton De Gruyter. pp. 265–291. CrossrefGoogle Scholar
Steen, Francis F., Anders Hougaard, Jungseock Joo, Inés Olza, Cristóbal Pagán Cánovas, Anna Pleshakova, Soumya Ray, Peter Uhrig, Javier Valenzuela, Jacek Woźny and Mark Turner
(2018) Toward an Infrastructure for Data-driven Multimodal Communication Research. Linguistics Vanguard. Volume 4, Issue1, Pages 1–9. CrossrefGoogle Scholar
Steen, Francis and Mark Turner
(2013) “Multimodal Construction Grammar.” In Borkent, Michael, Barbara Dancygier, and Jennifer Hinnell, editors, Language and the Creative Mind. Stanford, CA: CSLI Publications. Pages 255–274.Google Scholar
Turner, Mark
(1996) The Literary Mind: The Origins of Thought and Language. New York: Oxford University Press.Google Scholar
(2008) Frame Blending. In Frames, Corpora, and Knowledge Representation, edited by Rema Rossini Favretti. Bologna: Bononia University Press 2008 13–32.Google Scholar
(2014) The Origin of Ideas: Blending, Creativity, and the Human Spark. New York: Oxford University Press.Google Scholar
(2015) Blending in Language and Communication. In Ewa Dabrowska & Dagmar Divjak (eds.), Handbook of cognitive linguistics, Chapter 10. Berlin: De Gruyter Mouton. CrossrefGoogle Scholar
(2017a) Multimodal form-meaning pairs for blended classic joint attention. Linguistics Vanguard 3(s1). CrossrefGoogle Scholar
(2017b) Polytropos and communication in the wild. In Barbara Dancygier (ed.), The Cambridge handbook of cognitive linguistics, 93–98. Cambridge: Cambridge University Press. CrossrefGoogle Scholar
(2018) The Role of Creativity in Multimodal Construction Grammar. Zeitschrift für Anglistik und Amerikanistik. Volume 66, Issue 3. Pages 357–370. CrossrefGoogle Scholar
Turner, Mark, Maíra Avelar, & Milene Mendes de Oliveira
(2019) “Blended Classic Joint Attention and Multimodal Deixis.” Signo. 44:79, pages 03–09. CrossrefGoogle Scholar
Turner, Mark
(2020) “Constructions and Creativity.” In Hoffmann, Thomas, editor, Construction Grammar and creativity: Evolution, psychology, and cognitive science, a special issue of Cognitive Semiotics, 13:1. CrossrefGoogle Scholar
Zima, Elisabeth
(2014a) English multimodal motion constructions. A construction grammar perspective. Studies van de BKL – Travaux du CBL – Papers of the LSB, Volume 8. http://​uahost​.uantwerpen​.be​/linguist​/SBKL​/sbkl2013​/Zim2013​.pdf
(2014b) Gibt es multimodale Konstruktionen? Eine Studie zu [V(motion) in circles] und [all the way from X PREP Y]. Gesprächsforschung – Online-Zeitschrift zur verbalen Interaktion 15. 1–48. http://​www​.gespraechsforschung​-ozs​.de​/fileadmin​/dateien​/heft2014​/ga​-zima​.pdf
(2017) Multimodal constructional resemblance. The case of English circular motion constructions. In Francisco Ruiz de Mendoza, Alba Luzondo & Paula Pérez-Sobrino (eds.), Constructing families of constructions. Human Cognitive Processing Series. Amsterdam: John Benjamins. CrossrefGoogle Scholar