Multimodal body, multimodal mind, multimodal communication

Turner, Mark

doi:10.1075/milcc.9.05tur

Part of

Dynamism in Metaphor and Beyond
Edited by Herbert L. Colston, Teenie Matlock and Gerard J. Steen
[Metaphor in Language, Cognition, and Communication 9] 2022
► pp. 95–108

Multimodal body, multimodal mind, multimodal communication

Mark Turner | Institute Professor and Professor of Cognitive Science | Case Western Reserve University

This chapter reviews prospects for improving the scientific study of communication by developing big data infrastructure and computational, technical, and statistical tools. Such big data systems are typically free of the methodological threats to validity and underpowered datasets common in lab experiments.

Keywords: multimodality, big data, infrastructure, computational tools, communication

Article outline

An illumination
The foundation of communication is multimodal
To know a communicative system is to know a relational network of form-meaning pairs and how they blend
Communication science and multimodal big data
An infrastructure for research on multimodal communication
Examples
Conclusion
References

Published online: 9 September 2022

https://doi.org/10.1075/milcc.9.05tur

References (32)

References

Alexander, James. (2011). Blending in mathematics. Semiotica. 2011(187), 1–48. ISSN (Online) 1613-3692, ISSN (Print) 0037-1998,

Craddock, R. Cameron, Daniel S. Margulies, Pierre Bellec, B. Nolan Nichols, Sarael Alcauter, Fernando A. Barrios, Yves Burnod, et al. (2016). Brainhack: a collaborative workshop for the open neuroscience community. GigaScience 5:16.

Davies, Mark. (2015). The importance of robust corpora in providing more realistic descriptions of variation in English grammar. Linguistics Vanguard 1(1). 305–312.

Fauconnier, Gilles and Mark Turner. (1996). Blending as a Central Process of Grammar. In Conceptual Structure, Discourse, and Language. Edited by Adele Goldberg. Stanford: Center for the Study of Language and Information (CSLI), 113–130 [distributed by Cambridge University Press]

. (2002). The Way We Think: Conceptual Blending and the Mind’s Hidden Complexities. Basic Books.

Hampe, Beate, Irene Mittelberg, Peter Uhrig, and Mark Turner. (2017). There-constructions ‘in the wild.’ Presentation at the 2017 International Conference on Multimodal Communication. 9–11 July. University of Osnabrück.

Hinnell, Jennifer. (2018). The multimodal marking of aspect: The case of five periphrastic auxiliary constructions in North American English. Cognitive Linguistics, 29(4), 773–806.

Hoffmann, Thomas. (2017). From constructions to Construction Grammar. In: Barbara Dancygier, ed. The Cambridge Handbook of Cognitive Linguistics. Cambridge: Cambridge University Press, 284–309.

Hoffmann, Thomas & Graeme Trousdale. (2013). The Oxford Handbook of Construction Grammar. Oxford University Press.

Israel, Michael. (1996). The Way Constructions Grow. In Adele Goldberg, ed. Conceptual Structure, Discourse and Language. Stanford: CSLI, 217–230.

Joo, Jungseock, Francis F. Steen & Song-Chun Zhu. (2015). Automated facial trait judgment and election outcome prediction: Social dimensions of face. Proceedings of the IEEE International Conference on Computer Vision, 3712–3720. Los Alamitos, CA: Institute of Electrical and Electronics Engineers.

Kay, Paul, and Charles J. Fillmore. (1999). Grammatical constructions and linguistic generalizations: The What’s X doing Y? construction. Language 75: 1–33.

Li, Weixin, Jungseock Joo, Hang Qi, & Song-Chun Zhu. (2017). Joint image-text news topic detection and tracking by multimodal topic and-or graph. IEEE Transactions on Multimedia 19(2). 367–381.

Murphy, G.. (1996) On metaphoric representations. Cognition, 60, 173–204.

Nesset, T., A. Endresen, L. Janda, A. Makarova, F. Steen & M. Turner. (2013). How ‘here and ‘now in Russian and English establish joint attention in TV news broadcasts. Russian Linguistics 37. 229–251.

Pagán Cánovas, C. & J. Valenzuela. (2017). Timelines and multimodal constructions: Facing new challenges. Linguistic Vanguard 3(s1).

Stamenković, Dušan. (2015). “The Effects of Animated Visual Stimuli on the Process of Conceptual Blending in Riddle Solving.” Facta Universitatis. Vol. 13, No 1, 2015, pp. 11–19. Available at SSRN: [URL]

Stec, Kashmiri & Eve Sweetser. (2013). Borobudur and Chartres: Religious spaces as performative real-space blends. In R. Caballero & J. E. Diaz Vera (Eds.), Sensuous cognition Berlin: Mouton De Gruyter. pp. 265–291.

Steen, Francis F., Anders Hougaard, Jungseock Joo, Inés Olza, Cristóbal Pagán Cánovas, Anna Pleshakova, Soumya Ray, Peter Uhrig, Javier Valenzuela, Jacek Woźny and Mark Turner. (2018). Toward an Infrastructure for Data-driven Multimodal Communication Research. Linguistics Vanguard. Volume 4, Issue1, Pages 1–9.

Steen, Francis and Mark Turner. (2013). “Multimodal Construction Grammar.” In Borkent, Michael, Barbara Dancygier, and Jennifer Hinnell, editors, Language and the Creative Mind. Stanford, CA: CSLI Publications. Pages 255–274.

Turner, Mark. (1996). The Literary Mind: The Origins of Thought and Language. New York: Oxford University Press.

. (2008). Frame Blending. In Frames, Corpora, and Knowledge Representation, edited by Rema Rossini Favretti. Bologna: Bononia University Press, 2008. 13–32.

. (2014). The Origin of Ideas: Blending, Creativity, and the Human Spark. New York: Oxford University Press.

. (2015). Blending in Language and Communication. In Ewa Dabrowska & Dagmar Divjak (eds.), Handbook of cognitive linguistics, Chapter 10. Berlin: De Gruyter Mouton.

. (2017a). Multimodal form-meaning pairs for blended classic joint attention. Linguistics Vanguard 3(s1).

. (2017b). Polytropos and communication in the wild. In Barbara Dancygier (ed.), The Cambridge handbook of cognitive linguistics, 93–98. Cambridge: Cambridge University Press.

. (2018). The Role of Creativity in Multimodal Construction Grammar. Zeitschrift für Anglistik und Amerikanistik. Volume 66, Issue 3. Pages 357–370.

Turner, Mark, Maíra Avelar, & Milene Mendes de Oliveira. (2019). “Blended Classic Joint Attention and Multimodal Deixis.” Signo. 44:79, pages 03–09.

Turner, Mark. (2020). “Constructions and Creativity.” In Hoffmann, Thomas, editor, Construction Grammar and creativity: Evolution, psychology, and cognitive science, a special issue of Cognitive Semiotics, 13:1.

Zima, Elisabeth. (2014a). English multimodal motion constructions. A construction grammar perspective. Studies van de BKL – Travaux du CBL – Papers of the LSB, Volume 8. [URL]

. (2014b). Gibt es multimodale Konstruktionen? Eine Studie zu [V(motion) in circles] und [all the way from X PREP Y]. Gesprächsforschung – Online-Zeitschrift zur verbalen Interaktion 15. 1–48. [URL]

. (2017). Multimodal constructional resemblance. The case of English circular motion constructions. In Francisco Ruiz de Mendoza, Alba Luzondo & Paula Pérez-Sobrino (eds.), Constructing families of constructions. Human Cognitive Processing Series. Amsterdam: John Benjamins.