Article published In:
GestureVol. 15:1 (2016) ► pp.37–59
Mapping out the multifunctionality of speakers’ gestures
Although it is widely acknowledged that gestures are complex functional elements of human communication, many current functional classification systems are rather rigid, implicitly assuming gestures to perform only one function at any given time. In this paper, we present a theoretical view on the inherent multifunctionality of speakers’ gestures, inspired by frameworks in structural-functional linguistics (Halliday’s Systemic Functional Grammar and Hengeveld & Mackenzie’s Functional Discourse Grammar). Building upon this view, we report on a large-scale internet-based gesture perception study, designed in a way open to the potential for complex multifunctionality of gestural expression. The results provide quantitative support for the view that speakers’ gestures typically contribute to multiple semantic and meta-communicative functions of the ongoing discourse simultaneously. Furthermore, we identify clusters of functions that tend to be combined, as well as correlations between pairs of them. As a whole, this paper achieves some degree of convergence between ecological and experimental views on gesture functionality.
Keywords: communication, gestures, discourse, multifunctionality
References
Bavelas, Janet Beavin, Nicole Chovil, Douglas A. Lawrie, & Allan Wade
(1992) Interactive gestures. Discourse Processes, 15 (4), 469–489. 
Beattie, Geoffrey & Heather Shovelton
(1999) Mapping the range of information contained in the iconic hand gestures that accompany spontaneous speech. Journal of Language and Social Psychology, 18 (4), 438–462.
(2001) An experimental investigation of the role of different types of iconic gesture in communication: A semantic feature approach. Gesture, 1 (2), 129–149.
Bergmann, Kirsten & Stefan Kopp
(2006) Verbal or visual? How information is distributed across speech and gesture in spatial dialog. In David Schlangen & Raquel Fernández (Eds.), Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue (SemDial-10) (pp. 90–97). Potsdam, Germany: Universitätsverlag.
(2009) Increasing expressiveness for virtual agents – Autonomous generation of speech and gesture for spatial description tasks. In Keith S. Decker, Jaime S. Sichman, Carles Sierra, & Cristiano Castelfranchi (Eds.), Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2009) (pp. 361–368). Ann Arbor, MI: IFAAMAS.
Bergmann, Kirsten, Volkan Aksu, & Stefan Kopp
(2011) The relation of speech and gestures: Temporal synchrony follows semantic synchrony. In
Proceedings of the 2nd Workshop on Gesture and Speech in Interaction (GeSpIn 2011)
. Bielefeld, Germany.
Bressem, Jana, Silva H. Ladewig, & Cornelia Müller
(2013) Linguistic Annotation System for Gestures (LASG). In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva Ladewig, David McNeill, & Sedinha Teßendorf (Eds.), Body – language – communication: An international handbook on multimodality in human interaction, Vol. 11 (pp. 1098–1125). Berlin: Mouton de Gruyter.
Bühler, Karl
(1990/1934) Theory of language: The representational function of language, trans. Donald Fraser Goodwin. Amsterdam: John Benjamins (Sprachtheorie. Die Darstellungsfunktion der Sprache. Jena: Gustav Fischer).
Clark, Herbert H. & Jean E. Fox Tree
(2002) Using uh and um in spontaneous speaking. Cognition, 84 (1), 73–111.
Connolly, John H
(2010) Accommodating multimodality in Functional Discourse Grammar. Web Papers in Functional Discourse Grammar, 831, 1–18.
Denis, Michel
(1997) The description of routes: A cognitive approach to the production of spatial discourse. Current Psychology of Cognition, 161, 409–458.
Dik, Simon C
(1989) The theory of functional grammar, Part I: The structure of the clause. Dordrecht: Foris Publications.
Efron, David
(1972) Gesture, race and culture. The Hague: Mouton.
Enfield, Nicholas J
(2009) The anatomy of meaning: Speech, gesture, and composite utterances. Cambridge: Cambridge University Press.
Halliday, Michael A.K
(1985) An introduction to functional grammar. London: Edward Arnold.
Hengeveld, Kees & J. Lachlan Mackenzie
(2008) Functional Discourse Grammar: A typologically-based theory of language structure. Oxford: Oxford University Press.
Holler, Judith & Geoffrey Beattie
(2002) A micro-analytic investigation of how iconic gestures and speech represent core semantic features in talk. Semiotica, 1421, 31–69.
(2003) Pragmatic aspects of representational gestures: Do speakers use them to clarify verbal ambiguity for the listener? Gesture, 3 (2), 127–154.
Holler, Judith & Katie Wilkin
(2009) Communicating common ground: How mutually shared knowledge influences speech and gesture in a narrative task. Language and Cognitive Processes, 24 (2), 267–289.
Humboldt, Wilhelm von
(1905) Gesammelte Schriften, Bd. 41 [The collected works of Wilhelm von Humboldt, Vol. 4]. Berlin: Behr.
Kendon, Adam
(1972) Some relationships between body motion and speech. Studies in Dyadic Communication, 71, 177–210.
(1980) Gesticulation and speech: Two aspects of the process of utterance. In Mary R. Key (Ed.), The relationship of verbal and nonverbal communication (pp. 207–227). The Hague: Mouton.
(1981) Introduction: Current issues in the study of nonverbal communication. In Adam Kendon, Thomas A. Sebeok, & Jean Umiker-Sebeok (Eds.), Nonverbal communication, interaction, and gesture: Selections from Semiotica (pp. 1–53). Berlin: Walter de Gruyter.
(1995) Gestures as illocutionary and discourse structure markers in Southern Italian conversation. Journal of pragmatics, 23 (3), 247–279.
(2004) Gesture: Visible action as utterance. Cambridge: Cambridge University Press.
Kok, Kasper
in press). The grammatical potential of co-speech gesture: A Functional Discourse Grammar perspective. Functions of Language.
Lausberg, Hedda & Han Sloetjes
(2009) Coding gestural behavior with the NEUROGES-ELAN system. Behavior Research Methods, 41 (3), 841–849.
Levy, Elena T. & David McNeill
(1992) Speech, gesture, and discourse. Discourse Processes, 15 (3), 277–301.
Loehr, Daniel P
(2004)
Gesture and intonation
. PhD thesis, Georgetown University.
Lücking, Andy, Kirsten Bergman, Florian Hahn, Stefan Kopp, & Hannes Rieser
(2013) Data-based analysis of speech and gesture: The Bielefeld Speech and Gesture Alignment Corpus (SaGA) and its applications. Journal on Multimodal User Interfaces, 7 (1/2), 5–18.
Martinec, Radan
(2004) Gestures that co‐occur with speech as a systematic resource: The realization of experiential meanings in indexes. Social Semiotics, 14 (2), 193–213.
McNeill, David
(1992) Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.
(2005) Gesture and thought. Chicago: University of Chicago Press.
Müller, Cornelia
(1998) Redebegleitende Gesten. Kulturgeschichte – Theorie – Sprachvergleich. Berlin: Berlin Verlag A. Spitz.
(2013) Gestures as a medium of expression: The linguistic potential of gestures. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva Ladewig, David McNeill, & Sedinha Tessendorf (Eds.), Body – language – communication: An international handbook on multimodality in human interaction, Vol. 11 (pp. 202–217). Berlin: Mouton de Gruyter.
Müller, Cornelia, Silva Ladewig, & Jana Bressem
(2013) Gestures and speech from a linguistic perspective: A new field and its history. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva Ladewig, David McNeill, & Sedinha Tessendorf (Eds.), Body – language – communication: An international handbook on multimodality in human interaction, Vol. 11 (pp. 55–81). Berlin: Mouton de Gruyter.
Muntigl, Peter
(2004) Modelling multiple semiotic systems: The case of gesture and speech. In Eija Ventola, Cassily Charles, & Martin Kaltenbacher (Eds.), Perspectives on multimodality (pp. 31–49). Amsterdam: John Benjamins.
Parrill, Fey
(2010) Viewpoint in speech–gesture integration: Linguistic structure, discourse structure, and event structure. Language and Cognitive Processes, 25 (5), 650–668.
Sperber, Dan & Deirdre Wilson
(1986) Relevance: Communication and cognition. Oxford: Blackwell.
Streeck, Jürgen
Van Valin, Robert D., Jr
Wundt, Wilhelm
(1973) The language of gestures. The Hague: Mouton.
Cited by
Cited by 19 other publications
Afonso, Susana & Ana Sofia Afonso
2019. Accessing Science Museum Educators' Discourse Through Multimodal Narratives. In Multimodal Narratives in Research and Teaching Practices [Advances in Educational Technologies and Instructional Design, ], ► pp. 340 ff. 
Austin, Elizabeth E. & Naomi Sweller
2018. Gesturing Along the Way: Adults’ and Preschoolers’ Communication of Route Direction Information. Journal of Nonverbal Behavior 42:2 ► pp. 199 ff.
Cienki, Alan
2022. The study of gesture in cognitive linguistics: How it could inform and inspire other research in cognitive science. WIREs Cognitive Science 13:6
Feng, Shichang, Zhiquan Feng & Liujuan Cao
2019. Many-to-One Gesture-to-Command Flexible Mapping Approach for Smart Teaching Interface Interaction. IEEE Access 7 ► pp. 179517 ff.
Freigang, Farina & Stefan Kopp
2016. This Is What’s Important – Using Speech and Gesture to Create Focus in Multimodal Utterance. In Intelligent Virtual Agents [Lecture Notes in Computer Science, 10011], ► pp. 96 ff.
Gawne, Lauren, Chelsea Krajcik, Helene N. Andreassen, Andrea L. Berez-Kroeker & Barbara F. Kelly
Hassemer, Julius & Bodo Winter
2018. Decoding Gestural Iconicity . Cognitive Science 42:8 ► pp. 3034 ff.
Hodge, Gabrielle & Lindsay Ferrara
2022. Iconicity as Multimodal, Polysemiotic, and Plurifunctional. Frontiers in Psychology 13
Jehlička, Jakub & Eva Lehečková
2020. Multimodal Event Construals: The Role of Co-Speech Gestures in English vs. Czech Interactions. Zeitschrift für Anglistik und Amerikanistik 68:4 ► pp. 351 ff.
Kok, Kasper I.
2017. Functional and temporal relations between spoken and gestured components of language. International Journal of Corpus Linguistics 22:1 ► pp. 1 ff.
Laparle, Schuyler
2022. The Interaction Space. In Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Anthropometry, Human Behavior, and Communication [Lecture Notes in Computer Science, 13319], ► pp. 243 ff.
Laparle, Schuyler
2023. Gradient at-issueness and semiotic complexity in gesture: a response. Theoretical Linguistics 49:3-4 ► pp. 261 ff.
Lederer, Jenny
Leonteva, Anna V., Alan Cienki & Olga V. Agafonova
2023. Metaphoric gestures in simultaneous interpreting. Russian Journal of Linguistics 27:4 ► pp. 820 ff.
Lopez-Ozieblo, Renia
2023. Observations on the Progression of Gestures with L2 Proficiency: A Call for Further Research. Journal of Psycholinguistic Research 52:2 ► pp. 381 ff.
Martin, J. R. & Michele Zappavigna
2019. Embodied meaning: a systemic functional perspective on paralanguage. Functional Linguistics 6:1
Molodychenko, Evgeni N.
2023. Levels of context: how textual analysis becomes discourse analysis: the case of Internet lifestyle media. Slovo.ru: Baltic accent 14:1 ► pp. 107 ff.
Parrill, Fey & Kashmiri Stec
Uhrig, Peter
2022. Hand gestures with verbs of throwing: Collostructions, style and metaphor. Yearbook of the German Cognitive Linguistics Association 10:1 ► pp. 99 ff.
This list is based on CrossRef data as of 30 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.