Cover not available
Article published In:
Functions of Language
Vol. 23:2 (2016) ► pp.149–178

The grammatical potential of co-speech gesture

A Functional Discourse Grammar perspective

| Vrije Universiteit Amsterdam

In recent years, consensus has been growing that certain co-speech gestures must be incorporated in grammatical theory. Contrary to traditional beliefs, gestural expression has been found to be systematic in various ways, to exhibit some degree of compositional structure and to be to a certain extent language-specific. Most current work on the gesture-grammar interface is, nonetheless, rather programmatic in character or lacks an appropriate theoretical framework. This paper shows that Functional Discourse Grammar (FDG) provides a promising model for understanding the grammar-gesture relationship in a more comprehensive and detailed way than has thus far been achieved. It first evaluates how a FDG perspective on language structure aligns with the ways gestures have been studied. Then it demonstrates that various types of co-speech manual gestures can be classified (perhaps even exhaustively) according to the pragmatic and semantic layers recognized in FDG. On the basis of this survey, a first sketch of a multi-channel (speech-gesture) version of FDG’s architecture is presented. Finally, some challenges are put forward that arise when taking the multimodal grammar enterprise seriously. Altogether, the content of this paper suggests that a multimodal perspective on grammar can be mutually beneficial for functionally oriented linguists and gesture researchers.

Published online: 6 October 2016
DOI logo
https://doi.org/10.1075/fol.23.2.01kok
References
Alahverdzhieva, Katya
2013 Alignment of speech and co-speech gesture in a constraint-based grammar. Edinburgh: University of Edinburgh PhD thesis.Google Scholar
Allwood, Jens
2002 Bodily communication dimensions of expression and content. In Inger Karlsson, Björn Granström & David House (eds.), Multimodality in language and speech systems, 7–26. Dordrecht: Kluwer. DOI logoGoogle Scholar
Andrén, Mats
2010 Children’s gestures from 18 to 30 months. Lund: Lund University PhD thesis.Google Scholar
Anstey, Matthew P
2008 Functional Discourse Grammar: Multifunctional problems and constructional solutions. Linguistics 46(4). 831–859. DOI logoGoogle Scholar
Bavelas, Janet B., Nicole Chovil, Douglas A. Lawrie & Allan Wade
1992 Interactive gestures. Discourse Processes 15(4). 469–489. DOI logoGoogle Scholar
Beattie, Geoffrey & Heather Shovelton
1999 Mapping the range of information contained in the iconic hand gestures that accompany spontaneous speech. Journal of Language and Social Psychology 18(4). 438–462. DOI logoGoogle Scholar
Bergmann, Kirsten & Stefan Kopp
2010 Systematicity and idiosyncrasy in iconic gesture use: Empirical analysis and computational modeling. In Stefan Kopp & Ipke Wachsmuth (eds.), Gesture in embodied communication and human-computer interaction, 182–194. Berlin: Springer. DOI logoGoogle Scholar
Bressem, Jana
2012 Repetitions in gesture: Structures, functions, and cognitive aspects. Frankfurt (Oder): European University Viadrina PhD thesis.Google Scholar
Butler, Christopher S
2008 Cognitive adequacy in structural-functional theories of language. Language Sciences 30(1). 1–30. DOI logoGoogle Scholar
Calbris, Geneviève
1990 The semiotics of French gestures. Bloomington, IN: Indiana University Press.Google Scholar
2008 From left to right: Coverbal gestures and their symbolic use of space. In Alan Cienki & Cornelia Müller (eds.), Metaphor and gesture, 27–53. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Chomsky, Noam
1983 Noam Chomsky’s views on the psychology of language and thought. In Robert W. Rieber (ed.), Dialogues on the psychology of language and thought, 29–63. New York, NY: Plenum.Google Scholar
Cienki, Alan
2005 Image schemas and gesture. In Beate Hampe & Joseph E. Grady (eds.), From perception to meaning: Image schemas in cognitive linguistics, 421–441. Berlin: Mouton. DOI logoGoogle Scholar
2012 Usage events of spoken language and the symbolic units we (may) abstract from them. In Krzysztof Kosecki & Janusz Badio (eds.), Cognitive processes in language, 149–158. Frankfurt am Main: Lang.Google Scholar
2015 Spoken language usage events. Language and Cognition 7(4). 499–514. DOI logoGoogle Scholar
Connolly, John H
2010 Accommodating multimodality in Functional Discourse Grammar. Web Papers in Functional Discourse Grammar 831. 1–18.Google Scholar
Cooperrider, Kensy & Rafael Núñez
2012 Nose-pointing: Notes on a facial gesture of Papua New Guinea. Gesture 12(2). 103–129. DOI logoGoogle Scholar
De Ruiter, Jan-Peter
2000 The production of gesture and speech. In David McNeill (ed.), Language and gesture, 331–357. Cambridge: CUP. DOI logoGoogle Scholar
Debras, Camille & Alan Cienki
2012 Some uses of head tilts and shoulder shrugs during human interaction, and their relation to stancetaking. Proceedings of the International Conference on Social Computing (SocialCom) , 932–937.
Dik, Simon C
1986 On the notion ‘Functional Explanation’. Belgian Journal of Linguistics 11. 11–52. DOI logoGoogle Scholar
Duncan, Susan D
2002 Gesture, verb aspect, and the nature of iconic imagery in natural discourse. Gesture 2(2). 183–206. DOI logoGoogle Scholar
Efron, David
1972 Gesture, race and culture. The Hague: Mouton.Google Scholar
Ekman, Paul & Wallace V. Friesen
1969 Nonverbal leakage and clues to deception. Psychiatry 321. 88–106. DOI logoGoogle Scholar
2004 On linear segmentation and combinatorics in co-speech gesture: A symmetry-dominance construction in Lao fish trap descriptions. Semiotica 1491. 57–124.Google Scholar
2009 The anatomy of meaning: Speech, gesture, and composite utterances. Cambridge: CUP. DOI logoGoogle Scholar
Everett, Daniel
In preparation. Sculpting language: A review of the David McNeill Gesture Trilogy. Available online at [URL].
Feyereisen, Pierre, Michèle van de Wiele & Fabienne Dubois
1988 The meaning of gestures: What can be understood without speech? European Bulletin of Cognitive Psychology 81. 3–25.Google Scholar
Firbas, Jan
1964 On defining the theme in functional sentence analysis. Travaux linguistiques de Prague 11. 267–280.Google Scholar
Fricke, Ellen
2008 Grundlagen einer multimodalen Grammatik des Deutschen. Frankfurt (Oder): European University Viadrina PhD thesis.Google Scholar
2009 Multimodal attribution: How gestures are syntactically integrated into spoken language. Paper presented at the International workshop on Gesture and Speech in Interaction (GESPIN), Poznań.
2012 Grammatik multimodal: wie Wörter und Gesten zusammenwirken. Berlin: De Gruyter. DOI logoGoogle Scholar
2013 Towards a unified grammar of gesture and speech: A multimodal approach. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Teßendorf (eds.), 733–754.Google Scholar
Habets, Boukje, Sotaro Kita, Zeshu Shao, Asli Özyürek & Peter Hagoort
2011 The role of synchrony and ambiguity in speech–gesture integration during comprehension. Journal of Cognitive Neuroscience 23(8). 1845–1854. DOI logoGoogle Scholar
Harrison, Simon
2009 Grammar, gesture, and cognition: The case of negation in English. Bordeaux: University of Bordeaux PhD thesis.Google Scholar
Hassemer, Julius, Gina Joue, Klaus Willmes & Irene Mittelberg
2011 Dimensions and mechanisms of form constitution: Towards a formal description of gestures. Proceedings of the 2nd Conference on Gesture and Speech in Interaction (GESPIN) .
Haviland, John B
1993 Anchoring, iconicity, and orientation in Guugu Yimithirr pointing gestures. Journal of Linguistic Anthropology 31. 3–45. DOI logoGoogle Scholar
Hengeveld, Kees & Evelien Keizer
2011 Non-straightforward communication. Journal of Pragmatics 43(7). 1962–1976. DOI logoGoogle Scholar
Hengeveld, Kees & J. Lachlan Mackenzie
2006 Functional discourse grammar. In Keith Brown (ed.), Encyclopaedia of language and linguistics, vol. 41, 668–676. Oxford: Elsevier. DOI logoGoogle Scholar
2008 Functional Discourse Grammar: A typologically-based theory of language structure. Oxford: OUP. DOI logoGoogle Scholar
2014 Grammar and context in Functional Discourse Grammar. Pragmatics 24(2). 203–227. DOI logoGoogle Scholar
Kendon, Adam
1972 Some relationships between body motion and speech. In Aaron W. Siegman & Benjamin Pope (eds.), Studies in dyadic communication, 177–210. Elmsford, NY: Pergamon. DOI logoGoogle Scholar
1988 How gestures can become like words. In Fernando Poyatos (ed.), Cross-cultural perspectives in nonverbal communication, 131–141. Toronto: C. J. Hogrefe.Google Scholar
1992 Some recent work from Italy on quotable gestures (Emblems). Journal of Linguistic Anthropology 21. 92–108. DOI logoGoogle Scholar
1995 Gestures as illocutionary and discourse structure markers in Southern Italian conversation. Journal of Pragmatics 23(3). 247–279. DOI logoGoogle Scholar
2004 Gesture: Visible action as utterance. Cambridge: CUP. DOI logoGoogle Scholar
Kendon, Adam, Thomas A. Sebeok & Jean Umiker-Sebeok
1981 Nonverbal communication, interaction, and gesture: Selections from Semiotica. Berlin: de Gruyter. DOI logoGoogle Scholar
Kendon, Adam & Laura Versante
2003 Pointing by hand in Neapolitan. In Sotaro Kita (ed.), 109–137. Pointing: Where language, culture, and cognition meet. Mahwah, NJ: Erlbaum.Google Scholar
Kita, Sotaro
2000 How representational gestures help speaking. In David McNeill (ed.), Language and gesture, 162–185. Cambridge: CUP. DOI logoGoogle Scholar
2003 Pointing: Where language, culture, and cognition meet. New York, NY: Erlbaum. DOI logoGoogle Scholar
2009 Cross-cultural variation of speech-accompanying gesture: A review. Language and Cognitive Processes 24(2). 145–167. DOI logoGoogle Scholar
Kita, Sotaro & Asli Özyürek
2003 What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48(1). 16–32. DOI logoGoogle Scholar
Kress, Gunther R
2000 Multimodality: Challenges to thinking about language. Tesol Quarterly 34(2). 337–340. DOI logoGoogle Scholar
Kress, Gunther R. & Theo van Leeuwen
1996 Reading images: The grammar of visual design. London: Routledge.Google Scholar
Kroon, Caroline
2005 The relationship between grammar and discourse: Evidence from the Latin particle quidem . Latina Lingua. 576–590.Google Scholar
Ladewig, Silva H
2011 Putting the cyclic gesture on a cognitive basis. CogniTextes. Revue de l’Association française de linguistique cognitive 61. Available online at [URL].Google Scholar
2012 Syntactic and semantic integration of gestures into speech: Structural, cognitive, and conceptual aspects. Frankfurt (Oder): European University Viadrina PhD thesis.Google Scholar
Ladewig, Silva H. & Jana Bressem
2013 New insights into the medium hand: Discovering recurrent structures in gestures. Semiotica 1971. 203–231.Google Scholar
Langacker, Ronald W
2001 Discourse in cognitive grammar. Cognitive Linguistics 12(2). 143–188. DOI logoGoogle Scholar
2008 Cognitive grammar: A basic introduction. Oxford: OUP. DOI logoGoogle Scholar
Levelt, Wim J.M
1989 Speaking: From intention to articulation. Cambridge, MA: MIT Press.Google Scholar
Levinson, Stephen C
2003 Space in language and cognition: Explorations in cognitive diversity. Cambridge: CUP. DOI logoGoogle Scholar
Levy, Elena T. & David McNeill
1992 Speech, gesture, and discourse. Discourse Processes 15(3). 277–301. DOI logoGoogle Scholar
Lücking, Andy, Kirsten Bergmann, Hahn Florian, Stefan Kopp & Hannes Rieser
2010 The Bielefeld speech and gesture alignment corpus (SaGA). Proceedings of the LREC 2010 workshop on Multimodal corpora–advances in capturing, coding and analyzing multimodality.
Mackenzie, J. Lachlan
2010 More tiles on the roof: Further thoughts on incremental language production. In Kasper Boye & Elisabeth Engberg-Pedersen (eds.), Language usage and language structure, 263–294. Berlin: Mouton.Google Scholar
McNeill, David
1992 Hand and mind. What the hands reveal about thought. Chicago: University of Chicago Press.Google Scholar
2000 Language and gesture. Cambridge: CUP. DOI logoGoogle Scholar
2003 Aspects of aspect. Gesture 3(1). 1–17. DOI logoGoogle Scholar
2005 Gesture and thought. Chicago, IL: University of Chicago Press. DOI logoGoogle Scholar
McNeill, David & Susan D. Duncan
2000 Growth points in thinking-for-speaking. In David McNeill (ed.), 162–185. DOI logoGoogle Scholar
McNeill, David & Elena Levy
1982 Conceptual representations in language activity and gesture. In Robert J. Jarvella & Wolfgang Klein (eds.), Speech, place and action: Studies in deixis and related topics. New York, NY: Wiley.Google Scholar
Mittelberg, Irene & Tanja Mortelmans
2013 Grammaticalized manual actions of giving and holding: From ditransitive to existential constructions in speech and gesture. Paper presented at the International Conference of Cognitive Linguistics (ICLC12) , Edmonton, Canada.
Müller, Cornelia
1998 Iconicity and gesture. In Serge Santi (ed.), Oralité et Gestualité: Communication Multimodale, Interaction, 321–328. Montréal: L’Harmattan.Google Scholar
2004 Forms and uses of the Palm Up Open Hand: A case of a gesture family. In Cornelia Müller & Roland Posner (eds.), The semantics and pragmatics of everyday gestures, 234–256. Berlin: Weidler Verlag.Google Scholar
2013 Gestures as a medium of expression: The linguistic potential of gestures. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Teßendorf (eds.), 202–217.Google Scholar
Müller, Cornelia, Jana Bressem & Silva Ladewig
2013a Towards a grammar of gestures: A form-based view. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Teßendorf (eds.), 707–732.Google Scholar
Müller, Cornelia, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Teßendorf
(eds.) 2013b Body – Language – Communication: An international handbook on multimodality in human interaction, vol. 11. Berlin: Mouton.Google Scholar
Müller, Cornelia, Silva H. Ladewig & Jana Bressem
2013c Gestures and speech from a linguistic perspective: A new field and its history. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Teßendorf (eds.), 55–81.Google Scholar
Muntigl, Peter
2004 Modelling multiple semiotic systems: The case of gesture and speech. In Eija Ventola, Cassily Charles & Martin Kaltenbacher (eds.), Perspectives on multimodality, 31–50. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Okrent, Arika
2002 A modality-free notion of gesture and how it can help us with the morpheme vs. gesture question in sign language linguistics (or at least give us some criteria to work with). In Richard P. Meier, Kearsy Cormier & David Quinto-Pozos (eds.), Modality and structure in signed and spoken languages, 175–198. Cambridge: CUP. DOI logoGoogle Scholar
Özyürek, Asli, Sotaro Kita, Shanley E.M. Allen, Reyhan Furman & Amanda Brown
2005 How does linguistic framing of events influence co-speech gestures? Insights from crosslinguistic variations and similarities. Gesture 5(1–2). 219–240. DOI logoGoogle Scholar
Parrill, Fey
2010 Viewpoint in speech–gesture integration: Linguistic structure, discourse structure, and event structure. Language and Cognitive Processes 251. 650–668. DOI logoGoogle Scholar
Parrill, Fay, Benjamin K. Bergen & Patricia V. Lichtenstein
2013 Grammatical aspect, gesture, and conceptualization: Using co-speech gesture to reveal event representations. Cognitive Linguistics 24(1). 135–158. DOI logoGoogle Scholar
Payrató, Lluís
1993 A pragmatic view on autonomous gestures: A first repertoire of Catalan emblems. Journal of Pragmatics 20(3). 193–216. DOI logoGoogle Scholar
Poggi, Isabella
2008 Iconicity in different types of gestures. Gesture 8(1). 45–61. DOI logoGoogle Scholar
Sadeghipour, Amir & Stefan Kopp
2014 Learning a motor grammar of gesture. Proceedings of the Annual Conference of the Cognitive Science Society , Quebec City, Canada.
Sandoval, Richard
2014 Gestures in native Northern America: Bimodal talk in Arapaho. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Teßendorf (eds.), 1215–1226.Google Scholar
Schoonjans, Steven
2014 Is gesture subject to grammaticalization? Papers of the Linguistic Society of Belgium 81. Available online at [URL]
Sherzer, Joel
1973 Verbal and nonverbal deixis: The pointed lip gesture among the San Blas Cuna. Language in Society 21. 117–131. DOI logoGoogle Scholar
Slama-Cazacu, Tatiana
1976 Nonverbal components in message sequence: ‘Mixed syntax’. In William C. McCormack & Stephen A. Wurm (eds.), Language and man: Anthropological issues, 217–227. The Hague: Mouton.Google Scholar
Sparhawk, Carol M
1978 Contrastive-identificational features of Persian gesture. Semiotica 24(1–2). 49–86. DOI logoGoogle Scholar
Stokoe, William C
1960/2005 Sign language structure: An outline of the visual communication systems of the American deaf. Journal of Deaf Studies and Deaf Education 10(1). 3–37. DOI logoGoogle Scholar
Streeck, Jürgen
1993 Gesture as communication: Its coordination with gaze and speech. Communications Monographs 60(4). 275–299. DOI logoGoogle Scholar
2002 Grammars, words, and embodied meanings: On the uses and evolution of so and like . Journal of Communication 52(3). 581–596. DOI logoGoogle Scholar
2009 Gesturecraft: The manu-facture of meaning. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Sweetser, Eve
1998 Regular metaphoricity in gesture: Bodily-based models of speech interaction. Actes du 16e Congrès International des Linguistes .
Talmy, Leonard
1985 Lexicalization patterns: Semantic structure in lexical forms. In Timothy Shopen (ed.), Language typology and syntactic description Vol. 3: Grammatical categories and the lexicon, 57–149. Cambridge: CUP.Google Scholar
Turner, Mark & Francis Steen
2012 Multimodal Construction Grammar. In Michael Borkent, Barbara Dancygier & Jennifer A.J. Hinnell (eds.), Language and the creative mind, 255–274. Stanford, CA: CSLI Publications.Google Scholar
Webb, Rebecca A
1996 Linguistic features of metaphoric gestures. New York, NY: University of Rochester PhD thesis.Google Scholar
Wilcox, Sherman
2004 Cognitive iconicity: Conceptual spaces, meaning, and gesture in signed language. Cognitive Linguistics 15(2). 119–148. DOI logoGoogle Scholar
Wilcox, Sherman & André Nogueira Xavier
2013 A framework for unifying spoken language, signed language, and gesture. Revista Todas as Letras 15(1). 88–110.Google Scholar
Wilkins, David
2003 Why pointing with the index finger is not a universal (in sociocultural and semiotic terms). In Sotaro Kita (ed.), 171–215.Google Scholar
Zima, Elisabeth
2014 English multimodal motion constructions: A Construction Grammar perspective. Papers of the Linguistic Society of Belgium 81. Available online at [URL]
Cited by

Cited by 6 other publications

Cornish, Francis
2022. Text, discourse, context: A meta-trilogy for discourse analysis. Journal of Pragmatics 199 ► pp. 91 ff. DOI logo
Kok, Kasper I.
2017. Functional and temporal relations between spoken and gestured components of language. International Journal of Corpus Linguistics 22:1 ► pp. 1 ff. DOI logo
Laparle, Schuyler
2022. The Interaction Space. In Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Anthropometry, Human Behavior, and Communication [Lecture Notes in Computer Science, 13319], ► pp. 243 ff. DOI logo
Lepic, Ryan & Corrine Occhino
2018. A Construction Morphology Approach to Sign Language Analysis. In The Construction of Words [Studies in Morphology, 4], ► pp. 141 ff. DOI logo
Martin, J. R. & Michele Zappavigna
2019. Embodied meaning: a systemic functional perspective on paralanguage. Functional Linguistics 6:1 DOI logo
Mittelberg, Irene
2017. Multimodal existential constructions in German: Manual actions of giving as experiential substrate for grammatical and gestural patterns. Linguistics Vanguard 3:s1 DOI logo

This list is based on CrossRef data as of 29 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.