Article published In:
International Journal of Corpus Linguistics: Online-First Articles

Towards a corpus-based description of speech-gesture units of meaning

The case of the circular gesture

| Macau University of Science and Technology
| University of Nottingham

The theories and methods in corpus linguistics (CL) have had an impact on numerous areas in applied linguistics. However, the interface between CL and multimodal speech-gesture studies remains underexplored. One fundamental question is whether it is possible, and even appropriate, to apply the theories and paradigms established based on textual data to multimodal data. To explore this, we examine how CL can assist investigating lexico-grammatical patterns of speech co-occurring with a recurrent gesture (i.e. the circular gesture). Sinclair’s (1996) unit of meaning model is used to describe the co-gestural speech patterns. The study draws on a subset of the Nottingham Multimodal Corpus, in which 570 instances of circular gestures and their co-occurring speech are identified and analysed. We argue that Sinclair’s unit of meaning model can be extended to include speech-gesture patterns, and that those descriptions enable a more nuanced understanding of meaning in context.

Keywords: circular gestures, multimodal corpus analysis, co-gestural speech patterns, unit of meaning model
Article outline
  • 1.Introduction
  • 2.Recurrent gestures
  • 3.The unit of meaning model
  • 4.Corpus and methods
    • 4.1Data selection
    • 4.2Speech-gesture annotations and concordance
    • 4.3Cluster analysis
    • 4.4Concordance plot
    • 4.5Sampling techniques to develop a lexico-grammatical framework
    • 4.6Cross-checking and inter-rater reliability tests of speech patterns
  • 5.Analysis
    • 5.1The multimodal unit of meaning of the circular gesture: Overall distribution
    • 5.2Main category: Colligation
      • 5.2.1Clause
      • 5.2.2Verb phrase (+ ___)
      • 5.2.3Noun phrase (+ ___)
      • 5.2.4Modification (+ noun phrase)
      • 5.2.5Dysfluent speech
      • 5.2.6Prepositional phrase
      • 5.2.7Predictive (+___)
      • 5.2.8Discourse marker
      • 5.2.9Conjunction
    • 5.3Main category: Semantic preference
      • 5.3.1Deixis
      • 5.3.2Negation
      • 5.3.3Confirmation
    • 5.4Main category: Semantic prosody
      • 5.4.1Intensification
      • 5.4.2Vagueness
      • 5.4.3Monitoring mutual ground
    • 5.5Sub-category: Semantic preference (‘on-going process’)
  • 6.Towards a multimodal unit of meaning
  • Acknowledgements
  • References
Published online: 13 October 2022
DOI logo
https://doi.org/10.1075/ijcl.20174.che
References
Adolphs, S., & Carter, R.
(2013) Spoken Corpus Linguistics: From Monomodal to Multimodal. Routledge. DOI logoGoogle Scholar
Adolphs, S., Brown, B., Carter, R., Crawford, P., & Sahota, O.
(2004) Applying corpus linguistics in a health care context. Journal of Applied Linguistics, 1 (1), 9–28. DOI logoGoogle Scholar
Anthony, L.
(2019) AntConc (Version 3.5.8) [Computer Software]. [URL]
Atkins, S., & Harvey, K.
(2010) How to use corpus linguistics in the study of health communication. In A. O’Keeffe & M. McCarthy (Eds.), The Routledge Handbook of Corpus Linguistics (pp. 605–619). Routledge. DOI logoGoogle Scholar
Calbris, G.
(2011) Elements of Meaning in Gesture. John Benjamins. DOI logoGoogle Scholar
Cánovas, C. P., Valenzuela, J., Carrión, D. A., Olza, I., & Ramscar, M.
(2020) Quantifying the speech-gesture relation with massive multimodal datasets: Informativity in time expressions. PLoS ONE, 15 (6), 1–18. DOI logoGoogle Scholar
Chen, Y., Adolphs, S., & Knight, D.
(2021) Multimodal Discourse Analysis. In J. Hardy & E. Friginal (Eds.), Routledge Handbook of Corpus Approaches to Discourse Analysis (pp. 98–115). Routledge.Google Scholar
Cienki, A., & Müller, C.
(2008) Metaphor, gesture, and thought. In R. W. Gibbs (Ed.), The Cambridge Handbook of Metaphor and Thought (pp. 483–501). Cambridge University Press. DOI logoGoogle Scholar
Clift, R.
(2020) Stability and visibility in embodiment: The ‘Palm Up’ in interaction. Journal of Pragmatics, 169 1, 190–205. DOI logoGoogle Scholar
Debras, C.
(2017) The shrug. Gesture, 16 (1), 1–34. DOI logoGoogle Scholar
(2021) Multimodal profiles of je (ne) sais pas in spoken French. Journal of Pragmatics, 182 1, 42–62. DOI logoGoogle Scholar
ELAN (Version 6.0) [Computer software]
(2020) Nijmegen: Max Planck Institute for Psycholinguistics, The Language Archive. [URL]
Feyaerts, K., Rominger, C., Lackner, H. K., Brône, G., Jehoul, A., Oben, B., & Papousek, I.
(2022)  In your face? Exploring multimodal response patterns involving facial responses to verbal and gestural stance-taking expressions. Journal of Pragmatics, 190 1, 6–17. DOI logoGoogle Scholar
Gullberg, M., & McCafferty, S. G.
(2008) Introduction to gesture and SLA: Toward an integrated approach. Studies in Second Language Acquisition, 30 (02), 133–146. DOI logoGoogle Scholar
Hinnell, J.
(2018) The multimodal marking of aspect: The case of five periphrastic auxiliary constructions in North American English. Cognitive Linguistics, 29 (4), 773–806. DOI logoGoogle Scholar
Hunston, S.
(2002) Corpora in Applied Linguistics. Cambridge University Press. DOI logoGoogle Scholar
(2007) Semantic prosody revisited. International Journal of Corpus Linguistics, 12(2), 249–268. DOI logoGoogle Scholar
Kendon, A.
(2004) Gesture: Visible Action as Utterance. Cambridge University Press. DOI logoGoogle Scholar
Kita, S.
(2009) Cross-cultural variation of speech-accompanying gesture: A review. Language and Cognitive Processes, 24 (2), 145–167. DOI logoGoogle Scholar
Kita, S., van Gijn, I., & van der Hulst, H.
(1998) Movement phases in signs and co-speech gestures, and their transcription by human coders. In I. Wachsmuth & M. Fröhlich (Eds.), Gesture and Sign Language in Human-Computer Interaction: International Gesture Workshop Bielefeld, Germany, September 17–19, 1997 Proceedings (pp. 23–35). Springer. DOI logoGoogle Scholar
Knight, D.
(2011) Multimodality and Active Listenership: A Corpus Approach. Bloomsbury.Google Scholar
Kok, K.
(2017) Functional and temporal relations between spoken and gestured components of language. International Journal of Corpus Linguistics, 22 (1), 1–26. DOI logoGoogle Scholar
Ladewig, S. H.
(2011) Putting the cyclic gesture on a cognitive basis. CogniTextes, 6 1. DOI logoGoogle Scholar
(2014) Recurrent gestures. In C. Müller, A. Cienki, F. Ellen, S. H. Ladewig, D. McNeill, & J. Bressem (Eds.), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction (Volume 2) (pp. 1558–1574). De Gruyter Mouton.Google Scholar
Lausberg, H., & Sloetjes, H.
(2009) Coding gestural behavior with the NEUROGES-ELAN system. Behavior Research Methods, Instruments, & Computers, 41 (3), 841–849. DOI logoGoogle Scholar
Lin, P., & Chen, Y.
(2020) 5 Multimodality I: Speech, prosody and gesture. In S. Adophs & D. Knight (Eds.), The Routledge Handbook of Digital Humanities (pp. 66–84). Routledge. DOI logoGoogle Scholar
Louw, B.
(1993) Irony in the text or insincerity in the writer? The diagnostic potential of semantic prosodies. In M. Baker, G. Grancis, & E. Tognini-Bonelli (Eds.), Text and Technology: In Honour of John Sinclair (pp. 157–176). John Benjamins. DOI logoGoogle Scholar
McNeill, D.
(2005) Gesture and Thought. The University of Chicago Press. DOI logoGoogle Scholar
Mahlberg, M.
(2013) Corpus Stylistics and Dickens’s Fiction. Routledge. DOI logoGoogle Scholar
Müller, C., Bressem, J., & Ladewig, S. H.
(2013) Towards a grammar of gestures: A form-based view. In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, & S. Teßendorf (Eds.), Body-Language-Communication: An International Handbook on Multimodality in Human Interaction (Volume 1) (pp. 707–733). De Gruyter Mouton. DOI logoGoogle Scholar
Müller, Cornelia
(2004) The Palm-Up-Open-Hand. A case of a gesture family? In C. Müller & R. Posner (Eds.), The Semantics and Pragmatics of Everyday Gestures: Proceedings of the Berlin Conference April 1998 (pp. 233–256). Weidler.Google Scholar
Schegloff, E. A.
(1984) On some gestures’ relation to talk. In J. M. Atkinson & J. Heritage (Eds.), Structures of Social Action (pp. 266–296). Cambridge University Press.Google Scholar
Sinclair, J.
(1996) The search for units of meaning. Textus, 9 (1), 1000–1032.Google Scholar
(2004) Trust the Text: Language, Corpus and Discourse. Routledge. DOI logoGoogle Scholar
Streeck, J.
(2009) Gesturecraft: The Manu-facture of Meaning. John Benjamins. DOI logoGoogle Scholar
Zima, E.
(2017) On the multimodality of [all the way from X PREP Y]. Linguistics Vanguard, 3 (s1), 1–12. DOI logoGoogle Scholar