Cover not available
Article published In:
International Journal of Corpus Linguistics
Vol. 22:1 (2017) ► pp.1–26

Functional and temporal relations between spoken and gestured components of language

A corpus-based inquiry

| Vrije Universiteit Amsterdam

Based on the Bielefeld Speech and Gesture Alignment Corpus (Lücking et al. 2013), this paper presents a systematic comparison of the linguistic characteristics of unimodal (speech only) and multimodal (gesture-accompanied) forms of language use. The results suggest that each of these two modes of expression is characterized by statistical preferences for certain types of words and grammatical categories. The words that are most frequently accompanied by a manual gesture, when controlled for their total frequency, include unspecific spatial lexemes, various deictic words, and particles that express difficulty in word retrieval or formulation. Other linguistic items, including pronouns and verbs of cognition, show a strong dispreference for being gesture-accompanied. The second part of the paper shows that gestures do not occur within a fixed time window relative to the word(s) they relate to, but the preferred temporal distance varies with the type of functional relation that exists between the verbal and gestural channel.

Keywords: gesture, multimodal corpus, distributional analysis, relative frequency ratio
Article outline
  • 1.Introduction
  • 2.Previous research on gesture-accompanied linguistic structures
  • 3.The corpus
  • 4.Methodology
  • 5.Analyses
    • 5.1Lemma-level analyses
    • 5.2POS-based analyses
    • 5.3The effect of the choice of time window
  • 6.Discussion and conclusion
  • Acknowledgements
  • Notes
  • References


This article is currently available as a sample article.

Published online: 28 July 2017
DOI logo
https://doi.org/10.1075/ijcl.22.1.01kok
References
Adolphs, S., & Carter, R.
(2013) Spoken corpus linguistics: From monomodal to multimodal. New York, NY: Routledge. DOI logoGoogle Scholar
Alahverdzhieva, K.
(2013) Alignment of speech and co-speech gesture in a constraint-based grammar. (Unpublished doctoral dissertation), University of Edinburgh, Edinburgh.Google Scholar
Altman, D. G., & Bland, J. M.
(2011) How to obtain the P value from a confidence interval. British Medical Journal, 343(2304). Retrieved from [URL] (last accessed February 2017).
Bergmann, K., Aksu, V., & Kopp, S.
(2011) The relation of speech and gestures: Temporal synchrony follows semantic synchrony. Paper presented at the 2nd Workshop on Gesture and Speech in Interaction, Bielefeld, Germany.
Bergmann, K., & Kopp, S.
(2009) GNetIc – Using Bayesian decision networks for iconic gesture generation. In Z. Ruttkay, M. Kipp, A. Nijholt & H. H. Vilhjálmsson (Eds.), Proceedings of the 9th International Conference on Virtual Agents (pp. 76–89). Amsterdam: Springer.Google Scholar
Bergmann, K., Kopp, S., & Eyssel, F.
(2010) Individualized gesturing outperforms average gesturing – evaluating gesture production in virtual humans. In J. Allbeck, N. Badler, T. Bickmore, C. Pelachaud & A. Safonova (Eds.), Proceedings of the 10th Conference on Intelligent Virtual Agents (pp. 104–117). Philadelphia, PA: Springer. DOI logoGoogle Scholar
Cienki, A.
(2012) Usage events of spoken language and the symbolic units we (may) abstract from them. In K. Kosecki & J. Badio (Eds.), Cognitive Processes in Language (pp. 149–158). Frankfurt am Main: Peter Lang.Google Scholar
Damerau, F. J.
(1993) Generating and evaluating domain-oriented multi-word terms from texts. Information Processing & Management, 29(4), 433–447. DOI logoGoogle Scholar
Diemer, S., Brunner, M. L., & Schmidt, S.
(2016) Compiling computer-mediated spoken language corpora. International Journal of Corpus Linguistics, 21(3), 348–371. DOI logoGoogle Scholar
Enfield, N. J.
(2004) On linear segmentation and combinatorics in co-speech gesture: A symmetry-dominance construction in Lao fish trap descriptions. Semiotica, 149(1–4), 57–124.Google Scholar
Fricke, E.
(2007) Origo, Geste und Raum: Lokaldeixis im Deutschen. Berlin: Walter de Gruyter. DOI logoGoogle Scholar
(2009) Multimodal attribution: How gestures are syntactically integrated into spoken language. Paper presented at the first Gesture and Speech in Interaction conference (GeSpIn), Poznań, Poland.
(2012) Grammatik multimodal: Wie Wörter und Gesten zusammenwirken. Berlin: Walter de Gruyter. DOI logoGoogle Scholar
Hadar, U., & Krauss, R. K.
(1999) Iconic gestures: The grammatical categories of lexical affiliates. Journal of Neurolinguistics, 12(1), 1–12. DOI logoGoogle Scholar
Harrison, S.
(2008) The expression of negation through grammar and gesture. In J. Zlatev, M. Andrén, M. J. Falck & C. Lundmark (Eds.), Studies in Language and Cognition (pp. 405–419). Cambridge: Cambridge Scholars Press.Google Scholar
(2009) Grammar, gesture, and cognition: The case of negation in English (Unpublished doctoral dissertation). University of Bordeaux, Bordeaux, France.Google Scholar
Hinrichs, E., Hinrichs, M., & Zastrow, T.
(2010) WebLicht: Web-based LRT services for German. In S. Kübler (Ed.), Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics 2010 System Demonstrations (pp. 25–29). Uppsala, Sweden.Google Scholar
Hole, D., & Klumpp, G.
(2000) Definite type and indefinite token: The article son in colloquial German. Linguistische Berichte, 182(1), 231–244.Google Scholar
Kendon, A.
(1995) Gestures as illocutionary and discourse structure markers in Southern Italian conversation. Journal of Pragmatics, 23(3), 247–279. DOI logoGoogle Scholar
(2004) Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Knight, D.
(2011) Multimodality and Active Listenership: A Corpus Approach. London: Continuum Books.Google Scholar
Knight, D., Evans, D., Carter, R., & Adolphs, S.
(2009) HeadTalk, HandTalk and the corpus: Towards a framework for multi-modal, multi-media corpus development. Corpora, 4(1), 1–32. DOI logoGoogle Scholar
Kisler, T., Schiel, F., & Sloetjes, H.
(2012) Signal processing via web services: The use case WebMAUS. In E. Hinrichs, H. Neuroth & P. Wittenburg (Eds.), Proceedings of the Service-oriented Architectures (SOAs) workshop at the Digital Humanities Conference 2012 (pp. 30–34). Hamburg, Germany.Google Scholar
Kok, K. I.
(2016) The grammatical potential of co-speech gesture: A Functional Discourse Grammar perspective. Functions of Language, 23(2), 149–178. DOI logoGoogle Scholar
Kok, K. I., Bergmann, K., Cienki, A., & Kopp, S.
(2016) Mapping out the multifunctionality of speakers’ gestures. Gesture, 15(1), 37–59. DOI logoGoogle Scholar
Kok, K. I., & Cienki, A.
(2016) Cognitive Grammar and gesture: Points of convergence, advances and challenges. Cognitive Linguistics, 27(1), 67–100. DOI logoGoogle Scholar
Kopp, S., Bergmann, K., & Wachsmuth, I.
(2008) Multimodal communication from multimodal thinking – towards an integrated model of speech and gesture production. International Journal of Semantic Computing, 2(1), 115–136. DOI logoGoogle Scholar
Krauss, R. M., Chen, Y., & Gottesman, R. F.
(2000) Lexical gestures and lexical access: A process model. In D. McNeill (Ed.), Language and Gesture (pp. 261–283). Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Ladewig, S. H.
(2012) Syntactic and semantic integration of gestures into speech: Structural, cognitive, and conceptual aspects (Unpublished doctoral dissertation). European University Viadrina, Frankfurt (Oder).Google Scholar
Langacker, R. W.
(1987) Foundations of Cognitive Grammar, Volume I: Theoretical Prerequisites. Stanford: Stanford University Press.Google Scholar
Leonard, T., & Cummins, F.
(2011) The temporal relation between beat gestures and speech. Language and Cognitive Processes, 26(10), 1457–1471. DOI logoGoogle Scholar
Levy, E. T., & McNeill, D.
(1992) Speech, gesture, and discourse. Discourse Processes, 15(3), 277–301. DOI logoGoogle Scholar
Loehr, D. P.
(2004) Gesture and intonation (Unpublished doctoral dissertation). Georgetown University, Washington D.C.Google Scholar
Lücking, A., Bergmann, K., Hahn, F., Kopp, S., & Rieser, H.
(2010) The Bielefeld speech and gesture alignment corpus (SaGA). In M. Kipp, J. C. Martin, P. Paggio & D. Heylen (Eds.), Proceedings of the 7th International Conference for Language Resources and Evaluation (pp. 92–98). Valetta, Malta.Google Scholar
(2013) Data-based analysis of speech and gesture: The Bielefeld Speech and Gesture Alignment Corpus (SaGA) and its applications. Journal on Multimodal User Interfaces, 7(1–2), 5–18. DOI logoGoogle Scholar
McCarthy, M., & Carter, R.
(1996) Ten criteria for a spoken grammar. In E. Hinkel & S. Fotos (Eds.), New Perspectives in Grammar Teaching in Second Language Classrooms (pp. 51–75). Mahwah, NJ: Lawrence Erlbaum.Google Scholar
McNeill, D.
(1992) Hand and Mind: What Gestures Reveal about Thought. Chicago, IL: University of Chicago Press.Google Scholar
(2000) Catchments and contexts: Non-modular factors in speech and gesture production. In D. McNeill (Ed.), Language and Gesture (pp. 312–328). Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Morrel-Samuels, P., & Krauss, R. M.
(1992) Word familiarity predicts temporal asynchrony of hand gestures and speech. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18(3), 615–622.Google Scholar
Müller, C., Ladewig, S. H., & Bressem, J.
(2013) Gestures and speech from a linguistic perspective: A new field and its history. In C. Müller, A. Cienki, E. Fricke, S. Ladewig, D. McNeill & J. Bressem (Eds.), Body-Language-Communication: An International Handbook on Multimodality in Human Interaction (Vol. 11, pp. 55–81). Berlin/Boston: De Gruyter Mouton.Google Scholar
Schiller, A., Teufel, S., & Thielen, C.
(1995) Guidelines für das Tagging deutscher Textcorpora mit STTS. Unpublished report. University of Stuttgart.Google Scholar
Schoonjans, S.
(2014a) Is gesture subject to grammaticalization? Papers of the Linguistic Society of Belgium, 81. Retrieved from [URL] (last accessed May 2016).
(2014b) Modalpartikeln als multimodale Konstruktionen. Eine korpusbasierte Kookkurrenzanalyse von Modalpartikeln und Gestik im Deutschen (Unpublished doctoral dissertation). University of Leuven, Leuven, Belgium.Google Scholar
Streeck, J.
(1993) Gesture as communication: Its coordination with gaze and speech. Communications Monographs, 60(4), 275–299. DOI logoGoogle Scholar
(2002) Grammars, words, and embodied meanings: On the uses and evolution of so and like. Journal of Communication, 52(3), 581–596. DOI logoGoogle Scholar
Thurmair, M.
(1989) Modalpartikeln und ihre Kombinationen. Tübingen: Niemeyer. DOI logoGoogle Scholar
Turner, M., & Steen, F.
(2012) Multimodal Construction Grammar. In M. Borkent, B. Dancygier & J. A. J. Hinnell (Eds.), Language and the Creative Mind (pp. 255–274). Stanford, CA: CSLI Publications.Google Scholar
van Son, R., Wesseling, W., Sanders, E., & van den Heuvel, H.
(2008) The IFADV corpus: A free dialog video corpus. Proceedings of the sixth international conference on Language Resources and Evaluation (LREC) (pp. 501–508). Marrakech: European Language Resources Association (ELRA).Google Scholar
Zima, E.
(2014) English multimodal motion constructions. A construction grammar perspective. Papers of the Linguistic Society of Belgium, 81. Retrieved from [URL] (last accessed May 2016).
Cited by

Cited by 6 other publications

Chen, Yaoyao & Svenja Adolphs
2023. Towards a corpus-based description of speech-gesture units of meaning. International Journal of Corpus Linguistics 28:2 ► pp. 172 ff. DOI logo
Holler, Judith & Stephen C. Levinson
2019. Multimodal Language Processing in Human Communication. Trends in Cognitive Sciences 23:8 ► pp. 639 ff. DOI logo
Nirme, Jens, Agneta Gulz, Magnus Haake & Marianne Gullberg
2024. Early or synchronized gestures facilitate speech recall—a study based on motion capture data. Frontiers in Psychology 15 DOI logo
Rühlemann, Christoph
2018. Concluding remarks. In Corpus Linguistics for Pragmatics, ► pp. 199 ff. DOI logo
Rühlemann, Christoph & Alexander Ptak
2023. Reaching beneath the tip of the iceberg: A guide to the Freiburg Multimodal Interaction Corpus. Open Linguistics 9:1 DOI logo
Urbanik, Paweł & Jan Svennevig
2021. Action-Depicting Gestures and Morphosyntax: The Function of Gesture-Speech Alignment in the Conversational Turn. Frontiers in Psychology 12 DOI logo

This list is based on CrossRef data as of 31 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.