Cover not available
Part of
Why Gesture?: How the hands function in speaking, thinking and communicating
Edited by Ruth Breckinridge Church, Martha W. Alibali and Spencer D. Kelly
[Gesture Studies 7] 2017
► pp. 59–75

Chapter 4
The asymmetric redundancy of gesture and speech

| Tufts University

A number of studies from the last decades have demonstrated that iconic gestures are shaped not only by our mental imagery but also, quite strongly, by structural properties of the accompanying speech. These findings are problematic for the central assumption in the Sketch Model (De Ruiter, 2000) about the function of representational gesture. I suggest a seemingly small but fundamental modification to the processing assumptions in the Sketch Model that not only accommodates the discussed empirical findings, but also explains many other well-known gesture phenomena. The new model also generates new and testable predictions regarding the relationship between gesture and speech.

Article outline
  • Introduction
  • The Sketch Model
  • Effects of language on iconic gesture
    • Gestures are influenced by lexical and syntactic properties of language
    • Gesture and speech go hand in hand
    • Gesture and speech express similar types of information
    • Gestures adapt to the pragmatic aspects of speech
  • Implications for the function of iconic gesture
    • The Interface Model
    • The Sketch Model
  • Discussion
  • Notes
  • References
Published online: 26 April 2017
DOI logo
https://doi.org/10.1075/gs.7.04der
References
Alibali, M., & Goldin-Meadow, S.
1993 “Gesture-speech mismatch and mechanisms of learning: What the hands reveal about a child's state of mind.” Cognitive Psychology 25: 468–523. DOI logoGoogle Scholar
Arbib, M., Liebal, K., & Pika, S.
2008 “Primate vocalization, gesture, and the evolution of human language.” Current Anthropology 49(6): 1063–1076. DOI logoGoogle Scholar
Bangerter, A.
2004 “Using pointing and describing to achieve joint focus of attention in dialogue.” Psychological Science 15(6): 415–419. DOI logoGoogle Scholar
Bangerter, A., De Ruiter, J. P., Mayor, E., & Chevalley, E.
2010 “Gestural and speech patterns in collaborative multimodal referring,” Proceedings of the 4th Conference of the International Society for Gesture Studies. Frankfurt am Oder, Germany.Google Scholar
Bierwisch, M., & Schreuder, R.
1992 “From concepts to lexical items.” Cognition 42: 23–60. DOI logoGoogle Scholar
Butterworth, B., & Hadar, U.
1989 “Gesture, speech, and computational stages: A reply to McNeill.” Psychological Review 96: 168–174. DOI logoGoogle Scholar
Cassell, J., Bickmore, T., Billinghurst, L., Campbell, K., Chang, H., Vilhjalmsson, H., & Yan, H.
1999 “Embodiment in conversational faces: Rea.” Proceedings of the CHI'99 Conference, 520–527. DOI logoGoogle Scholar
Church, R. B., & Goldin-Meadow, S.
1986 “The mismatch between gesture and speech as an index of transitional knowledge.” Cognition 23: 43–71. DOI logoGoogle Scholar
Clark, H. H., & Murphy, G. L.
1983 “Audience design in meaning and reference.” In Language and Comprehension, J. F. Leny & W. Kintsch (eds), 287–299.Google Scholar
Corbalis, M.
2003 From Hand to Mouth: The Origins of Language. Princeton: Princeton University Press.Google Scholar
De Ruiter, J. P.
1998 Gesture and Speech Production. Doctoral Dissertation. Radboud University Nijmegen.Google Scholar
2000 “The production of gesture and speech.” In Language and Gesture, D. McNeill (ed), 284–311. Cambridge, UK: Cambridge University Press. DOI logoGoogle Scholar
2006 “Can gesticulation help aphasic people speak, or rather, communicate?” Advances in Speech-Language Pathology 8(2): 124–127. DOI logoGoogle Scholar
2007 “Postcards from the mind: the relationship between thought, imagistic gesture, and speech.” Gesture 7(1):21–38.Google Scholar
De Ruiter, J. P., Bangerter, A., & Dings, P.
2012 “The interplay between gesture and speech in the production of referring expression: investigating the tradeoff hypothesis.” Topics in Cognitive Science 4: 232–248. DOI logoGoogle Scholar
De Ruiter, J. P., & De Beer, C.
2013 “A critical evaluation of models of gesture and speech production for understanding gesture in aphasia.” Aphasiology 27(9): 1015–1030. DOI logoGoogle Scholar
De Ruiter, J. P., & Wilkins, D. P.
1998 The synchronisation of gesture and speech in Dutch and Arrernte (an Australian Aboriginal language): a cross cultural comparison. Paper presented at the Conférence Oralité et Gestualité, Besançon, France.
Enfield, N. J., Kita, S., & De Ruiter, J. P.
2007 “Primary and secondary pragmatic functions of pointing gestures.” Journal of Pragmatics 39(10): 1722–1741. DOI logoGoogle Scholar
Engle, R. A.
1998 Not channels but composite signals: Speech, gesture, diagrams, and object demonstrations in explanations of mechanical devices. Paper presented at the Twentieth Annual Conference of the Cognitive Science Society, Madison, Wisconsin.
Goldin-Meadow, S., Alibali, M., & Church, R. B.
1993 “Transitions in concept acquisition: Using the hand to read the mind.” Psychological Review 100: 279–297. DOI logoGoogle Scholar
Gullberg, M.
2011 “Multilingual multimodality: Communicative difficulties and their solutions in second language use.” In Embodied Interaction, J. Streeck, C. Goodwin, & C. LeBaron (eds), Cambridge: Cambridge University Press.Google Scholar
Hamming, R. W.
1950 “Error correction and error detection codes.” Bell Systems Technical Journal 29(2): 147–160. DOI logoGoogle Scholar
Kendon, A.
1972 “Some relationships between body motion and speech.” In Studies in Dyadic Communication, A. W. Sigman & B. Pope (eds), New York: Pergamon Press. DOI logoGoogle Scholar
1975 “Gesticulation, speech, and the gesture theory of language origins.” Sign Language Studies 9(1): 349–373. DOI logoGoogle Scholar
1980 “Gesticulation and speech: two aspects of the process of an utterance.“ In The Relationship of Verbal and Nonverbal Communication, M. R. Key (ed), 207–227. The Hague: Mouton.Google Scholar
2004 Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Kita, S.
2003 Pointing: Where Language, Culture, and Cognition Meet: Psychology Press.Google Scholar
Kita, S., & Özyürek, A.
2003 “What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking.” Journal of Memory and Language 48: 16–32. DOI logoGoogle Scholar
Kopp, S., Tepper, P., & Cassell, J.
2004 “Towards integrated microplanning of language and iconic gesture for multimodal output.” Proceedings of 6th International Conference on Multimodal Interfaces. DOI logoGoogle Scholar
Krauss, R. M., Morrel-Samuels, P., & Colasante, C.
1991 “Do conversational hand gestures communicate?” Journal of Personality & Social Psychology 61(5): 743–754. DOI logoGoogle Scholar
Levelt, W. J. M.
1989 Speaking: From Intention to Articulation. Cambridge, Massachusetts: The MIT Press.Google Scholar
Levelt, W. J. M., Roelofs, A., & Meyer, A.
1999 “A theory of lexical access in speech production.” Behavioral and Brain Sciences 22: 1–75. DOI logoGoogle Scholar
Mayberry, R.
2000 “Gesture production during stuttered speech: insights into the nature of gesture-speech integration.” In Language and Gesture, D. McNeill (ed), 199–214. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
McClave, E. Z.
1994 “Gestureal beats: The rhythm hypothesis.” Journal of Psycholinguistic Research 23(1): 45–66. DOI logoGoogle Scholar
McNeill, D.
1985 “So you think gestures are nonverbal?” Psychological Review 92: 350–371. DOI logoGoogle Scholar
1992 Hand and Mind. Chicago, London: The Chicago University Press.Google Scholar
McNeill, D., & Duncan, S.
2000 “Growth points in thinking-for-speaking.” In Language and Gesture, D. McNeill (ed), 141–161. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Morrel-Samuels, P., & Krauss, R. M.
1992 “Word familiarity predicts temporal asynchrony of hand gestures and speech.” Journal of Experimental Psychology: Learning, Memory, and Cognition 18(3): 615–622.Google Scholar
Roelofs, A.
1992 “A spreading activation theory of lemma retrieval in speaking.” Cognition 42: 107–142. DOI logoGoogle Scholar
Rose, M.
2006 “The utility of arm and hand gestures in the treatment of aphasia.” International Journal of Speech-Language Pathology 8(2): 92–109. DOI logoGoogle Scholar
2013 “Releasing the constraints on aphasia therapy: The positive impact of gesture and multimodality treatments.” American Journal of Speech-Language Pathology 22: 227–239. DOI logoGoogle Scholar
Sacks, H., & Schegloff, E. A.
1979 “Two preferences in the organization of reference to persons in conversation and their interaction.” In Everyday Language: Studies in Ethnomethodology, G. Psathas (ed), 15–21. New York: Irvington.Google Scholar
Sacks, H., Schegloff, E. A., & Jefferson, G.
1974 “A simplest systematics for the organization of turn-taking for conversation.” Language 50: 696–735. DOI logoGoogle Scholar
Schegloff, E. A.
1984 “On some gesture relation to speech.” In Structures of Social Action: Studies in Conversational Analysis, J. M. Atkinson & J. Heritage (eds), Cambridge: Cambridge University Press.Google Scholar
Seyfeddinipur, M.
2006 Disfluency: Interrupting speech and gesture. (PhD Thesis), Radboud University Nijmegen.Google Scholar
So, W. C., Kita, S., & Goldin-Meadow, S.
2009 “Using the hands to identify who does what to whom: Gesture and speech go hand-in-hand.” Cognitive Science 33: 115–125. DOI logoGoogle Scholar
Van der Sluis, I., & Krahmer, E.
2007 “Generating multimodal references.” Discourse Processes 44(3): 145–174. DOI logoGoogle Scholar
Cited by

Cited by 12 other publications

Bryła-Cruz, Agnieszka
2019. Communicative responsibility in non-native speech. Overcoming foreign accent in English in religious discourse. Crossroads. A Journal of English Studies :26(3) ► pp. 4 ff. DOI logo
Clough, Sharice & Melissa C. Duff
2020. The Role of Gesture in Communication and Cognition: Implications for Understanding and Treating Neurogenic Communication Disorders. Frontiers in Human Neuroscience 14 DOI logo
Cravotta, Alice, M. Grazia Busà & Pilar Prieto
2019. Effects of Encouraging the Use of Gestures on Speech. Journal of Speech, Language, and Hearing Research 62:9 ► pp. 3204 ff. DOI logo
Cravotta, Alice, Pilar Prieto & M. Grazia Busà
2021. Exploring the effects of restraining the use of gestures on narrative speech. Speech Communication 135 ► pp. 25 ff. DOI logo
de Beer, Carola, Jan P. de Ruiter, Martina Hielscher-Fastabend & Katharina Hogrefe
2019. The Production of Gesture and Speech by People With Aphasia: Influence of Communicative Constraints. Journal of Speech, Language, and Hearing Research 62:12 ► pp. 4417 ff. DOI logo
de Beer, Carola, Katharina Hogrefe, Martina Hielscher‐Fastabend & Jan P. de Ruiter
2020. Evaluating Models of Gesture and Speech Production for People With Aphasia. Cognitive Science 44:9 DOI logo
de Kleine, Naomi, Miranda L. Rose, Michael Weinborn, Robert Knox & Nicolas Fay
2024. Does Gesture Improve the Communication Success of People with Aphasia?: A Systematic Review. Aphasiology 38:3 ► pp. 462 ff. DOI logo
Kita, Sotaro & Karen Emmorey
2023. Gesture links language and cognition for spoken and signed languages. Nature Reviews Psychology 2:7 ► pp. 407 ff. DOI logo
Marentette, Paula, Reyhan Furman, Marcus E. Suvanto & Elena Nicoladis
2020. Pantomime (Not Silent Gesture) in Multimodal Communication: Evidence From Children’s Narratives. Frontiers in Psychology 11 DOI logo
Pouw, Wim, Lisette de Jonge‐Hoekstra, Steven J. Harrison, Alexandra Paxton & James A. Dixon
2021. Gesture–speech physics in fluent speech and rhythmic upper limb movements. Annals of the New York Academy of Sciences 1491:1 ► pp. 89 ff. DOI logo
Stark, Brielle C., Sharice Clough & Melissa Duff
2021. Suggestions for Improving the Investigation of Gesture in Aphasia. Journal of Speech, Language, and Hearing Research 64:10 ► pp. 4004 ff. DOI logo
Wilson, Jack
2024. Pragmatics, Utterance Meaning, and Representational Gesture, DOI logo

This list is based on CrossRef data as of 12 april 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.