Cover not available
Part of
Why Gesture?: How the hands function in speaking, thinking and communicating
Edited by Ruth Breckinridge Church, Martha W. Alibali and Spencer D. Kelly
[Gesture Studies 7] 2017
► pp. 243–265

Chapter 11
Exploring the boundaries of gesture-speech integration during language comprehension

 | Colgate University
The present review explores the integrated relationship between gesture and speech during language comprehension. Taking a broad view, it presents a conceptual framework that approaches the comprehension of gesture and speech along three different dimensions: (1) components of language (semantics, pragmatics, phonetics and syntax), (2) levels of analysis (social, cognitive, and biological) and (3) timeframes of integration (online, moment-to-moment, developmental). The evidence suggests that some linguistic components (e.g., concrete semantic and pragmatic) are deeply connected to gestures, but others (e.g., abstract semantic, syntactic and phonetic) are less so. In this way, the hands help to delineate what aspects of language function as part of the body and what aspects operate independently of it.
Article outline
  • General introduction
    • Converging methods
  • The four linguistic components
    • Semantics
      • Online and moment-to-moment integration
      • Developmental integration
      • Further exploration
    • Pragmatics
      • Moment-to-moment and developmental integration
      • Online integration
      • Further exploration
    • Phonetics
      • Online and moment-to-moment integration
      • Developmental integration
      • Further exploration
    • Syntax
      • Moment-to-moment and developmental integration
      • Online integration
      • Further exploration
  • Conclusion
  • Notes
  • References
Published online: 26 April 2017
DOI logo
https://doi.org/10.1075/gs.7.12kel
References (95)
References
Alibali, Martha W., & Goldin-Meadow, Susan 1993. “Gesture-speech mismatch and mechanisms of learning: What the hands reveal about a child’s state of mind.” Cognitive Psychology 25: 468–523. DOI logoGoogle Scholar
Armstrong, David F., & Wilcox, Sherman E. 2007. The Gestural Origin of Language. Oxford; New York: Oxford University Press. DOI logoGoogle Scholar
Austin, John L. 1975. How To Do Things With Words (Vol. 1955). Oxford University Press. DOI logoGoogle Scholar
Bailey, April, & Kelly, Spencer D. 2015. “Picture power: Gender versus body language in perceived status.” Journal of Nonverbal Behavior 39: 317–337. DOI logoGoogle Scholar
Baron-Cohen, Simon 1989. “Perceptual role taking and protodeclarative pointing in autism.” British Journal of Developmental Psychology 7 (2): 113–127. DOI logoGoogle Scholar
Barsalou, Lawrence W. 1999. “Perceptions of perceptual symbols.” Behavioral and Brain Sciences, 22(04), 637–660. DOI logoGoogle Scholar
Bates, Elizabeth, & Dick, Fredrick 2002. “Language, gesture, and the developing brain.” Developmental Psychobiology 40 (3): 293–310. DOI logoGoogle Scholar
Biau, Emmanuel, & Soto-Faraco, Salvador 2013. “Beat gestures modulate auditory integration in speech perception.” Brain and Language 124 (2): 143–152. DOI logoGoogle Scholar
Binder, Jeffrey. R., & Desai, Rutvik H. 2011. “The neurobiology of semantic memory.” Trends in Cognitive Sciences 15 (11): 527–536. DOI logoGoogle Scholar
Chomsky, Noam 1980. “Rules and representations.” Behavioral and Brain Sciences 3 (01): 1–15. DOI logoGoogle Scholar
Church, Ruth B. 1999. “Using gesture and speech to capture transitions in learning.” Cognitive Development 14: 313–342. DOI logoGoogle Scholar
Church, Ruth B., Garber, Philip, & Rogalsky, Kathryn 2007. “The role of gesture in social communication and memory.” Gesture 7: 137–157. DOI logoGoogle Scholar
Clark, Herbert H., & Marshall, Catherine R. 1981. “Definite reference and mutual knowledge.” In Elements of Discourse Understanding, A.K. Joshi, B. Webber, I.A. Sag (eds), 10–63. Cambridge University Press, Cambridge.Google Scholar
Corballis, Michael C. 2003. “From mouth to hand: gesture, speech, and the evolution of right-handedness.” Behavioral and Brain Sciences 26 (02): 199–208. DOI logoGoogle Scholar
de Langavant, Laurent C., et al. 2011. “Behavioral and neural correlates of communication via pointing.” PloS one 6 (3): e17719.Google Scholar
de Gelder, Beatrice 2006. “Towards the neurobiology of emotional body language.” Nature Reviews Neuroscience 7 (3): 242–249. DOI logoGoogle Scholar
Dingemanse, Mark, Blasi, Damián E., Lupyan, Gary, Christiansen, Morten H., & Monaghan, Padraic 2015. “Arbitrariness, iconicity, and systematicity in language.” Trends in Cognitive Sciences 19: 603–615. DOI logoGoogle Scholar
Dodd, Barbara 1979. “Lip reading in infants: Attention to speech presented in-and out-of-synchrony.” Cognitive Psychology 11 (4): 478–484. DOI logoGoogle Scholar
Fodor, Jerry A. 1983. The Modularity of Mind: An Essay on Faculty Psychology. MIT Press.Google Scholar
Gallese, Vittorio, & Lakoff, George 2005. “The brain's concepts: The role of the sensory-motor system in conceptual knowledge.” Cognitive Neuropsychology 22: 455–479. DOI logoGoogle Scholar
Gibbs, Raymond W. Jr. 2006. Embodiment and Cognitive Science. Cambridge University Press.Google Scholar
Goldin-Meadow, Susan, McClintock, Martha, & Wimsatt, William Unpublished manuscript. “Solving psychological problems in four dimensions: Heuristics for the social and biological sciences.”
Goldin-Meadow, Susan, Alibali, Martha W., & Church, Ruth B. 1993. “Transitions in concept acquisition: Using the hand to read the mind.” Psychological Review 100 (2): 279–297. DOI logoGoogle Scholar
Goldin-Meadow, Susan 2005. Hearing Gesture: How Our Hands Help Us Think. Harvard University Press.Google Scholar
Glenberg, Arthur M., & Kaschak, Michael P. 2002. “Grounding language in action.” Psychonomic Bulletin & Review 9 (3): 558–565. DOI logoGoogle Scholar
Grice, H. Paul 1975. “Logic and conversation.” In P. Cole and J. L. Morgan (eds), Syntax and semantics, (Vol. 3). New York: Academic Press.Google Scholar
Gullberg, Marianne 2006. “Some reasons for studying gesture and second language acquisition (Hommage à Adam Kendon).” IRAL-International Review of Applied Linguistics in Language Teaching 44 (2): 103–124. DOI logoGoogle Scholar
Gunter, Thomas C., & Bach, Patric 2004. “Communicating hands: ERPs elicited by meaningful symbolic hand postures.” Neuroscience Letters 372 (1): 52–56. DOI logoGoogle Scholar
Hagoort, Peter, Hald, Lea, Bastiaansen, Marcel, & Petersson, Karl Magnus 2004. “Integration of word meaning and world knowledge in language comprehension.” Science 304 (5669): 438–441. DOI logoGoogle Scholar
Hagoort, Peter 2005. “On Broca, brain, and binding: a new framework.” Trends in Cognitive Sciences 9 (9): 416–423. DOI logoGoogle Scholar
Hickok, Gregory 2014. The Myth of Mirror Neurons: The Real Neuroscience of Communication and Cognition. WW Norton & Company.Google Scholar
Hinzman, Lindsay & Kelly, Spencer D. 2013. “Effects of emotional body language on rapid out-group judgments.” Journal of Experimental Social Psychology 49: 152–155. DOI logoGoogle Scholar
Hirata, Yukari & Kelly, Spencer D. 2010. “Effects of lips and hands on auditory learning of second-language speech sounds.” Journal of Speech, Language, and Hearing Research 53 (2): 298–310. DOI logoGoogle Scholar
Hirata, Yukari, Kelly, Spencer D., Huang, Jessica, & Manansala, Michael 2014. “Effects of hand gestures on auditory learning of second-language vowel length contrasts.” Journal of Speech, Language, and Hearing Research 57 (6): 2090–2101. DOI logoGoogle Scholar
Holle, Henning, & Gunter, Thomas C. 2007. “The role of iconic gestures in speech disambiguation: ERP evidence.” Journal of Cognitive Neuroscience 19 (7): 1175–1192. DOI logoGoogle Scholar
Holler, Judith, Schubotz, Louise, Kelly, Spence D., Hagoort, Peter, Schuetze, Manuela & Özyürek, Asli 2014. “Social eye gaze modulates processing of speech and co-speech gesture.” Cognition 132: 696–697.Google Scholar
Holler, Judith, Kokal, Idil, Toni, Ivan, Hagoort, Peter, Kelly, Spencer D., & Özyürek, Asli 2014. “Eye'm talking to you: Speakers' gaze direction modulates co-speech gesture processing in the right MTG.” Social Cognitive and Affective Neuroscience 10 (2): 255–261. DOI logoGoogle Scholar
Hostetter, Autumn B., & Alibali, Martha W. 2008. “Visible embodiment: Gestures as simulated action.” Psychonomic Bulletin & Review 15 (3): 495–514. DOI logoGoogle Scholar
Hostetter, Autumn B. 2011. “When do gestures communicate? A meta-analysis.” Psychological Bulletin 137 (2): 297. DOI logoGoogle Scholar
Hubbard, Amy L., Wilson, Stephen M., Callan, Daniel E., & Dapretto, Mirella 2009. “Giving speech a hand: gesture modulates activity in auditory cortex during speech perception.” Human Brain Mapping 30 (3): 1028–1037. DOI logoGoogle Scholar
Karmiloff-Smith, Anette 1995. Beyond Modularity: A Developmental Perspective on Cognitive Science. MIT Press.Google Scholar
Kelly, Spencer D., Barr, Dale J., Church, Ruth B., & Lynch, Katheryn 1999. “Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory.” Journal of Memory and Language 40 (4): 577–592. DOI logoGoogle Scholar
Kelly, Spencer D., Kravitz, Corinne, & Hopkins, Hopkins 2004. “Neural correlates of bimodal speech and gesture comprehension.” Brain and language 89: 253–260. DOI logoGoogle Scholar
Kelly, Spencer D. 2001. “Broadening the units of analysis in communication: Speech and nonverbal behaviours in pragmatic comprehension.” Journal of Child Language 28 (02): 325–349. DOI logoGoogle Scholar
Kelly, Spencer D., Iverson, Jana M., Terranova, Joseph, Niego, Julia, Hopkins, Michael, & Goldsmith, Leslie 2002. “Putting language back in the body: Speech and gesture on three time frames.” Developmental Neuropsychology 22 (1): 323–349. DOI logoGoogle Scholar
Kelly, Spencer D., Ward, Sarah, Creigh, Peter, & Bartolotti, James 2007. “An intentional stance modulates the integration of gesture and speech during comprehension.” Brain and Language 101: 222–233. DOI logoGoogle Scholar
Kelly, Spencer D., Manning, Sarah M., & Rodak, Sabrina 2008. “Gesture gives a hand to language and learning: Perspectives from cognitive neuroscience, developmental psychology and education.” Language and Linguistics Compass 2 (4): 569–588. DOI logoGoogle Scholar
Kelly, Spencer D., McDevitt, Tara, & Esch, Megan 2009. “Brief training with co-speech gesture lends a hand to word learning in a foreign language.” Language and Cognitive Processes 24 (2): 313–334. DOI logoGoogle Scholar
Kelly, Spencer D., Özyürek, Asli, & Maris, Eric. 2010. “Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension.” Psychological Science 21: 260–267. DOI logoGoogle Scholar
Kelly, Spencer D., Creigh, Peter, & Bartolotti, James 2010. “Integrating speech and iconic gestures in a Stroop-like task: Evidence for automatic processing.” Journal of Cognitive Neuroscience 22: 683–694. DOI logoGoogle Scholar
Kelly, Spencer D., & Lee, Angela L. 2012. “When actions speak too much louder than words: Hand gestures disrupt word learning when phonetic demands are high.” Language and Cognitive Processes 27 (6): 793–807. DOI logoGoogle Scholar
Kelly, Spencer D., Healey, Meghan, Özyürek, Asli, & Holler, Judith 2014. “The processing of speech, gesture and action during language comprehension.” Psychonomic Bulletin & Review 22 (2): 517–523. DOI logoGoogle Scholar
Kelly, Spencer D., Hirata, Yukari, Manansala, Michael, & Huang, Jessica 2014. “Exploring the role of hand gestures in learning novel phoneme contrasts and vocabulary in a second language.” Frontiers in Psychology, 5. DOI logoGoogle Scholar
Kendon, Adam 1997. “Gesture.” Annual Review of Anthropology 26 (1): 109–128.Google Scholar
2004. Gesture: Visible Action as Utterance. Cambridge University Press. DOI logoGoogle Scholar
Kita, Sotaro, & Özyürek, Asli 2003. “What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking.” Journal of Memory and Language 48 (1): 16–32. DOI logoGoogle Scholar
Klaver, Peter, et al. 2005. “Word imageability affects the hippocampus in recognition memory.” Hippocampus 15 (6): 704–712. DOI logoGoogle Scholar
Knill, David C., & Pouget, Alexandre 2004. “The Bayesian brain: the role of uncertainty in neural coding and computation.” Trends in Neurosciences 27 (12): 712–719. DOI logoGoogle Scholar
Krahmer, Emile, & Swerts, Marc 2007. “The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception.” Journal of Memory and Language 57 (3): 396–414. DOI logoGoogle Scholar
Kuhl, Patricia K., & Meltzoff, Andrew N. 1984. “The intermodal representation of speech in infants.” Infant Behavior and Development 7 (3): 361–381. DOI logoGoogle Scholar
Kutas, Marta, & Hillyard, Steven A. 1980. “Reading senseless sentences: Brain potentials reflect semantic incongruity.” Science 207 (4427): 203–205. DOI logoGoogle Scholar
Lakoff, George, & Johnson, Mark 1999. Philosophy In The Flesh: The Embodied Mind And Its Challenge To Western Thought. Basic Books.Google Scholar
Loehr, Daniel 2007. “Aspects of rhythm in gesture and speech.” Gesture 7 (2): 179–214. DOI logoGoogle Scholar
Macedonia, Manuela, Müller, Karsten, & Friederici, Angela D. 2011. “The impact of iconic gestures on foreign language word learning and its neural substrate.” Human Brain Mapping 32 (6): 982–998. DOI logoGoogle Scholar
Masumoto, Kouhei, et al. 2006. “Reactivation of physical motor information in the memory of action events.” Brain Research 1101 (1): 102–109. DOI logoGoogle Scholar
Mahon, Bradford Z., & Caramazza, Alfonso 2008. “A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content.” Journal of Physiology-Paris 102 (1): 59–70. DOI logoGoogle Scholar
McGurk, Harry, & MacDonald, John 1976. “Hearing lips and seeing voices.” Nature 264: 746–748 DOI logoGoogle Scholar
McNeill, David 1992. Hand and Mind: What Gestures Reveal About Thought. University of Chicago Press.Google Scholar
2008. Gesture and Thought. University of Chicago Press.Google Scholar
2012. How Language Began: Gesture and Speech in Human Evolution. Cambridge University Press. DOI logoGoogle Scholar
2013. “Why we gesture.” On Research Gate.Google Scholar
Morford, Marolyn, & Goldin-Meadow, Susan 1992. “Comprehension and production of gesture in combination with speech in one-word speakers.” Journal of Child Language 19 (03): 559–580. DOI logoGoogle Scholar
Murphy, Catherine M., & Messer, David J. 1977. “Mothers, infants and pointing: A study of a gesture.” Studies in Mother-Infant Interaction: 325–354.Google Scholar
Özyürek, Asli, Willems, Roel, Kita, Sotaro, & Hagoort, Peter 2007. “On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials.” Journal of Cognitive Neuroscience 19 (4): 605–616. DOI logoGoogle Scholar
Özyürek, Asli 2014. “Hearing and seeing meaning in speech and gesture: insights from brain and behaviour.” Philosophical Transactions of the Royal Society B: Biological Sciences 369 (1651): 20130296. DOI logoGoogle Scholar
Perry, Michelle, Church, R. Breckinridge, & Goldin-Meadow, Susan 1992. “Is gesture-speech mismatch a general index of transitional knowledge?” Cognitive Development 7 (1): 109–122. DOI logoGoogle Scholar
Post, Lysanne S., Van Gog, Tamara, Paas, Fred, & Zwaan, Rolf A. 2013. “Effects of simultaneously observing and making gestures while studying grammar animations on cognitive load and learning.” Computers in Human Behavior 29 (4): 1450–1455. DOI logoGoogle Scholar
Povinelli, Daniel J. 1993. “Reconstructing the evolution of mind.” American Psychologist 48: 493–509. DOI logoGoogle Scholar
Pulvermüller, F. 2005. “Brain mechanisms linking language and action.” Nature Reviews Neuroscience 6 (7): 576–582. DOI logoGoogle Scholar
Rizzolatti, Giacomo, & Arbib, Michael A. 1998. “Language within our grasp.” Trends in Neurosciences 21 (5): 188–194. DOI logoGoogle Scholar
Skipper, Jeremy I. 2014. “Echoes of the spoken past: how auditory cortex hears context during speech perception.” Philosophical Transactions of the Royal Society B: Biological Sciences 369 (1651): 20130297. DOI logoGoogle Scholar
So, Wing C., Kita, Sotaro, & Goldin-Meadow, Susan 2009. “Using the hands to identify who does what to whom: Gesture and speech go hand-in-hand.” Cognitive Science 33 (1): 115–125. DOI logoGoogle Scholar
Stam, Gale 2006. “Thinking for speaking about motion: L1 and L2 speech and gesture.” IRAL-International Review of Applied Linguistics in Language Teaching 44 (2): 145–171. DOI logoGoogle Scholar
Straube, Benjamin, Green, Antonia, Jansen, Andreas, Chatterjee, Anjan, & Kircher, Tilo 2010. “Social cues, mentalizing and the neural processing of speech accompanied by gestures.” Neuropsychologia 48 (2): 382–393. DOI logoGoogle Scholar
Straube, Benjamin, Green, Antonia, Weis, Susanne, & Chatterjee, Anjan 2009. “Memory effects of speech and gesture binding: cortical and hippocampal activation in relation to subsequent memory performance.” Journal of Cognitive Neuroscience 21 (4): 821–836. DOI logoGoogle Scholar
Sumby, Willaim H., & Pollack, Irwin 1954. “Visual contribution to speech intelligibility in noise.” The Journal of the Acoustical Society of America 26 (2): 212–215. DOI logoGoogle Scholar
Tenenbaum, Joshua B., Kemp, Charles, Griffiths, Thomas L., & Goodman Noah D. 2011. “How to grow a mind: Statistics, structure, and abstraction.” Science 331: 1279–1285. DOI logoGoogle Scholar
Thompson, Laura A., & Massaro, Dominic W. 1986. “Evaluation and integration of speech and pointing gestures during referential understanding.” Journal of Experimental Child Psychology 42 (1): 144–168. DOI logoGoogle Scholar
1994. “Children's integration of speech and pointing gestures in comprehension.” Journal of Experimental Child Psychology 57 (3): 327–354. DOI logoGoogle Scholar
Tomasello, Michael 2008. Origins of Human Communication. Cambridge: MIT Press.Google Scholar
van Wassenhove, Virgini, Grant, Ken W., & Poeppel, David 2005. “Visual speech speeds up the neural processing of auditory speech.” Proceedings of the National Academy of Sciences of the United States of America 102 (4): 1181–1186. DOI logoGoogle Scholar
Willems, Roel M., & Hagoort, Peter 2007. “Neural evidence for the interplay between language, gesture, and action: A review.” Brain and Language 101 (3): 278–289. DOI logoGoogle Scholar
Wimsatt, William 1981. “Robustness, reliability and overdetermination.” In Scientific Inquiry and the Social Sciences, M. Brewer and B. Collins (eds), 124–163. San Francisco: Jossey-Bass.Google Scholar
Wu, Ying Choon, & Coulson, Seana 2010. “Gestures modulate speech processing early in utterances.” Neuroreport 21 (7): 522.Google Scholar
Zwaan, Rolf A. 2004. “The immersed experiencer: Toward an embodied theory of language comprehension.” Psychology of Learning and Motivation 44: 35–62. DOI logoGoogle Scholar
Cited by (15)

Cited by 15 other publications

Gullberg, Marianne
2024. Gesture and Second/Foreign Language Acquisition. In The Cambridge Handbook of Gesture Studies, ► pp. 398 ff. DOI logo
Nirme, Jens, Agneta Gulz, Magnus Haake & Marianne Gullberg
2024. Early or synchronized gestures facilitate speech recall—a study based on motion capture data. Frontiers in Psychology 15 DOI logo
Arbona, Eléonore, Kilian G. Seeber & Marianne Gullberg
2023. The role of manual gestures in second language comprehension: a simultaneous interpreting experiment. Frontiers in Psychology 14 DOI logo
Arbona, Eléonore, Kilian G. Seeber & Marianne Gullberg
2023. Semantically related gestures facilitate language comprehension during simultaneous interpreting. Bilingualism: Language and Cognition 26:2 ► pp. 425 ff. DOI logo
Arbona, Eléonore, Kilian G. Seeber & Marianne Gullberg
2024. The role of semantically related gestures in the language comprehension of simultaneous interpreters in noise. Language, Cognition and Neuroscience 39:5 ► pp. 584 ff. DOI logo
Janzen Ulbricht, Natasha & Shane Lindsay
2023. Can grammatical morphemes be taught? Evidence of gestures influencing second language procedural learning in middle childhood. PLOS ONE 18:2 ► pp. e0280543 ff. DOI logo
Kelly, Spencer D. & Quang‐Anh Ngo Tran
2023. Exploring the Emotional Functions of Co‐Speech Hand Gesture in Language and Communication. Topics in Cognitive Science DOI logo
Mlakar, Izidor, Darinka Verdonik, Simona Majhenič & Matej Rojc
2023. Understanding conversational interaction in multiparty conversations: the EVA Corpus. Language Resources and Evaluation 57:2 ► pp. 641 ff. DOI logo
Billot-Vasquez, Kiana, Zhongwen Lian, Yukari Hirata & Spencer D. Kelly
2020. Emblem Gestures Improve Perception and Evaluation of Non-native Speech. Frontiers in Psychology 11 DOI logo
Iizuka, Takehiro, Kimi Nakatsukasa & Aaron Braver
2020. The Efficacy of Gesture on Second Language Pronunciation: An Exploratory Study of Handclapping as a Classroom Instructional Tool. Language Learning 70:4 ► pp. 1054 ff. DOI logo
Janzen Ulbricht, Natasha
2020. The Embodied Teaching of Spatial Terms: Gestures Mapped to Morphemes Improve Learning. Frontiers in Education 5 DOI logo
Levy, Rachel S. & Spencer D. Kelly
2020. Emotion matters. Gesture 19:1 ► pp. 41 ff. DOI logo
van Berkel-van Hoof, Lian, Daan Hermans, Harry Knoors & Ludo Verhoeven
2020. Sign effects in spoken word learning by oral deaf and hard-of-hearing children, and by hearing children. First Language 40:3 ► pp. 300 ff. DOI logo
Zheng, Annie, Yukari Hirata & Spencer D. Kelly
2018. Exploring the Effects of Imitating Hand Gestures and Head Nods on L1 and L2 Mandarin Tone Production. Journal of Speech, Language, and Hearing Research 61:9 ► pp. 2179 ff. DOI logo
DINGEMANSE, MARK & KIMI AKITA
2017. An inverse relation between expressiveness and grammatical integration: On the morphosyntactic typology of ideophones, with special reference to Japanese. Journal of Linguistics 53:3 ► pp. 501 ff. DOI logo

This list is based on CrossRef data as of 15 october 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.