The present review explores the integrated relationship between gesture and speech during language comprehension. Taking a broad view, it presents a conceptual framework that approaches the comprehension of gesture and speech along three different dimensions: (1) components of language (semantics, pragmatics, phonetics and syntax), (2) levels of analysis (social, cognitive, and biological) and (3) timeframes of integration (online, moment-to-moment, developmental). The evidence suggests that some linguistic components (e.g., concrete semantic and pragmatic) are deeply connected to gestures, but others (e.g., abstract semantic, syntactic and phonetic) are less so. In this way, the hands help to delineate what aspects of language function as part of the body and what aspects operate independently of it.
1981 “Definite reference and mutual knowledge.” In Elements of Discourse Understanding, A.K. Joshi, B. Webber, I.A. Sag (eds), 10–63. Cambridge University Press, Cambridge.
Corballis, Michael C.
2003 “From mouth to hand: gesture, speech, and the evolution of right-handedness.” Behavioral and Brain Sciences 26 (02): 199–208.
de Langavant, Laurent C., et al.
2011 “Behavioral and neural correlates of communication via pointing.” PloS one 6 (3): e17719.
de Gelder, Beatrice
2006 “Towards the neurobiology of emotional body language.” Nature Reviews Neuroscience 7 (3): 242–249.
2015 “Arbitrariness, iconicity, and systematicity in language.” Trends in Cognitive Sciences 19: 603–615.
Dodd, Barbara
1979 “Lip reading in infants: Attention to speech presented in-and out-of-synchrony.” Cognitive Psychology 11 (4): 478–484.
Fodor, Jerry A.
1983The Modularity of Mind: An Essay on Faculty Psychology. MIT Press.
Gallese, Vittorio, & Lakoff, George
2005 “The brain's concepts: The role of the sensory-motor system in conceptual knowledge.” Cognitive Neuropsychology 22: 455–479.
Gibbs, Raymond W. Jr.
2006Embodiment and Cognitive Science. Cambridge University Press.
Goldin-Meadow, Susan, McClintock, Martha, & Wimsatt, William
Unpublished manuscript. “Solving psychological problems in four dimensions: Heuristics for the social and biological sciences.”
Goldin-Meadow, Susan, Alibali, Martha W., & Church, Ruth B.
1993 “Transitions in concept acquisition: Using the hand to read the mind.” Psychological Review 100 (2): 279–297.
Goldin-Meadow, Susan
2005Hearing Gesture: How Our Hands Help Us Think. Harvard University Press.
Glenberg, Arthur M., & Kaschak, Michael P.
2002 “Grounding language in action.” Psychonomic Bulletin & Review 9 (3): 558–565.
Grice, H. Paul
1975 “Logic and conversation.” In P. Cole and J. L. Morgan (eds), Syntax and semantics, (Vol. 3). New York: Academic Press.
Gullberg, Marianne
2006 “Some reasons for studying gesture and second language acquisition (Hommage à Adam Kendon).” IRAL-International Review of Applied Linguistics in Language Teaching 44 (2): 103–124.
Gunter, Thomas C., & Bach, Patric
2004 “Communicating hands: ERPs elicited by meaningful symbolic hand postures.” Neuroscience Letters 372 (1): 52–56.
Hagoort, Peter, Hald, Lea, Bastiaansen, Marcel, & Petersson, Karl Magnus
2004 “Integration of word meaning and world knowledge in language comprehension.” Science 304 (5669): 438–441.
Hagoort, Peter
2005 “On Broca, brain, and binding: a new framework.” Trends in Cognitive Sciences 9 (9): 416–423.
Hickok, Gregory
2014The Myth of Mirror Neurons: The Real Neuroscience of Communication and Cognition. WW Norton & Company.
Hinzman, Lindsay & Kelly, Spencer D.
2013 “Effects of emotional body language on rapid out-group judgments.” Journal of Experimental Social Psychology 49: 152–155.
Hirata, Yukari & Kelly, Spencer D.
2010 “Effects of lips and hands on auditory learning of second-language speech sounds.” Journal of Speech, Language, and Hearing Research 53 (2): 298–310.
Hirata, Yukari, Kelly, Spencer D., Huang, Jessica, & Manansala, Michael
2014 “Effects of hand gestures on auditory learning of second-language vowel length contrasts.” Journal of Speech, Language, and Hearing Research 57 (6): 2090–2101.
Holle, Henning, & Gunter, Thomas C.
2007 “The role of iconic gestures in speech disambiguation: ERP evidence.” Journal of Cognitive Neuroscience 19 (7): 1175–1192.
Holler, Judith, Schubotz, Louise, Kelly, Spence D., Hagoort, Peter, Schuetze, Manuela & Özyürek, Asli
2014 “Social eye gaze modulates processing of speech and co-speech gesture.” Cognition 132: 696–697.
Holler, Judith, Kokal, Idil, Toni, Ivan, Hagoort, Peter, Kelly, Spencer D., & Özyürek, Asli
2014 “Eye'm talking to you: Speakers' gaze direction modulates co-speech gesture processing in the right MTG.” Social Cognitive and Affective Neuroscience 10 (2): 255–261.
1999 “Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory.” Journal of Memory and Language 40 (4): 577–592.
Kelly, Spencer D., Kravitz, Corinne, & Hopkins, Hopkins
2004 “Neural correlates of bimodal speech and gesture comprehension.” Brain and language 89: 253–260.
Kelly, Spencer D.
2001 “Broadening the units of analysis in communication: Speech and nonverbal behaviours in pragmatic comprehension.” Journal of Child Language 28 (02): 325–349.
Kelly, Spencer D., Iverson, Jana M., Terranova, Joseph, Niego, Julia, Hopkins, Michael, & Goldsmith, Leslie
2002 “Putting language back in the body: Speech and gesture on three time frames.” Developmental Neuropsychology 22 (1): 323–349.
Kelly, Spencer D., Ward, Sarah, Creigh, Peter, & Bartolotti, James
2007 “An intentional stance modulates the integration of gesture and speech during comprehension.” Brain and Language 101: 222–233.
Kelly, Spencer D., Manning, Sarah M., & Rodak, Sabrina
2008 “Gesture gives a hand to language and learning: Perspectives from cognitive neuroscience, developmental psychology and education.” Language and Linguistics Compass 2 (4): 569–588.
Kelly, Spencer D., McDevitt, Tara, & Esch, Megan
2009 “Brief training with co-speech gesture lends a hand to word learning in a foreign language.” Language and Cognitive Processes 24 (2): 313–334.
Kelly, Spencer D., Özyürek, Asli, & Maris, Eric
2010 “Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension.” Psychological Science 21: 260–267.
Kelly, Spencer D., Creigh, Peter, & Bartolotti, James
2010 “Integrating speech and iconic gestures in a Stroop-like task: Evidence for automatic processing.” Journal of Cognitive Neuroscience 22: 683–694.
Kelly, Spencer D., & Lee, Angela L.
2012 “When actions speak too much louder than words: Hand gestures disrupt word learning when phonetic demands are high.” Language and Cognitive Processes 27 (6): 793–807.
Kelly, Spencer D., Healey, Meghan, Özyürek, Asli, & Holler, Judith
2014 “The processing of speech, gesture and action during language comprehension.” Psychonomic Bulletin & Review 22 (2): 517–523.
Kelly, Spencer D., Hirata, Yukari, Manansala, Michael, & Huang, Jessica
2014 “Exploring the role of hand gestures in learning novel phoneme contrasts and vocabulary in a second language.” Frontiers in Psychology, 5.
Kendon, Adam
1997. “Gesture.” Annual Review of Anthropology 26 (1): 109–128.
Kendon, Adam
2004Gesture: Visible Action as Utterance. Cambridge University Press.
Kita, Sotaro, & Özyürek, Asli
2003 “What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking.” Journal of Memory and Language 48 (1): 16–32.
Klaver, Peter, et al.
2005 “Word imageability affects the hippocampus in recognition memory.” Hippocampus 15 (6): 704–712.
Knill, David C., & Pouget, Alexandre
2004 “The Bayesian brain: the role of uncertainty in neural coding and computation.” Trends in Neurosciences 27 (12): 712–719.
Krahmer, Emile, & Swerts, Marc
2007 “The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception.” Journal of Memory and Language 57 (3): 396–414.
Kuhl, Patricia K., & Meltzoff, Andrew N.
1984 “The intermodal representation of speech in infants.” Infant Behavior and Development 7 (3): 361–381.
Macedonia, Manuela, Müller, Karsten, & Friederici, Angela D.
2011 “The impact of iconic gestures on foreign language word learning and its neural substrate.” Human Brain Mapping 32 (6): 982–998.
Masumoto, Kouhei, et al.
2006 “Reactivation of physical motor information in the memory of action events.” Brain Research 1101 (1): 102–109.
Mahon, Bradford Z., & Caramazza, Alfonso
2008 “A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content.” Journal of Physiology-Paris 102 (1): 59–70.
McGurk, Harry, & MacDonald, John
1976 “Hearing lips and seeing voices.” Nature 264: 746–748
McNeill, David
1992Hand and Mind: What Gestures Reveal About Thought. University of Chicago Press.
McNeill, David
2008Gesture and Thought. University of Chicago Press.
McNeill, David
2012How Language Began: Gesture and Speech in Human Evolution. Cambridge University Press.
McNeill, David
2013 “Why we gesture.” On Research Gate.
Morford, Marolyn, & Goldin-Meadow, Susan
1992 “Comprehension and production of gesture in combination with speech in one-word speakers.” Journal of Child Language 19 (03): 559–580.
Murphy, Catherine M., & Messer, David J.
1977 “Mothers, infants and pointing: A study of a gesture.” Studies in Mother-Infant Interaction: 325–354.
Özyürek, Asli, Willems, Roel, Kita, Sotaro, & Hagoort, Peter
2007 “On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials.” Journal of Cognitive Neuroscience 19 (4): 605–616.
Özyürek, Asli
2014 “Hearing and seeing meaning in speech and gesture: insights from brain and behaviour.” Philosophical Transactions of the Royal Society B: Biological Sciences 369 (1651): 20130296.
Perry, Michelle, Church, R. Breckinridge, & Goldin-Meadow, Susan
1992 “Is gesture-speech mismatch a general index of transitional knowledge?”Cognitive Development 7 (1): 109–122.
Post, Lysanne S., Van Gog, Tamara, Paas, Fred, & Zwaan, Rolf A.
2013 “Effects of simultaneously observing and making gestures while studying grammar animations on cognitive load and learning.” Computers in Human Behavior 29 (4): 1450–1455.
Povinelli, Daniel J.
1993 “Reconstructing the evolution of mind.” American Psychologist 48: 493–509.
Pulvermüller, F.
2005 “Brain mechanisms linking language and action.” Nature Reviews Neuroscience 6 (7): 576–582.
Rizzolatti, Giacomo, & Arbib, Michael A.
1998 “Language within our grasp.” Trends in Neurosciences 21 (5): 188–194.
Skipper, Jeremy I.
2014 “Echoes of the spoken past: how auditory cortex hears context during speech perception.” Philosophical Transactions of the Royal Society B: Biological Sciences 369 (1651): 20130297.
So, Wing C., Kita, Sotaro, & Goldin-Meadow, Susan
2009 “Using the hands to identify who does what to whom: Gesture and speech go hand-in-hand.” Cognitive Science 33 (1): 115–125.
Stam, Gale
2006 “Thinking for speaking about motion: L1 and L2 speech and gesture.” IRAL-International Review of Applied Linguistics in Language Teaching 44 (2): 145–171.
2009 “Memory effects of speech and gesture binding: cortical and hippocampal activation in relation to subsequent memory performance.” Journal of Cognitive Neuroscience 21 (4): 821–836.
Sumby, Willaim H., & Pollack, Irwin
1954 “Visual contribution to speech intelligibility in noise.” The Journal of the Acoustical Society of America 26 (2): 212–215.
Tenenbaum, Joshua B., Kemp, Charles, Griffiths, Thomas L., & Goodman Noah D.
2011 “How to grow a mind: Statistics, structure, and abstraction.” Science 331: 1279–1285.
Thompson, Laura A., & Massaro, Dominic W.
1986 “Evaluation and integration of speech and pointing gestures during referential understanding.” Journal of Experimental Child Psychology 42 (1): 144–168.
Thompson, Laura A., & Massaro, Dominic W.
1994 “Children's integration of speech and pointing gestures in comprehension.” Journal of Experimental Child Psychology 57 (3): 327–354.
Tomasello, Michael
2008Origins of Human Communication. Cambridge: MIT Press.
van Wassenhove, Virgini, Grant, Ken W., & Poeppel, David
2005 “Visual speech speeds up the neural processing of auditory speech.” Proceedings of the National Academy of Sciences of the United States of America 102 (4): 1181–1186.
Willems, Roel M., & Hagoort, Peter
2007 “Neural evidence for the interplay between language, gesture, and action: A review.” Brain and Language 101 (3): 278–289.
Wimsatt, William
1981 “Robustness, reliability and overdetermination.” In Scientific Inquiry and the Social Sciences, M. Brewer and B. Collins (eds), 124–163. San Francisco: Jossey-Bass.
Wu, Ying Choon, & Coulson, Seana
2010 “Gestures modulate speech processing early in utterances.” Neuroreport 21 (7): 522.
Zwaan, Rolf A.
2004 “The immersed experiencer: Toward an embodied theory of language comprehension.” Psychology of Learning and Motivation 44: 35–62.
Cited by
Cited by 12 other publications
Arbona, Eléonore, Kilian G. Seeber & Marianne Gullberg
2023. The role of manual gestures in second language comprehension: a simultaneous interpreting experiment. Frontiers in Psychology 14
Arbona, Eléonore, Kilian G. Seeber & Marianne Gullberg
2023. Semantically related gestures facilitate language comprehension during simultaneous interpreting. Bilingualism: Language and Cognition 26:2 ► pp. 425 ff.
Billot-Vasquez, Kiana, Zhongwen Lian, Yukari Hirata & Spencer D. Kelly
2020. Emblem Gestures Improve Perception and Evaluation of Non-native Speech. Frontiers in Psychology 11
DINGEMANSE, MARK & KIMI AKITA
2017. An inverse relation between expressiveness and grammatical integration: On the morphosyntactic typology of ideophones, with special reference to Japanese. Journal of Linguistics 53:3 ► pp. 501 ff.
Iizuka, Takehiro, Kimi Nakatsukasa & Aaron Braver
2020. The Efficacy of Gesture on Second Language Pronunciation: An Exploratory Study of Handclapping as a Classroom Instructional Tool. Language Learning 70:4 ► pp. 1054 ff.
Janzen Ulbricht, Natasha
2020. The Embodied Teaching of Spatial Terms: Gestures Mapped to Morphemes Improve Learning. Frontiers in Education 5
Janzen Ulbricht, Natasha & Shane Lindsay
2023. Can grammatical morphemes be taught? Evidence of gestures influencing second language procedural learning in middle childhood. PLOS ONE 18:2 ► pp. e0280543 ff.
Kelly, Spencer D. & Quang‐Anh Ngo Tran
2023. Exploring the Emotional Functions of Co‐Speech Hand Gesture in Language and Communication. Topics in Cognitive Science
Mlakar, Izidor, Darinka Verdonik, Simona Majhenič & Matej Rojc
2023. Understanding conversational interaction in multiparty conversations: the EVA Corpus. Language Resources and Evaluation 57:2 ► pp. 641 ff.
van Berkel-van Hoof, Lian, Daan Hermans, Harry Knoors & Ludo Verhoeven
2020. Sign effects in spoken word learning by oral deaf and hard-of-hearing children, and by hearing children. First Language 40:3 ► pp. 300 ff.
Zheng, Annie, Yukari Hirata & Spencer D. Kelly
2018. Exploring the Effects of Imitating Hand Gestures and Head Nods on L1 and L2 Mandarin Tone Production. Journal of Speech, Language, and Hearing Research 61:9 ► pp. 2179 ff.
This list is based on CrossRef data as of 12 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.