Speech-gesture mismatches
Evidence for one underlying representation of linguistic and nonlinguistic information
Justine Cassell | Media Laboratory, MIT
David McNeill | Psychology and Linguistics, University of Chicago
Karl-Erik McCullough | Psychology and Linguistics, University of Chicago
Adults and children spontaneously produce gestures while they speak, and such gestures appear to support and expand on the information communicated by the verbal channel. Little research, however, has been carried out to examine the role played by gesture in the listener's representation of accumulating information. Do listeners attend to the gestures that accompany narrative speech? In what kinds of relationships between gesture and speech do listeners attend to the gestural channel? If listeners do attend to information received in gesture, how is this information represented— is it 'tagged' as originating in the gestural channel? In this article research is described that addresses these questions. Results show that listeners do attend to information conveyed in gesture, when that information supplements or even contradicts the information conveyed by speech. And information received via gesture is available for retelling in speech. These results are taken to demonstrate that gesture is not taken by the listener to be epiphenomenal to the act of speaking, or a simple manual translation of speech. But they also suggest that the information conveyed in a discourse may be represented in a manner that is neither gesture nor language, although accessible to both channels.
Published online: 22 May 1999
https://doi.org/10.1075/pc.7.1.03cas
https://doi.org/10.1075/pc.7.1.03cas
Cited by
Cited by 114 other publications
Alviar, Camila, Rick Dale & Alexia Galati
2019. Complex Communication Dynamics: Exploring the Structure of an Academic Talk.
Cognitive Science 43:3 ► pp. e12718 ff. 
Austin, Elizabeth E. & Naomi Sweller
2018. Gesturing Along the Way: Adults’ and Preschoolers’ Communication of Route Direction Information.
Journal of Nonverbal Behavior 42:2 ► pp. 199 ff.
Beege, Maik, Manuel Ninaus, Sascha Schneider, Steve Nebel, Julia Schlemmel, Jasmin Weidenmüller, Korbinian Moeller & Günter Daniel Rey
2020. Investigating the effects of beat and deictic gestures of a lecturer in educational videos.
Computers & Education 156 ► pp. 103955 ff.
Beilock, S.L. & S. Goldin-Meadow
2010. Gesture Changes Thought by Grounding It in Action.
Psychological Science 21:11 ► pp. 1605 ff.
Bergman, Helena Fatouros, Gunilla Preisler & Andrzej Werbart
2006. Communicating with patients with schizophrenia: characteristics of well functioning and poorly functioning communication.
Qualitative Research in Psychology 3:2 ► pp. 121 ff.
Bremner, Paul & Ute Leonards
2015.
In 2015 IEEE International Conference on Robotics and Automation (ICRA),
► pp. 1999 ff.
Bremner, Paul & Ute Leonards
2016. Iconic Gestures for Robot Avatars, Recognition and Integration with Speech.
Frontiers in Psychology 7
Buisine, Stéphanie & Jean-Claude Martin
2007. The effects of speech–gesture cooperation in animated agents’ behavior in multimedia presentations.
Interacting with Computers 19:4 ► pp. 484 ff.
Cabibihan, John-John, Wendy Yusson, Saba Salehi & Shuzhi Sam Ge
2010.
In Social Robotics [Lecture Notes in Computer Science, 6414],
► pp. 63 ff.
Carlson, Crystal, Steven A. Jacobs, Michelle Perry & Ruth Breckinridge Church
2014. The effect of gestured instruction on the learning of physical causality problems.
Gesture 14:1 ► pp. 26 ff.
Carrazza, Cristina, Elizabeth M. Wakefield, Naureen Hemani-Lopez, Kristin Plath & Susan Goldin-Meadow
2021. Children integrate speech and gesture across a wider temporal window than speech and action when learning a math concept.
Cognition 210 ► pp. 104604 ff.
Chan, Dana Michelle & Spencer Kelly
2021. Construing events first-hand: Gesture viewpoints interact with speech to shape the attribution and memory of agency.
Memory & Cognition
Chen, Lele, Guofeng Cui, Celong Liu, Zhong Li, Ziyi Kou, Yi Xu & Chenliang Xu
2020.
In Computer Vision – ECCV 2020 [Lecture Notes in Computer Science, 12354],
► pp. 35 ff.
Clough, Sharice & Melissa C. Duff
2020. The Role of Gesture in Communication and Cognition: Implications for Understanding and Treating Neurogenic Communication Disorders.
Frontiers in Human Neuroscience 14
Cocks, Naomi, Suzanne Byrne, Madeleine Pritchard, Gary Morgan & Lucy Dipper
2018. Integration of speech and gesture in aphasia.
International Journal of Language & Communication Disorders 53:3 ► pp. 584 ff.
Cook, Susan & Kimberly M. Fenn
2017.
In Why Gesture? [Gesture Studies, 7],
► pp. 129 ff.
Cook, Susan Wagner & Michael K. Tanenhaus
2009. Embodied communication: Speakers’ gestures affect listeners’ actions.
Cognition 113:1 ► pp. 98 ff.
Cook, Susan Wagner, Terina Kuangyi Yip & Susan Goldin-Meadow
2012. Gestures, but not meaningless movements, lighten working memory load when explaining math.
Language and Cognitive Processes 27:4 ► pp. 594 ff.
Corradini, A.
2001.
In Proceedings IEEE ICCV Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems,
► pp. 82 ff.
Cutica, Ilaria & Monica Bucciarelli
2015. Non-determinism in the Uptake of Gestural Information.
Journal of Nonverbal Behavior 39:4 ► pp. 289 ff.
Dargue, Nicole & Naomi Sweller
2018. Donald Duck’s garden: The effects of observing iconic reinforcing and contradictory gestures on narrative comprehension.
Journal of Experimental Child Psychology 175 ► pp. 96 ff.
Dargue, Nicole & Naomi Sweller
2018. Not All Gestures are Created Equal: The Effects of Typical and Atypical Iconic Gestures on Narrative Comprehension.
Journal of Nonverbal Behavior 42:3 ► pp. 327 ff.
de Marchena, A., E. S. Kim, A. Bagdasarov, J. Parish-Morris, B. B. Maddox, E. S. Brodkin & R. T. Schultz
2019. Atypicalities of Gesture Form and Function in Autistic Adults.
Journal of Autism and Developmental Disorders 49:4 ► pp. 1438 ff.
Debreslioska, Sandra & Marianne Gullberg
2019. Discourse Reference Is Bimodal: How Information Status in Speech Interacts with Presence and Viewpoint of Gestures.
Discourse Processes 56:1 ► pp. 41 ff.
Debreslioska, Sandra, Joost van de Weijer & Marianne Gullberg
2019. Addressees Are Sensitive to the Presence of Gesture When Tracking a Single Referent in Discourse.
Frontiers in Psychology 10
Ducasse, Ana Maria
2013. Such a Nice Gesture: Paired Spanish Interaction in Oral Test Discourse.
Journal of Language Teaching and Research 4:6
Ebert, Cornelia
2018. A comparison of sign language with speech plus gesture.
Theoretical Linguistics 44:3-4 ► pp. 239 ff.
Eggenberger, Noëmi, Basil C. Preisig, Rahel Schumacher, Simone Hopfner, Tim Vanbellingen, Thomas Nyffeler, Klemens Gutbrod, Jean-Marie Annoni, Stephan Bohlhalter, Dario Cazzoli, René M. Müri & Antoni Rodriguez-Fornells
2016. Comprehension of Co-Speech Gestures in Aphasic Patients: An Eye Movement Study.
PLOS ONE 11:1 ► pp. e0146583 ff.
Evans, Kelli & Karen Hux
2011. Comprehension of indirect requests by adults with severe traumatic brain injury: Contributions of gestural and verbal information.
Brain Injury 25:7-8 ► pp. 767 ff.
Ferri, Francesca, Marianna Busiello, Giovanna C. Campione, Elisa De Stefani, Alessandro Innocenti, Gian Luca Romani, Marcello Costantini & Maurizio Gentilucci
2014. The eye contact effect in request and emblematic hand gestures.
European Journal of Neuroscience 39:5 ► pp. 841 ff.
Fröhlich, Marlen, Christine Sievers, Simon W. Townsend, Thibaud Gruber & Carel P. Schaik
2019. Multimodal communication and language origins: integrating gestures and vocalizations.
Biological Reviews 94:5 ► pp. 1809 ff.
Galati, Alexia & Arthur G. Samuel
2011. The role of speech-gesture congruency and delay in remembering action events.
Language and Cognitive Processes 26:3 ► pp. 406 ff.
GARCÍA-GÁMEZ, ANA B. & PEDRO MACIZO
2019. Learning nouns and verbs in a foreign language: The role of gestures.
Applied Psycholinguistics 40:2 ► pp. 473 ff.
Genty, E.
2019. Vocal–gestural combinations in infant bonobos: new insights into signal functional specificity.
Animal Cognition 22:4 ► pp. 505 ff.
Ginosar, Shiry, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens & Jitendra Malik
2019.
In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
► pp. 3492 ff.
Goldin-Meadow, Susan
2010. When gesture does and does not promote learning.
Language and Cognition 2:1 ► pp. 1 ff.
Goldin-Meadow, Susan & Sian L. Beilock
2010. Action’s Influence on Thought: The Case of Gesture.
Perspectives on Psychological Science 5:6 ► pp. 664 ff.
GOODRICH SMITH, WHITNEY & CARLA L. HUDSON KAM
2015. Children's use of gesture in ambiguous pronoun interpretation.
Journal of Child Language 42:3 ► pp. 591 ff.
Greenwood, David, Stephen Laycock & Iain Matthews
2017.
In Intelligent Virtual Agents [Lecture Notes in Computer Science, 10498],
► pp. 160 ff.
Gullberg, Marianne
2003.
In The Mind's Eye,
► pp. 685 ff.
Gullberg, Marianne
2005. L’expression orale et gestuelle de la cohésion dans le discours de locuteurs langue 2 débutants.
Acquisition et interaction en langue étrangère :23 ► pp. 153 ff.
Gullberg, Marianne
2006. Handling Discourse: Gestures, Reference Tracking, and Communication Strategies in Early L2.
Language Learning 56:1 ► pp. 155 ff.
Gullberg, Marianne
2010. Methodological reflections on gesture analysis in second language acquisition and bilingualism research.
Second Language Research 26:1 ► pp. 75 ff.
Gullberg, Marianne
2012.
In The Encyclopedia of Applied Linguistics,
Gullberg, Marianne & Kenneth Holmqvist
2002.
In Gesture and Sign Language in Human-Computer Interaction [Lecture Notes in Computer Science, 2298],
► pp. 206 ff.
Gullberg, Marianne & Sotaro Kita
2009. Attention to Speech-Accompanying Gestures: Eye Movements and Information Uptake.
Journal of Nonverbal Behavior 33:4 ► pp. 251 ff.
Gunter, Thomas C., J. E. Douglas Weinbrenner & Henning Holle
2015. Inconsistent use of gesture space during abstract pointing impairs language comprehension.
Frontiers in Psychology 6
Gurney, Pine & Wiseman
2013. The Gestural Misinformation Effect: Skewing Eyewitness Testimony Through Gesture.
The American Journal of Psychology 126:3 ► pp. 301 ff.
Gurney, Daniel J., Louise R. Ellis & Emily Vardon-Hynard
2016. The saliency of gestural misinformation in the perception of a violent crime.
Psychology, Crime & Law 22:7 ► pp. 651 ff.
Gurney, Daniel J., Neil Howlett, Karen Pine, Megan Tracey & Rachel Moggridge
2017. Dressing up posture: The interactive effects of posture and clothing on competency judgements.
British Journal of Psychology 108:2 ► pp. 436 ff.
Gurney, Daniel John
2015. What's Left Unsaid: How Nonverbal Influence Compares with Verbal Influence.
Psychiatry, Psychology and Law 22:3 ► pp. 465 ff.
Habets, Boukje, Sotaro Kita, Zeshu Shao, Asli Özyurek & Peter Hagoort
2011. The Role of Synchrony and Ambiguity in Speech–Gesture Integration during Comprehension.
Journal of Cognitive Neuroscience 23:8 ► pp. 1845 ff.
Hasegawa, Dai, Shinichi Shirakawa, Naoya Shioiri, Toshiki Hanawa, Hiroshi Sakuta & Kouzou Ohara
2015.
In Learning and Collaboration Technologies [Lecture Notes in Computer Science, 9192],
► pp. 361 ff.
Hasegawa, Komei & Yasushi Nakauchi
2013.
In 2013 IEEE RO-MAN,
► pp. 350 ff.
HASEGAWA, Komei & Yasushi NAKAUCHI
2014. Unconscious gestures that empower turn taking for telepresence robot.
Transactions of the JSME (in Japanese) 80:819 ► pp. DR0321 ff.
Heeyoung Kim, Sonya S. Kwak & Myungsuk Kim
2008.
In RO-MAN 2008 - The 17th IEEE International Symposium on Robot and Human Interactive Communication,
► pp. 494 ff.
Hirata, Yukari & Spencer D. Kelly
2010. Effects of Lips and Hands on Auditory Learning of Second-Language Speech Sounds.
Journal of Speech, Language, and Hearing Research 53:2 ► pp. 298 ff.
Holle, Henning, Thomas C. Gunter, Shirley-Ann Rüschemeyer, Andreas Hennenlotter & Marco Iacoboni
2008. Neural correlates of the processing of co-speech gestures.
NeuroImage 39:4 ► pp. 2010 ff.
Ianì, Francesco, Andrea Foiadelli & Monica Bucciarelli
2019. Mnemonic effects of action simulation from pictures and phrases.
Acta Psychologica 194 ► pp. 37 ff.
Johanson, Deborah L., Ho Seok Ahn & Elizabeth Broadbent
2020. Improving Interactions with Healthcare Robots: A Review of Communication Behaviours in Social and Healthcare Contexts.
International Journal of Social Robotics
Joue, Gina, Linda Boven, Klaus Willmes, Vito Evola, Liliana R. Demenescu, Julius Hassemer, Irene Mittelberg, Klaus Mathiak, Frank Schneider & Ute Habel
2020. Metaphor processing is supramodal semantic processing: The role of the bilateral lateral temporal regions in multimodal communication.
Brain and Language 205 ► pp. 104772 ff.
Kamiya, Nobuhiro
2018. The effect of learner age on the interpretation of the nonverbal behaviors of teachers and other students in identifying questions in the L2 classroom.
Language Teaching Research 22:1 ► pp. 47 ff.
Kelly, Spencer D., Peter Creigh & James Bartolotti
2010. Integrating Speech and Iconic Gestures in a Stroop-like Task: Evidence for Automatic Processing.
Journal of Cognitive Neuroscience 22:4 ► pp. 683 ff.
Kelly, Spencer D., Corinne Kravitz & Michael Hopkins
2004. Neural correlates of bimodal speech and gesture comprehension.
Brain and Language 89:1 ► pp. 253 ff.
Kelly, Spencer D., Tara McDevitt & Megan Esch
2009. Brief training with co-speech gesture lends a hand to word learning in a foreign language.
Language and Cognitive Processes 24:2 ► pp. 313 ff.
Kelly, Spencer D., Sarah Ward, Peter Creigh & James Bartolotti
2007. An intentional stance modulates the integration of gesture and speech during comprehension.
Brain and Language 101:3 ► pp. 222 ff.
Kendon, Adam
2001. Gesture as communication strategy.
Semiotica 2001:135
Kita, Sotaro & Thomas Stephen Davies
2009. Competing conceptual representations trigger co-speech representational gestures.
Language and Cognitive Processes 24:5 ► pp. 761 ff.
Krahmer, Emiel & Marc Swerts
2007. The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception.
Journal of Memory and Language 57:3 ► pp. 396 ff.
Latif, Nida, Agnès Alsius & K. G. Munhall
2018. Knowing when to respond: the role of visual information in conversational turn exchanges.
Attention, Perception, & Psychophysics 80:1 ► pp. 27 ff.
Le, Quoc Anh, Souheil Hanoune & Catherine Pelachaud
2011.
In 2011 11th IEEE-RAS International Conference on Humanoid Robots,
► pp. 134 ff.
Leonard, Thomas & Fred Cummins
2011. The temporal relation between beat gestures and speech.
Language and Cognitive Processes 26:10 ► pp. 1457 ff.
Levinson, Stephen C.
2003.
In Space in Language and Cognition,
Lopez-Ozieblo, Renia
2018. Disagreeing without a ‘no’: How teachers indicate disagreement in a Hong Kong classroom.
Journal of Pragmatics 137 ► pp. 1 ff.
Macoun, Alice & Naomi Sweller
2016. Listening and watching: The effects of observing gesture on preschoolers’ narrative comprehension.
Cognitive Development 40 ► pp. 68 ff.
McNeill, D.
2006.
In Encyclopedia of Language & Linguistics,
► pp. 58 ff.
McNeill, David
2015.
In International Encyclopedia of the Social & Behavioral Sciences,
► pp. 109 ff.
Mendoza-Denton, Norma & Stefanie Jannedy
2011. Semiotic Layering through Gesture and Intonation.
Journal of English Linguistics 39:3 ► pp. 265 ff.
Nathan, Mitchell J., Amelia Yeo, Rebecca Boncoddo, Autumn B. Hostetter & Martha W. Alibali
2019. Teachers’ attitudes about gesture for learning and instruction.
Gesture 18:1 ► pp. 31 ff.
Nirme, Jens, Magnus Haake, Agneta Gulz & Marianne Gullberg
2020. Motion capture-based animated characters for the study of speech–gesture integration.
Behavior Research Methods 52:3 ► pp. 1339 ff.
Otero, Nuno, Chrystopher Nehaniv, Dag Syrdal & Kerstin Dautenhahn
2006.
In ROMAN 2006 - The 15th IEEE International Symposium on Robot and Human Interactive Communication,
► pp. 533 ff.
Piper, Anne Marie & James D. Hollan
2009.
In ECSCW 2009,
► pp. 283 ff.
Pook, Polly K. & Dana H. Ballard
1996. Deictic human/robot interaction.
Robotics and Autonomous Systems 18:1-2 ► pp. 259 ff.
Provost, Emily Mower, Yuan Shangguan & Carlos Busso
2015. UMEME: University of Michigan Emotional McGurk Effect Data Set.
IEEE Transactions on Affective Computing 6:4 ► pp. 395 ff.
Rago, Francesco
2006.
In Fuzzy Logic and Applications [Lecture Notes in Computer Science, 2955],
► pp. 46 ff.
Roth, Wolff-Michael
2001. Gestures: Their Role in Teaching and Learning.
Review of Educational Research 71:3 ► pp. 365 ff.
Scandurra, Anna, Alessandra Alterisio, Lieta Marinelli, Paolo Mongillo, Gün Refik Semin & Biagio D’Aniello
2017. Effectiveness of verbal and gestural signals and familiarity with signal-senders on the performance of working dogs.
Applied Animal Behaviour Science 191 ► pp. 78 ff.
Sekine, Kazuki & Sotaro Kita
2015. Development of multimodal discourse comprehension: cohesive use of space by gestures.
Language, Cognition and Neuroscience 30:10 ► pp. 1245 ff.
Sekine, Kazuki & Sotaro Kita
2017. The listener automatically uses spatial story representations from the speaker's cohesive gestures when processing subsequent sentences without gestures.
Acta Psychologica 179 ► pp. 89 ff.
Sekine, Kazuki, Hannah Sowden & Sotaro Kita
2015. The Development of the Ability to Semantically Integrate Information in Speech and Iconic Gesture in Comprehension.
Cognitive Science 39:8 ► pp. 1855 ff.
Sime, Daniela
2006. What do learners make of teachers' gestures in the language classroom?.
IRAL - International Review of Applied Linguistics in Language Teaching 44:2
Skipper, Jeremy I., Susan Goldin-Meadow, Howard C. Nusbaum & Steven L. Small
2007. Speech-associated gestures, Broca’s area, and the human mirror system.
Brain and Language 101:3 ► pp. 260 ff.
Skipper, Jeremy I., Susan Goldin-Meadow, Howard C. Nusbaum & Steven L. Small
2009. Gestures Orchestrate Brain Networks for Language Understanding.
Current Biology 19:8 ► pp. 661 ff.
Smith, Whitney Goodrich & Carla L. Hudson Kam
2012. Knowing ‘who she is’ based on ‘where she is’: The effect of co-speech gesture on pronoun comprehension.
Language and Cognition 4:2 ► pp. 75 ff.
So, Wing Chee, Colin Sim Chen-Hui & Julie Low Wei-Shan
2012. Mnemonic effect of iconic gesture and beat gesture in adults and children: Is meaning in gesture important for memory recall?.
Language and Cognitive Processes 27:5 ► pp. 665 ff.
Sowińska, Agnieszka & Monika Boruta-Żywiczyńska
2021. Gestures in patients’ presentation of medically unexplained symptoms (MUS).
Gesture 19:1 ► pp. 97 ff.
Stam, Gale & Marion Tellier
2017.
In Why Gesture? [Gesture Studies, 7],
► pp. 353 ff.
Stam, Gale A.
2012.
In The Encyclopedia of Applied Linguistics,
Stolzenwald, Janis & Paul Bremner
2017.
In 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN),
► pp. 430 ff.
Stone, Matthew & Insuk Oh
2008.
In Modeling Communication with Robots and Virtual Humans [Lecture Notes in Computer Science, 4930],
► pp. 57 ff.
Sueyoshi, Ayano & Debra M. Hardison
2005. The Role of Gestures and Facial Cues in Second Language Listening Comprehension.
Language Learning 55:4 ► pp. 661 ff.
ten Brinke, Leanne & Max Weisbuch
2020. How verbal-nonverbal consistency shapes the truth.
Journal of Experimental Social Psychology 89 ► pp. 103978 ff.
Tree, Jean E. Fox
2007. Functional Spontaneous Speech Phenomena.
Perspectives on Fluency and Fluency Disorders 17:2 ► pp. 17 ff.
Vulchanova, Mila, Valentin Vulchanov, Isabella Fritz & Evelyn A. Milburn
2019. Language and perception: Introduction to the Special Issue “Speakers and Listeners in the Visual World”.
Journal of Cultural Cognitive Science 3:2 ► pp. 103 ff.
Wanphet, Phalangchok & Jalila Sfaxi
2019. Exploring bilingual children’s integration of gestures into talk-in-interaction.
Language and Dialogue 9:3 ► pp. 349 ff.
Wessler, Janet & Jochim Hansen
2017. Temporal Closeness Promotes Imitation of Meaningful Gestures in Face-to-Face Communication.
Journal of Nonverbal Behavior 41:4 ► pp. 415 ff.
Willems, Roel M., Aslı Özyürek & Peter Hagoort
2007. When Language Meets Action: The Neural Integration of Gesture and Speech.
Cerebral Cortex 17:10 ► pp. 2322 ff.
Wu, Ying Choon & Seana Coulson
2005. Meaningful gestures: Electrophysiological indices of iconic gesture comprehension.
Psychophysiology 42:6 ► pp. 654 ff.
Wu, Ying Choon & Seana Coulson
2007. How iconic gestures enhance communication: An ERP study.
Brain and Language 101:3 ► pp. 234 ff.
Wu, Ying Choon & Seana Coulson
2007. Iconic gestures prime related concepts: An ERP study.
Psychonomic Bulletin & Review 14:1 ► pp. 57 ff.
Wu, Ying Choon, Seana Coulson & Krish Sathian
2014. A Psychometric Measure of Working Memory Capacity for Configured Body Movement.
PLoS ONE 9:1 ► pp. e84834 ff.
Wu, Zhiming, Tao Lin, Hongyan Xu & Ningjiu Tang
2017. Predicting personality associations evoked by multicolored appearance of virtual agents: An exploratory study.
Color Research & Application 42:6 ► pp. 753 ff.
Xenofontos, Constantinos & Paul Andrews
2014. Defining mathematical problems and problem solving: prospective primary teachers’ beliefs in Cyprus and England.
Mathematics Education Research Journal 26:2 ► pp. 279 ff.
Özer, Demet & Tilbe Göksun
2020. Visual-spatial and verbal abilities differentially affect processing of gestural vs. spoken expressions.
Language, Cognition and Neuroscience 35:7 ► pp. 896 ff.
This list is based on CrossRef data as of 18 april 2021. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.