Cover not available
Article published In:
Gesture
Vol. 17:2 (2018) ► pp.268–290

Does seeing gesture lighten or increase the load?

Effects of processing gesture on verbal and visuospatial cognitive load

| Kalamazoo College
| Kalamazoo College
| Kalamazoo College
| Kalamazoo College

We examined the cognitive resources involved in processing speech with gesture compared to the same speech without gesture across four studies using a dual-task paradigm. Participants viewed videos of a woman describing spatial arrays either with gesture or without. They then attempted to choose the target array from among four choices. Participants’ cognitive load was measured as they completed this comprehension task by measuring how well they could remember the location and identity of digits in a secondary task. We found that addressees experience additional visuospatial load when processing gestures compared to speech alone, and that the load primarily comes when addressees attempt to use their memory of the descriptions with gesture to choose the target array. However, this cost only occurs when gestures about horizontal spatial relations (i.e., left and right) are produced from the speaker’s egocentric perspective.

Keywords: gesture comprehension, cognitive load, visuospatial resources, verbal resources, horizontal spatial relations
Article outline
  • Study 1
    • Method
      • Participants
      • Materials and measures
        • Primary comprehension task
        • Secondary memory task
    • Procedure
    • Results and discussion
  • Study 2
    • Method
      • Participants
      • Procedure
      • Results and discussion
  • Study 3
    • Method
      • Participants
      • Procedure
      • Results and discussion
  • Study 4
    • Method
      • Participants
      • Procedure
      • Results and discussion
  • General discussion
  • Conclusion
  • References
Published online: 26 June 2019
DOI logo
https://doi.org/10.1075/gest.17017.hos
References
Alibali, Martha W.
(2005) Gesture in spatial cognition: Expressing, communicating, and thinking about spatial information. Spatial Cognition & Computation, 51, 307–331. DOI logoGoogle Scholar
Baddeley, Alan
(1992) Working memory. Science, 2551, 556–559. DOI logoGoogle Scholar
Goldin-Meadow, Susan, Howard Nusbaum, Spencer D. Kelly, & Susan Wagner
(2001) Explaining math: Gesturing lightens the load. Psychological Science, 121, 516–522. DOI logoGoogle Scholar
Holler, Judith, Heather Shovelton, & Geoffrey Beattie
(2009) Do iconic hand gestures really contribute to the communication of semantic information in a face-to-face context? Journal of Nonverbal Behavior, 331, 73–88. DOI logoGoogle Scholar
Hostetter, Autumn B.
(2011) When do gestures communicate? A meta-analysis. Psychological Bulletin, 1371, 297–315. DOI logoGoogle Scholar
Kelly, Spencer D., Peter Creigh, & James Bartolotti
(2009) Integrating speech and iconic gestures in a Stroop-like task: Evidence for automatic processing. Journal of Cognitive Neuroscience, 221, 683–694. DOI logoGoogle Scholar
Kelly, Spencer D. & Leslie H. Goldsmith
(2004) Gesture and right hemisphere involvement in evaluating lecture material. Gesture, 41, 25–42. DOI logoGoogle Scholar
Kelly, Spencer, Meghan Healey, Aslı Özyürek, & Judith Holler
(2015) The processing of speech, gesture, and action during language comprehension. Psychonomic Bulletin & Review, 221, 517–523. DOI logoGoogle Scholar
Kelly, Spencer D., Corinne Kravitz, & Michael Hopkins
(2004) Neural correlates of bimodal speech and gesture comprehension. Brain and Language, 891, 253–260. DOI logoGoogle Scholar
Kelly, Spencer D., Aslı Özyürek, & Eric Maris
(2010) Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension. Psychological Science, 211, 260–267. DOI logoGoogle Scholar
Kelly, Spencer D., Sarah Ward, Peter Creigh, & James Bartolotti
(2007) An intentional stance modulates the integration of gesture and speech during comprehension. Brain and Language, 1011, 222–233. DOI logoGoogle Scholar
Kendon, Adam
(1994) Do gestures communicate? A review. Research on Language and Social Interaction, 271, 175–200. DOI logoGoogle Scholar
Lakens, Daniël
(2013) Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 41, 863. DOI logoGoogle Scholar
Maricchiolo, Fridanna, Augusto Gnisci, Marino Bonaiuto, & Gianluca Ficca
(2009) Effects of different types of hand gestures in persuasive speech on receivers’ evaluations. Language and Cognitive Processes, 241, 239–266. DOI logoGoogle Scholar
Paivio, Allan
(1991) Dual coding theory: Retrospect and current status. Canadian Journal of Psychology, 451, 255–287. DOI logoGoogle Scholar
Post, Lysanne S., Tamara van Gog, Fred Paas, & Rolf A. Zwaan
(2013) Effects of simultaneously observing and making gestures while studying grammar animations on cognitive load and learning. Computers in Human Behavior, 291, 1450–1455. DOI logoGoogle Scholar
Pyers, Jennie E., Pamela Perniss, & Karen Emmorey
(2015) Viewpoint in the visual-spatial modality: The coordination of spatial perspective. Spatial Cognition & Computation: An Interdisciplinary Journal, 151, 143–169. DOI logoGoogle Scholar
Rueckert, Linda, Ruth Breckinridge Church, Andrea Avila, & Theresa Trejo
(2017) Gesture enhances learning of a complex statistical concept. Cognitive Research, 21, 1–6. DOI logoGoogle Scholar
Sueyoshi, Ayano & Debra M. Hardison
(2005) The role of gestures and facial cues in second language listening comprehension. Language Learning, 551, 661–699. DOI logoGoogle Scholar
Sweller, John
(2005) The redundancy principle in multimedia learning. In Richard E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 159–168). Cambridge University Press: Cambridge, UK. DOI logoGoogle Scholar
Wagner, Susan M., Howard Nusbaum, & Susan Goldin-Meadow
(2004) Probing the mental representation of gesture: Is hand-waving spatial? Journal of Memory and Language, 501, 395–407. DOI logoGoogle Scholar
Wickelgren, Wayne A.
(1965) Acoustic similarity and intrusion errors in short-term memory. Journal of Experimental Psychology, 701, 102–108. DOI logoGoogle Scholar
Wu, Ying Choon & Seana Coulson
(2014) Co-speech iconic gestures and visuo-spatial working memory. Acta Psychologica, 1531, 39–50. DOI logoGoogle Scholar
Cited by

Cited by 5 other publications

Nathan, Mitchell J., Amelia Yeo, Rebecca Boncoddo, Autumn B. Hostetter & Martha W. Alibali
2019. Teachers’ attitudes about gesture for learning and instruction. Gesture 18:1 ► pp. 31 ff. DOI logo
Wu, Ying Choon, Horst M. Müller & Seana Coulson
2022. Visuospatial Working Memory and Understanding Co-Speech Iconic Gestures: Do Gestures Help to Paint a Mental Picture?. Discourse Processes 59:4 ► pp. 275 ff. DOI logo
Yuan, Qingshu, Keming Chen, Qihao Yang, Zhigeng Pan, Jin Xu & Zhengwei Yao
2023. Exploring Intuitive Visuo-Tactile Interaction Design for Culture Education: A Chinese-Chess-Based Case Study. International Journal of Human–Computer Interaction ► pp. 1 ff. DOI logo
Yuan, Qingshu, Ruonan Wang, Zhigeng Pan, QianYu Meng, Shuchang Xu, Zihan Wang & Jiaxin Liu
2022. TIPTAB: A tangible interactive projection tabletop for virtual experiments. Computer Applications in Engineering Education 30:5 ► pp. 1350 ff. DOI logo
Özer, Demet & Tilbe Göksun
2020. Gesture Use and Processing: A Review on Individual Differences in Cognitive Resources. Frontiers in Psychology 11 DOI logo

This list is based on CrossRef data as of 30 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.