Article published In:
GestureVol. 20:3 (2021) ► pp.354–375
High verbal working memory load impairs gesture-speech integration
Evidence from a dual task paradigm
While previous studies have shown the importance of visuo-spatial working memory in the processing of co-speech iconic gestures, clear evidence for a potential involvement of the verbal working memory (vWM) is currently lacking. To address this issue, participants in the present study were presented with a dual task paradigm. The main outcome variable was the performance on a Stroop-like gesture task which provides a behavioural index of gesture-speech integration. Participants performed this task under conditions of either high or low concurrent vWM load. Unlike in previous studies, the number of words to remember in the high load condition was determined by their individual verbal span rather than being fixed. Results showed reaction time costs in the form of longer reaction times for semantically incongruent gesture-speech combinations as compared to congruent combinations. However, this semantic congruency effect disappeared when the vWM load increased. This result suggests a causal involvement of verbal working memory capacity in gesture-speech integration.
Keywords: gesture-speech integration, verbal resources hypothesis, iconic gestures, gesture-speech comprehension, verbal working memory
Article outline
- Introduction
- Methods
- Participants
- Materials
- Digit Span task
- Dual-task paradigm
- Experimental design and statistical analysis
- Results
- Main analysis
- Gender Classification Task
- Secondary verbal working memory task
- Main analysis
- Discussion
- Conclusion
-
References
References
Alibali, Martha W.
(2005) Gesture in spatial cognition: Expressing, communicating, and thinking about spatial information. Spatial Cognition and Computation,
5
(4), 307–331. 
Alibali, Martha. W., Flevares, Lucia M., & Goldin-Meadow, Susan
(1997) Assessing knowledge conveyed in gesture: Do teachers have the upper hand? Journal of Educational Psychology,
89
(1), 183–193.
Baddeley, Alan D.
(1992) Working memory. Science,
225
(5044), 556–559.
(2012) Working memory: Theories, Models and Controversies. Annual Reviews of Psychology,
63
1, 1–29.
Baddeley, Alan D., & Hitch, Graham J.
(1974) Working memory. In Gordon H. Bower (Ed.), The psychology of learning and motivation: Advances in research and theory (Vol. 81, pp. 47–89). London: Academic Press.
Beattie, Geoffrey & Shovelton, Heather
(2002) What properties of talk are associated with the generation of spontaneous hand gestures? British Journal of Social Psychology,
41
1, 403–417.
Besner, Derek & Stolz, Jennifer A.
(1999) Unconsciously controlled processing: The Stroop effect reconsidered. Psychonomic Bulletin & Review,
6
(3), 449–455.
Boehringer, Andreas, Macher, Katja, Dukart, Juergen, Villringer, Arno, & Pleger, Burkhard
(2013) Cerebellar transcranial direct stimulation modulates verbal working memory. Brain Stimulation,
6
1, 649–653.
Chu, Minguyan, Meyer, Antje, Foulkes, Lucy, & Kita, Sotaro
(2014) Individual differences in frequency and saliency of speech-accompanying gestures: The role of cognitive abilities and empathy. Journal of Experimental Psychology: General,
143
(2), 694–709.
Daneman, Meredith & Carpenter, Patricia A.
(1980) Individual differences in working memory and reading. Journal of Verbal Learning and Verbal Behavior,
19
1, 450–466.
Delaloye, Christophe, Ludwig, Catherine, Borella, Erika, Chicherio, Christian, & de Ribaupierre, Anik
(2008) The reading span as a measure of working memory capacity: Norms based on a French speaking population of 775 younger and older adults. Revue européenne de Psychologie Appliquée,
58
1, 89–103.
Dick, Anthony, Goldin-Meadow, Susan, Hasson, Uri, Skipper, Jeremy, & Small, Steven L.
(2009) Co-speech gestures influence neural activity in brain regions associated with processing semantic information. Human Brain Mapping,
30
1, 3509–3526.
Engel, Randall W.
(2002) Working memory capacity as executive attention. Current Directions in Psychological Science,
11
(1), 19–23.
Feyereisen, Pierre & Havard, Isabelle
(1999) Mental imagery and production of hand gestures while speaking in younger and older adults. Journal of Non-Verbal Behavior,
23
1, 153–171.
Gauthier, Janel & Bouchard, Stéphane
(1993) A French-Canadian adaptation of the revised version of Spielberger’s State-Trait Anxiety Inventory. Canadian Journal of Behavioural Science,
25
(4), 559–578.
Gillepsie, Maureen, James, Ariel N., Federmeier, Kara D., & Watson, Duane G.
(2014) Verbal working memory predicts co-speech gesture: Evidence from individual differences. Cognition,
132
1, 174–180.
Green, Antonia, Straube, Benjamin, Weis, Susanne, Jansen, Andreas, Wilmes, Klaus, Konrad, Kerstin, & Kircher, Tilo
(2009) Neural integration of iconic and unrelated co-verbal gestures: A functional MRI study. Human Brain Mapping,
30
(10), 3309–3324.
Habets, Boukje, Kita, Sotaro, Shao, Zeshu, Özyürek, Aslı, & Hagoort, Peter
(2010) The role of synchrony and ambiguity in speech-gesture integration during comprehension. Journal of Cognitive Neuroscience,
23
(8), 1845–1854.
Hadar, Uri & Butterworth, Brian
(1997) Iconic gestures, imagery and word retrieval in speech. Semiotica,
115
(1/2), 147–172.
Hadar, Uri & Pinchas-Zamir, Lian
(2004) The semantic specificity of gesture. Implications for gesture classification and function. Journal of Language and Social Psychology,
23
(2), 204–214.
Hadwin, Julie, Brogan, Joanna, & Stevenson, Jim
(2005) State anxiety and working memory in children: A test of processing efficiency theory. Educational Psychology,
25
(4), 379–393.
Holle, Henning & Gunter, Thomas C.
(2007) The role of iconic gestures in speech disambiguation: ERP evidence. Journal of Cognitive Neuroscience,
19
(7), 1175–1192.
Holle, Henning, Gunter, Thomas C., Rueschemeyer, Shirley, Hennenlotter, Andreas, & Iacoboni, Marco
(2008) Neural correlates of the processing of co-speech gestures. NeuroImage,
39
(4), 2010–2024.
Holler, Judith, Shovelton, Heather, & Beattie, Geoffrey
(2009) Do iconic hand gestures really contribute to the communication of semantic information in face-to-face context? Journal of Non-Verbal Behavior,
33
(2), 73–88.
Hostetter, Autumn B.
(2011) When do gestures communicate? A meta-analysis. Psychological Bulletin,
137
(2), 297–315.
Kelly, Spencer D., Barr, Dale J., Church, R. Breckinridge, & Lynch, Katheryn
(1999) Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory. Journal of Memory and Langage,
40
1, 577–592.
Kelly, Spencer D., Creigh, Peter, & Bartolotti, James
(2009) Integrating speech and iconic gestures in a stroop-like task: Evidence for automatic processing. Journal of Cognitive Neuroscience,
22
(4), 683–694.
Kelly, Spencer, Kravitz, Corinne, & Hopkins, Michael
(2004) Neural correlates of bimodal speech and gesture comprehension. Brain and Language,
89
(1), 253–260.
Kelly, Spencer, Özyürek, Aslı, & Maris, Eric
(2010) Two sides of the same coin: Speech and gesture mutually interact to enhance communication. Psychological Science,
21
(2), 260–267.
Kelly, Spencer, Ward, Sarah, Creigh, Peter, & Bartolotti, James
(2007) An intentional stance modulates the integration of gesture and speech during comprehension. Brain and Language,
101
(3), 222–233.
Kita, Sotaro
(2000) How representational gestures help speaking. In David McNeill (Ed.), Language and gesture (pp. 162–185). Cambridge, UK: Cambridge University Press.
Kita, Sotaro & Özyürek, Aslı
(2003) What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language,
48
(1), 16–32.
Krauss, Robert M., Chen, Yihsiu, & Chawla, Purnima
(1996) Non-verbal behavior and non-verbal communication: What do conversational hand gesture tell us? In M. Zanna (Ed.), Advances in experimental social psychology (pp. 389–450). San Diego: Academic Press.
Krauss, Robert M., Morrel-Samuels, Palmer, & Colasante, Christina
(1991) Do conversational hand gestures communicate? Journal of Personality and Social Psychology,
61
(5), 743–754.
McNeill, David
(1985) So you think gestures are non-verbal? Psychological Review,
92
(3), 350–371.
(1992) Hand and mind. Chicago: University of Chicago Press.
McNeill, David, Cassell, Justine, & McCullough, Karl-Erik
(1994) Communicative effects of speech-mismatched gestures. Research on Language and Social Interaction,
27
(3), 223–237.
Momsen, Jacob, Gordon, Jared, Wu, Ying Choon, & Coulson, Seana
(2020) Verbal working memory and co- speech gesture processing. Brain and Cognition,
146
1, 105640.
Morrel-Samuels, Palmer & Krauss, Robert M.
(1992) Word familiarity predicts asynchrony of hand gestures and speech. Journal of Experimental Psychology: Learning, Memory, and Cognition,
18
(3), 615–622.
Navon, David & Miller, Jeff
(2002) Queuing or sharing? A critical evaluation of the single- bottleneck notion. Cognitive Psychology,
44
1, 193–251.
Obermeier, Christian, Holle, Henning, & Gunter, Thomas
(2011) What iconic gesture fragments reveal about gesture-speech integration: When synchrony is lost, memory can help. Journal of Cognitive Neuroscience,
23
(7), 1648–1663.
Ovroye, Acacia L. & Storm, Benjamin C.
(2019) Remembering what was said and done: The activation and facilitation of memory for gesture as a consequence of retrieval. Journal of Experimental Psychology: Learning, Memory and Cognition,
45
(3), 526–534.
Owens, Matthews, Stevenson, Jim, Hadwin, Julie A., & Norgate, Roger
(2014) When does anxiety help or hinder test performance? The role of working memory capacity. British Journal of Psychology,
105
(1), 92–101.
Özer, Demet & Göksun, Tilbe
(2020a) Gesture use and processing: A review on individual differences in cognitive resources. Frontiers in Psychology,
11
1, 573555.
(2020b) Visual-spatial and verbal abilities differentially affect processing of gestural vs. spoken expressions. Language, Cognition and Neuroscience,
35
(7), 896–914.
Özyürek, Aslı, Willems, Roel M., Kita, Sotaro, & Hagoort, Peter
(2007) On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials. Journal of Cognitive Neuroscience,
19
1, 605–616.
Smyth, Mary M. & Pendleton, Lindsay R.
(1990) Space and movement in working memory. The Quarterly Journal of Experimental Psychology, Section A: Human Experimental Psychology,
42
(2), 291–304.
Vytal, Katherine E., Cornwell, Brian R., Arkin, Nicole Esther, Letkiewicz, Allison M., & Grillon, Christian
(2013) The complex interaction between anxiety and cognition: insight from spatial and verbal working memory. Frontiers in Human Neuroscience,
7
1, 93.
Wagner, Susan M., Nusbaum, Howard, & Goldin-Meadow, Susan
(2004) Probing the mental representation of gesture: Is handwaving spatial? Journal of Memory and Language,
50
1, 395–407.
Wechsler, David
(2008) Wechsler adult intelligence scale – fourth edition.
Wolf, Dhana, Rekittke, Linn-Marlen, Mittelberg, Irene, Klasen, Martin, & Mathiak, Klaus
(2017) Perceived conventionality in co-speech gestures involves the fronto-temporal language network. Frontiers in Human Neuroscience,
11
1, 573.
Wu, Ying Choon & Coulson, Seana
(2007) How iconic gestures enhance communication: An ERP study. Brain & Language,
101
1, 234–245.
(2011) Are depictive gestures like pictures? Commonalities and differences in semantic processing. Brain & Language,
119
1, 184–195.
(2014) Co-speech iconic gestures and visuo-spatial working memory. Acta Psychologia,
153
1, 39–50.
(2015) Iconic gestures facilitate discourse comprehension in individuals with superior immediate memory for body configurations. Psychological Science,
26
(11), 1717–1727.
Zhao, Wanying, Riggs, Kevin, Schindler, Igor, & Holle, Henning
(2018) Transcranial Magnetic Stimulation over left inferior frontal and posterior temporal cortex disrupts gesture-speech integration. The Journal of Neuroscience,
38
(8), 1891–1900.