Cover not available
Article published In:
Gesture
Vol. 19:1 (2020) ► pp.41–71

Emotion matters

The effect of hand gesture on emotionally valenced sentences

| Colgate University
| Colgate University
Recent theories and neural models of co-speech gesture have extensively considered its cognitive role in language comprehension but have ignored the emotional function. We investigated the integration of speech and co-speech gestures in memory for verbal information with different emotional connotations (either positive, negative, or neutral). In a surprise cued-recall task, gesture boosted memory for speech with all three emotional valences. Interestingly, gesture was more likely to become integrated into memory of neutrally and positively valenced speech than negatively valenced speech. The results suggest that gesture-speech integration is modulated by emotional valence of speech, which has implications for the emotional function of gesture in language comprehension.
Keywords: gesture, language, multimodal, emotion, emotional valence, memory
Article outline
  • Cognitive function of gesture: Production
  • Cognitive function of gesture: Comprehension
  • Emotional valence and gesture
  • Present study
  • Method
    • Participants
    • Materials
      • Video stimulus
      • Stimulus norming
      • Filler task
      • Memory tests
    • Procedure
    • Coding
    • Design
  • Results
    • Cued-recall responses
      • Remembering of speech
      • Addition of information from gesture
      • Remembering speech and adding from gesture
  • Discussion
    • Gesture and memory
    • Emotional valence and memory
  • Conclusion
  • Acknowledgements
  • Notes
  • Cued-recall response coding manual
  • References
Published online: 18 May 2021
DOI logo
https://doi.org/10.1075/gest.19029.lev
References (54)
References
Adelman, James S. & Estes, Zachary. (2013). Emotion and memory: A recognition advantage for positive and negative words independent of arousal. Cognition, 1291, 530–535. DOI logoGoogle Scholar
Baumeister, Roy F., Bratslavsky, Ellen, Finkenauer, Catrin, & Vohs, Kathleen D. (2001). Bad is stronger than good. Review of General Psychology, 5 (4), 323–370. DOI logoGoogle Scholar
Bohn, Annette & Berntsen, Dorthe. (2007). Pleasantness bias in flashbulb memories: Positive and negative flashbulb memories of the fall of the Berlin Wall among east and west Germans. Memory & Cognition, 35 (3), 565–577. DOI logoGoogle Scholar
Borg, Céline, Bouazza, Zineb, Godeau, Marielle, Getenet, Jean-Claude, & Chainay, Hanna. (2018). Effect of emotion and type of encoding on memory for actions: Verbal and subjected-performed tasks. Dementia and Geriatric Cognitive Disorders, 45 (3/4), 162–179. DOI logoGoogle Scholar
Bowen, Holly J., Kark, Sarah M., & Kensinger, Elizabeth A. (2018). NEVER forget: negative emotional valence enhances recapitulation. Psychonomic Bulletin & Review, 25 (3), 870–891. DOI logoGoogle Scholar
Broaders, Sara C. & Goldin-Meadow, Susan. (2010). Truth is at hand: How gesture adds information during investigative interviews. Psychological Science, 21 (5), 623–628. DOI logoGoogle Scholar
Charles, Susan T., Mather, Mara, & Carstensen, Laura L. (2003). Aging and emotional memory: The forgettable nature of negative images for older adults. Journal of Experimental Psychology: General, 132 (2), 310–324. DOI logoGoogle Scholar
Church, R. Breckinridge, Alibali, Martha W., & Kelly, Spencer D. (Eds.). (2017). Why gesture? How the hands function in speaking, thinking and communicating. Amsterdam & Philadelphia: John Benjamins Publishing Company. DOI logoGoogle Scholar
Church, R. Breckinridge, Garber, Phillip, & Rogalski, Kathryn. (2007). The role of gesture in memory and social communication. Gesture, 7 (2), 137–158. DOI logoGoogle Scholar
Cohen, Ronald L. (1989). Memory for action events: The power of enactment. Educational Psychology Review, 1 (1), 57–80. DOI logoGoogle Scholar
Darwin, Charles. (1872). The expression of the emotions in man and animals. London, England: John Murray. DOI logoGoogle Scholar
De Stefani, Elisa & De Marco, Doriana. (2019). Gesture, language and emotional communication: An embodied view of social interaction. Frontiers in Psychology, 101, 2063. DOI logoGoogle Scholar
Dewhurst, Stephen A. & Parry, Lisa A. (2000). Emotionality, distinctiveness, and recollective experience. European Journal of Cognitive Psychology, 12 (4), 541–551. DOI logoGoogle Scholar
Ekman, Paul & Friesen, Wallace V. (1969). The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica, 1 (1), 49–98. DOI logoGoogle Scholar
(1972). Hand movements. Journal of Communication, 22 (4), 353–374. DOI logoGoogle Scholar
Feyereisen, Pierre. (2009). Enactment effects and integration processes in younger and older adults’ memory for actions. Memory, 17 (4), 374–385. DOI logoGoogle Scholar
Gallese, Vittorio. (2018). Embodied simulation and its role in cognition. Reti, saperi, linguaggi, 7 (13), 31–46.Google Scholar
Goldin-Meadow, Susan & Morford, Marolyn. (1985). Gesture in early child language: Studies of deaf and hearing children. Merrill-Palmer Quarterly, 31 (2), 145–176.Google Scholar
Hirata, Yukari, Kelly, Spencer D., Huang, Jessica, & Manansala, Michael. (2014). Effects of hand gestures on auditory learning of second-language vowel length contrasts. Journal of Speech, Language, and Hearing Research, 57 (6), 2090–2101. DOI logoGoogle Scholar
Holland, Alisha C. & Kensinger, Elizabeth A. (2012). Younger, middle-aged, and older adults’ memories for the 2008 Presidential election. Journal of Applied Research on Memory and Cognition, 1 (3), 163–170. DOI logoGoogle Scholar
Hostetter, Autumn B. (2011). When do gestures communicate? A meta-analysis. Psychological Bulletin, 137 (2), 297–315. DOI logoGoogle Scholar
Hostetter, Autumn B. & Alibali, Martha W. (2008). Visible embodiment: Gestures as simulated action. Psychonomic Bulletin & Review, 15 (3), 495–514. DOI logoGoogle Scholar
(2018). Gesture as simulated action: Revisiting the framework. Psychonomic Bulletin & Review, 261, 721–752. DOI logoGoogle Scholar
Kelly, Spencer D. (2017). Exploring the boundaries of gesture-speech integration during language comprehension. In R. B. Church, M. W. Alibali, & S. D. Kelly (Eds.), Why gesture? How the hands function in speaking, thinking and communicating (pp. 243–266). Amsterdam & Philadelphia: John Benjamins Publishing Company. DOI logoGoogle Scholar
Kelly, Spencer D., Barr, Dale J., Church, R. Breckinridge, & Lynch, Katheryn. (1999). Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory. Journal of Memory and Language, 401, 577–592. DOI logoGoogle Scholar
Kelly, Spencer D., Hirata, Yukari, Manansala, Michael, & Huang, Jessica. (2014). Exploring the role of hand gestures in learning novel phoneme contrasts and vocabulary in a second language. Frontiers in Psychology, 5, 673, 1–11. DOI logoGoogle Scholar
Kelly, Spencer D. & Lee, Angela L. (2012). When actions speak too much louder than words: Hand gestures disrupt word learning when phonetic demands are high. Language and Cognitive Processes, 27 (6), 793–807. DOI logoGoogle Scholar
Kendon, Adam. (1972). Some relationships between body motion and speech. In Aron Wolfe Siegman & Benjamin Pope (Eds.), Studies in dyadic communication (pp. 177–210). New York: Pergamon Press. DOI logoGoogle Scholar
. (1980). Gesticulation and speech: Two Aspects of the. In Mary R. Key (Ed.), The relationship of verbal and nonverbal communication (pp. 207–227). The Hague: Mouton. DOI logoGoogle Scholar
Kensinger, Elizabeth A. (2009). Remembering the details: Effects of emotion. Emotion Review, 1 (2), 99–113. DOI logoGoogle Scholar
Kensinger, Elizabeth A. & Choi, Elizabeth S. (2009). When side matters: Hemispheric processing and the visual specificity of emotional memories. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35 (1), 247–253.Google Scholar
Kensinger, Elizabeth A. & Corkin, Suzanne. (2004). Two routes to emotional memory: Distinct neural processes for valence and arousal. Proceedings of the National Academy of Sciences, 101 (9), 3310–3315. DOI logoGoogle Scholar
Kensinger, Elizabeth A., Garoff-Eaton, Rachel J., & Schacter, Daniel L. (2006). Memory for specific visual details can be enhanced by negative arousing content. Journal of Memory and Language, 541, 99–112. DOI logoGoogle Scholar
Kensinger, Elizabeth A. & Schacter, Daniel L. (2006). When the red sox shocked the yankees: Comparing negative and positive memories. Psychonomic Bulletin & Review, 13 (5), 757–763. DOI logoGoogle Scholar
Krahmer, Emiel & Swerts, Marc. (2007). The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception. Journal of Memory and Language, 57 (3), 396–414. DOI logoGoogle Scholar
Levine, Linda J. & Bluck, Susan. (2004). How emotions fade: valence, appraisals, and the emotional impact of remembered events. In S. P. Shohov (Ed.), Advances in psychology research (Vol. 301, pp. 3–20). Hauppauge, NY: Nova Science Publishers.Google Scholar
Macedonia, Manuela, Müller, Karsten, & Friederici, Angela D. (2011). The impact of iconic gestures on foreign language word learning and its neural substrate. Human Brain Mapping, 32 (6), 982–998. DOI logoGoogle Scholar
Matheson, Heath E. & Barsalou, Lawrence W. (2018). Embodiment and grounding in cognitive neuroscience. Stevens’ handbook of experimental psychology and cognitive neuroscience, 31, 1–27. DOI logoGoogle Scholar
McNeill, David. (1992). Hand and mind. Chicago: The University of Chicago Press.Google Scholar
McNeill, David & Levy, Elena. (1982). Conceptual representations in language activity and gesture. In R. J. Jarvella & W. Klein (Eds.), Speech, place, and action: Studies in deixis and related topics (pp. 271–295). New York: John Wiley & Sons.Google Scholar
McNeill, David, Pedelty, Laura L., & Levy, Elena T. (1990). Speech and gesture. Advances in Psychology, 701, 203–256. DOI logoGoogle Scholar
Nascimento, Juliana Maria Steffen do, Grüdtner Buratto, Luciano, Schaefer, Alexandre, & Milnitsky Stein, Lilian. (2016). Emotional valence and perceived event frequency affect memory accuracy for a personally relevant life event. Applied Cognitive Psychology, 301, 1020–1029. DOI logoGoogle Scholar
Nilsson, Lars-Göran & Bäckman, Lars. (1989). Implicit memory and the enactment of verbal instructions. In Stephan Lewandowsky, John C. Dunn, & Kim Kirsner (Eds.), Implicit memory: Theoretical issues (pp. 173–183). Hillsdale, NJ: Erlbaum.Google Scholar
Ochsner, Kevin N. (2000). Are affective events richly recollected or simply familiar? The experience and process of recognizing feelings past. Journal of Experimental Psychology: General, 129 (2), 242–261. DOI logoGoogle Scholar
Post, Lysanne S., van Gog, Tamara, Paas, Fred, & Zwaan, Rolf A. (2013). Effects of simultaneously observing and making gestures while studying grammar animations on cognitive load and learning. Computers in Human Behavior, 29 (4), 1450–1455. DOI logoGoogle Scholar
Pouw, Wim T. J. L., de Nooijer, Jaqueline A., van Gog, Tamara, Zwaan, Rolf A., & Paas, Fred. (2014). Toward a more embedded/extended perspective on the cognitive function of gestures. Frontiers in Psychology, 5, 359, 1–14. DOI logoGoogle Scholar
Rozin, Paul & Royzman, Edward B. (2001). Negativity bias, negativity dominance, and contagion. Personality and social psychology review, 5 (4), 296–320. DOI logoGoogle Scholar
Sekine, Kazuki, Sowden, Hannah, & Kita, Sotaro. (2015). The development of the ability to semantically integrate information in speech and iconic gesture in comprehension. Cognitive Science, 391, 1855–1880. DOI logoGoogle Scholar
Skipper, Jeremy I. (2015). The NOLB model: A model of the natural organization of language and the brain. In Roel M. Willems (Ed.), Cognitive neuroscience of natural language use (pp. 101–134). Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Spranger, Tina, Schatz, Tanja R., & Knopf, Monika. (2008). Does action make you faster? A retrieval-based approach to investigating the origins of the enactment effect. Scandinavian Journal of Psychology, 49 (6), 487–495. DOI logoGoogle Scholar
Straube, Benjamin, Green, Antonia, Weis, Susanne, Chatterjee, Anjan, & Kircher, Tilo. (2009). Memory effects of speech and gesture binding: cortical and hippocampal activation in relation to subsequent memory performance. Journal of Cognitive Neuroscience, 21 (4), 821–836. DOI logoGoogle Scholar
Wang, Bo. (2013). Facial expression influences recognition memory for faces: Robust enhancement effect of fearful expression. Memory, 21 (3), 301–314. DOI logoGoogle Scholar
Wilson, Margaret. (2002). Six views of embodied cognition. Psychonomic Bulletin & Review, 9 (4), 625–636. DOI logoGoogle Scholar
Yeo, Amelia, Ledesma, Iasmine, Nathan, Mitchell J., Alibali, Martha W., & Church, Ruth Breckinridge. (2017). Teachers’ gestures and students’ learning: Sometimes “hands off” is better. Cognitive Research: Principles and Implications, 2 (1), 411, 1–11. DOI logoGoogle Scholar
Cited by (2)

Cited by two other publications

Tian, Lixian, Steven G. McCafferty & Man Zhu
2024. The cognitive-emotional dialectics in second language development: speech and co-speech gestures of Chinese learners of English. International Review of Applied Linguistics in Language Teaching 0:0 DOI logo
Kelly, Spencer D. & Quang‐Anh Ngo Tran
2023. Exploring the Emotional Functions of Co‐Speech Hand Gesture in Language and Communication. Topics in Cognitive Science DOI logo

This list is based on CrossRef data as of 4 july 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

video

video 1

video 2

Video clips