Cover not available
Part of
Multifaceted Multilingualism
Edited by Kleanthes K. Grohmann
[Studies in Bilingualism 66] 2024
► pp. 232–253

Chapter 9
Multimodal story-retelling

Influences of cognitive load on co‑speech and co‑thought gestures for conceptualization

| National Taipei University of Technology
| National Taipei University of Technology

Cognitive load has been shown to influence speakers’ gesture production, but how it effects the duration, form, and functions of gestures is not yet well understood. This study examines how increased cognitive load influenced the production of co‑speech and co‑thought gestures by second language (L2) learners, and further looks at how different types of gestures correspond to specific functions. Twenty-four English as an L2 participants were recruited and required to watch a pre-determined cartoon and then retell the story in English. Participants were divided into two cognitive load conditions for the story retelling: one group was given a task structure for story retelling, while the other was not. The results of this study show that enhancing cognitive load significantly increased the frequency of both co‑speech and co‑thought gestures and the duration of co‑speech gestures, but not co‑thought gestures. With regard to the correspondence of gesture types and their functions for conceptualization (i.e., activation, manipulation, packaging, exploration), this study found co‑speech and co‑thought gestures serve different functions in L2 narrative tasks. Activation is dominant in every type of co‑speech gesture, especially in iconic and metaphoric gestures. Deictic gestures mostly function for packaging and exploration and co‑thought gestures for exploration and manipulation. This study provides necessary clarification on the relationship between cognitive load and gestures, including novel interactions of gesture duration and functions of gestures for conceptualization.

Article outline
  • 1.Introduction
    • 1.1Types of gestures
    • 1.2Cognitive load and gestures
    • 1.3Information Packaging Hypothesis
    • 1.4Gesture-for-Conceptualization Hypothesis
  • 2.Methodology
    • 2.1Participants and stimuli
    • 2.2Tasks
    • 2.3Gesture coding
  • 3.Results and discussion
    • 3.1Cognitive load and frequency of gestures
    • 3.2Cognitive load and duration of gestures
    • 3.3Functions of co‑speech and co‑thought gestures
  • 4.Conclusion
  • References
  • Appendix
Published online: 4 April 2024
DOI logo
https://doi.org/10.1075/sibil.66.09yan
References
Alibali, M. W., Spencer, R. C., Knox, L., & Kita, S.
(2011) Spontaneous gestures influence strategy choices in problem solving. Psychological Science, 22(9), 1138–1144. DOI logoGoogle Scholar
Alibali, M. W., Kita, S., & Young, A. J.
(2000) Gesture and the process of speech production: We think, therefore we gesture. Language and Cognitive Processes, 15(6), 593–613. DOI logoGoogle Scholar
Anthony, L., Carrington, P., Chu, P., Kidd, C., Lai, J., & Sears, A.
(2011) Gesture dynamics: Features sensitive to task difficulty and correlated with physiological sensors. Stress, 1418(360), 312–316.Google Scholar
Bava Harji, M., & Gheitanchian, M.
(2017) Effects of multimedia task-based teaching and learning approach on EFL learners’ accuracy, fluency and complexity of oral production. Turkish Online Journal of Educational Technology-TOJET, 16(2), 25–34.Google Scholar
Butcher, C., & Goldin-Meadow, S.
(2000) Gesture and the transition from one- to two-word speech: When hand and mouth come together. In D. McNeill (Ed.), Language and gesture (pp. 235–257). Cambridge University Press. DOI logoGoogle Scholar
Chen, F., Ruiz, N., Choi, E., Epps, J., Khawaja, M. A., Taib, R., & Wang, Y.
(2012) Multimodal behavior and interaction as indicators of cognitive load. ACM Transactions on Interactive Intelligent Systems (TiiS), 2(4), 22. DOI logoGoogle Scholar
Chu, M., & Kita, S.
(2011) The nature of gestures’ beneficial role in spatial problem solving. Journal of Experimental Psychology: General, 140(1), 102. DOI logoGoogle Scholar
(2016) co‑thought and co‑speech gestures are generated by the same action generation process. Journal of Experimental Psychology: Learning, Memory, and Cognition, 42(2), 257.Google Scholar
Cook, S. W., Yip, T. K., & Goldin-Meadow, S.
(2012) Gestures, but not meaningless movements, lighten working memory load when explaining math. Language and Cognitive Processes, 27(4), 594–610. DOI logoGoogle Scholar
Cornoldi, C., & Vecchi, T.
(2003) Visuo-spatial working memory and individual differences. Psychology Press.Google Scholar
Eielts, C., Pouw, W., Ouwehand, K., Van Gog, T., Zwaan, R. A., & Paas, F.
(2018) co‑thought gesturing supports more complex problem solving in subjects with lower visual working-memory capacity. Psychological Research, 84, 502–513. DOI logoGoogle Scholar
Goldin-Meadow, S., Nusbaum, H., Kelly, S. D., & Wagner, S.
(2001) Explaining math: Gesturing lightens the load. Psychological Science, 12(6), 516–522. DOI logoGoogle Scholar
Hadar, U., Dar, R., & Teitelman, A.
(2001) Gesture during speech in first and second language: Implications for lexical retrieval. Gesture, 1(2), 151–165. DOI logoGoogle Scholar
Hostetter, A. B.
(2011) When do gestures communicate? A meta-analysis. Psychological Bulletin, 137(2), 297. DOI logoGoogle Scholar
Hostetter, A. B., Alibali, M. W., & Kita, S.
(2007) I see it in my hands’ eye: Representational gestures reflect conceptual demands. Language and Cognitive Processes, 22(3), 313–336. DOI logoGoogle Scholar
Hostetter, A. B., & Alibali, M. W.
(2019) Gesture as simulated action: Revisiting the framework. Psychonomic Bulletin & Review, 26(3), 721–752. DOI logoGoogle Scholar
Hussain, M.
(2014) Cross linguistic variation in the gestures accompanying manner of motion event descriptions by native speakers of English and Urdu. Balochistan Journal of Linguistics, 2, 139–167.Google Scholar
Kendon, A.
(2004) Gesture: Visible action as utterance. Cambridge University Press. DOI logoGoogle Scholar
Khatib, M., & Farahanynia, M.
(2020) Planning conditions (strategic planning, task repetition, and joint planning), cognitive task complexity, and task type: Effects on L2 oral performance. System, 93, 102297. DOI logoGoogle Scholar
Kita, S.
(2000) How representational gestures help speaking. In D. McNeil (Ed.), Language and gesture (pp. 162–185). Cambridge University Press. DOI logoGoogle Scholar
Kita, S., & Özyürek, A.
(2003) What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48(1), 16–32. DOI logoGoogle Scholar
Kita, S., Alibali, M. W., & Chu, M.
(2017) How do gestures influence thinking and speaking? The gesture-for-conceptualization hypothesis. Psychological Review, 124(3), 245. DOI logoGoogle Scholar
Krauss, R. M., Chen, Y., & Gottesman, R. F.
(2000) Lexical gestures and lexical access: A process model. In D. McNeill (Ed.), Language and gesture (pp. 261–283). Cambridge University Press. DOI logoGoogle Scholar
Lee, H. H.
(2017) Linguistic-gestural representations in motion event narrations: Mandarin Chinese as L1 and English as L2 (Unpublished Master’s thesis). National Taipei University of Technology.Google Scholar
Lee, H., Hampel, R., & Kukulska-Hulme, A.
(2019) Gesture in speaking tasks beyond the classroom: An exploration of the multimodal negotiation of meaning via Skype videoconferencing on mobile devices. System, 81, 26–38. DOI logoGoogle Scholar
Lin, Y. L.
(2017) Co-occurrence of speech and gestures: A multimodal corpus linguistic approach to intercultural interaction. Journal of Pragmatics, 117, 155–167. DOI logoGoogle Scholar
(2022) Speech-accompanying gestures in L1 and L2 conversational interaction by speakers of different proficiency levels. International Review of Applied Linguistics in Language Teaching, 60(2), 123–142.Google Scholar
(2020) A helping hand for thinking and speaking: Effects of gestures and task planning on second language narrative discourse, System, 91: 102243. DOI logoGoogle Scholar
(2021) Gestures as scaffolding for L2 narrative recall: The role of gesture type, task complexity, and working memory. Language Teaching Research. DOI logoGoogle Scholar
Logan, T., Lowrie, T., & Diezmann, C. M.
(2014) co‑thought gestures: Supporting students to successfully navigate map tasks. Educational Studies in Mathematics, 87(1), 87–102. DOI logoGoogle Scholar
Lopez-Ozieblo, R.
(2022) Cut-offs and co-occurring gestures: Similarities between speakers’ first and second languages. International Review of Applied Linguistics in Language Teaching. DOI logoGoogle Scholar
McNeill, D.
(1992) Hand and mind: What gestures reveal about thought. The University of Chicago Press.Google Scholar
(2005) Gesture and thought. The University of Chicago Press. DOI logoGoogle Scholar
(2016) Why we gesture: The surprising role of hand movement in communication. Cambridge University Press.Google Scholar
McNeill, D., & Duncan, S.
(2000) Growth points in thinking-for-speaking. In D. McNeill (Ed.). Language and gesture (pp. 141–161). Cambridge University Press.Google Scholar
Mehrang, F., & Rahimpour, M.
(2010) The impact of task structure and planning conditions on oral performance of EFL learners. Procedia-Social and Behavioral Sciences, 2(2), 3678–3686. DOI logoGoogle Scholar
Melinger, A., & Kita, S.
(2007) Conceptualisation load triggers gesture production. Language and Cognitive Processes, 22(4), 473–500. DOI logoGoogle Scholar
Ping, R., & Goldin-Meadow, S.
(2010) Gesturing saves cognitive resources when talking about nonpresent objects. Cognitive Science, 34(4), 602–619. DOI logoGoogle Scholar
Nakatsukasa, K.
(2021) Gesture-enhanced recasts have limited effects: A case of the regular past tense. Language Teaching Research, 25, 587–612. DOI logoGoogle Scholar
Nicoladis, E., Pika, S., Yin, H. U. I., & Marentette, P.
(2007) Gesture use in story recall by Chinese – English bilinguals. Applied Psycholinguistics, 28(4), 721–735. DOI logoGoogle Scholar
Robinson, P.
(2001) Task complexity, task difficulty, and task production: Exploring interactions in a componential framework. Applied Linguistics, 22(1), 27–57. DOI logoGoogle Scholar
(2005) Cognitive complexity and task sequencing: A review of studies in a componential framework for second language task design. International Review of Applied Linguistics in Language Teaching, 43(1), 1–33. DOI logoGoogle Scholar
(2007) Criteria for classifying and sequencing pedagogic tasks. In M. d. P. Garcia Mayo (Ed.), Investigating tasks in formal language learning (pp. 7–27). Multilingual Matters.Google Scholar
Sassenberg, U., & Van Der Meer, E.
(2010) Do we really gesture more when it is more difficult? Cognitive Science, 34(4), 643–664. DOI logoGoogle Scholar
Skehan, P.
(1998) A cognitive approach to language learning. Oxford University Press.Google Scholar
Skehan, P., & Foster, P.
(1999) The influence of task structure and processing conditions on narrative retellings. Language Learning, 49(1), 93–120. DOI logoGoogle Scholar
Smithson, L., & Nicoladis, E.
(2014) Lending a hand to imagery? The impact of visuospatial working memory interference upon iconic gesture production in a narrative task. Journal of Nonverbal Behavior, 38(2), 247–258. DOI logoGoogle Scholar
Talmy, L.
(2000) Toward a cognitive semantics (Vol. 2.). The MIT Press.Google Scholar
Wagner, S. M., Nusbaum, H., & Goldin-Meadow, S.
(2004) Probing the mental representation of gesture: Is handwaving spatial? Journal of Memory and Language, 50(4), 395–407. DOI logoGoogle Scholar