Cover not available
Article published In:
Pragmatics & Cognition
Vol. 31:1 (2024) ► pp.49–96

Four-year-olds’ visuospatial cognitive abilities and their relation to observer‑viewpoint gestures across three communicative tasks

| Paderborn University
| Bielefeld University
| Bielefeld University
| Bielefeld University
| Bielefeld University
| Bielefeld University
| Paderborn University
The gesture-as-simulated-action framework explains the occurrence of iconic gestures. Accordingly, simulated visual imagery gives rise to observer-viewpoint, whereas simulated motor imagery gives rise to character-viewpoint gestures. Because little is known about whether this relationship is either the product of becoming a competent speaker in different communicative tasks or exists from an early age, we investigated 4-year-olds. In the first session, 55 children performed three different communicative tasks. In the second session, we administered a SON-R non-verbal intelligence test to assess children’s cognitive abilities. Results revealed a strong association between the frequency with which children used observer-viewpoints gestures and their scores in visual processing abilities. Because the association between gestures and visuospatial cognitive abilities was found across three communicative tasks, we can propose that it is a general ability rather than a phenomenon of task-specific discourse competence. We discuss these findings in terms of gestural threshold and cognitive offloading.
Keywords: language as situated multimodal system, gesture-as-simulated-action (GSA) framework, viewpoints in gesture, cognitive abilities, mental imagery
Article outline
  • 1.Introduction
    • 1.1Iconic gestures and visual stimuli
    • 1.2Iconic gestures and individual differences in cognitive skills
    • 1.3Viewpoints in iconic gestures
    • 1.4Development of the integration of gesture, language, and event knowledge
    • 1.5The present study
  • 2.Methods
    • 2.1Pilot study
    • 2.2Participants
    • 2.3Procedure
    • 2.4Communicative tasks
      • 2.4.1Explanation
      • 2.4.2Retelling
      • 2.4.3Illustration
    • 2.5Stimuli
      • 2.5.1Explanation
      • 2.5.2Retelling
      • 2.5.3Illustration
    • 2.6Intelligence test
    • 2.7Coding
      • 2.7.1Speech
      • 2.7.2Gesture
      • 2.7.3Speech about event aspects related to gestures
    • 2.8Data analysis
  • 3.Results
    • 3.1Communicative tasks
    • 3.2Gestures and SON-R scores
      • 3.2.1Visual processing
      • 3.2.2Fluid intelligence
    • 3.3Speech and visual processing
    • 3.4Alignment between children’s and caregiver’s communication
  • 4.Discussion
    • 4.1Viewpoints across the communicative tasks
    • 4.2Children’s and caregivers’ interactive behavior
    • 4.3Children’s speech and their use of viewpoints
    • 4.4Children’s speech, viewpoints, and their visual processing scores
    • 4.5Generalizability of the results across tasks
    • 4.6Possible mechanisms underlying the relation of the frequency in viewpoint gestures and children’s scores in visual processing abilities
  • 5.Conclusion
  • Authors’ contribution
  • Acknowledgements
  • Supplementary items
    • A.Further analyses of the task conditions in relation to children’s gestures and IQ
    • B.Script for the game explanation
    • C.Analysis of the event aspects
  • References
Published online: 11 October 2024
DOI logo
https://doi.org/10.1075/pc.00040.roh
References (79)
References
Abramov, Olga, Friederike Kern, Sofia Koutalidis, Ulrich Mertens, Katharina J. Rohlfing & Stefan Kopp. 2021. The relation between cognitive abilities and the distribution of semantic features across speech and gesture in 4-year-olds. Cognitive Science 45(7). e13012. DOI logoGoogle Scholar
Alamillo, Asela Reig, Jean-Marc Colletta & Michèle Guidetti. 2013. Gesture and language in narratives and explanations: The effects of age and communicative activity on late multimodal discourse development. Journal of Child Language 40(3). 511–538. DOI logoGoogle Scholar
Alibali, Martha W. & Sotaro Kita. 2010. Gesture highlights perceptually present information for speakers. Gesture 10(1). 3–28. DOI logoGoogle Scholar
Austin, Elizabeth E. & Naomi Sweller. 2014. Presentation and production: The role of gesture in spatial communication. Journal of Experimental Child Psychology 1221. 92–103. DOI logoGoogle Scholar
Baldwin, Dare A. & Jessica E. Kosie. 2021. How does the mind render streaming experience as events? Topics in Cognitive Science 13(1). 79–105. DOI logoGoogle Scholar
Bergmann, Kirsten & Stefan Kopp. 2012. Gestural alignment in natural dialogue. In Proceedings of the Annual Meeting of the Cognitive Science Society 341. 1326–1331.Google Scholar
Capirci, Olga, Annarita Contaldo, Maria Cristina Caselli & Virginia Volterra. 2005. From action to language through gesture: A longitudinal perspective. Gesture 51. 155–177. DOI logoGoogle Scholar
Cartmill, Erica A., Sian Beilock & Susan Goldin-Meadow. 2012. A word in the hand: Action, gesture and mental representation in humans and non-human primates. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 367(1585). 129–143. DOI logoGoogle Scholar
Cartmill, Erica A., Lilia Rissman, Miriam A. Novack & Susan Goldin-Meadow. 2017. The development of iconicity in children’s co-speech gesture and homesign. Language, Interaction and Acquisition 8(1). 42–68. DOI logoGoogle Scholar
Cattell, Raymond B. 1963. Theory of fluid and crystallized intelligence: A critical experiment. Journal of Educational Psychology 54(1). 1–22. DOI logoGoogle Scholar
Child, Simon, Anna Theakston & Simone Pika. 2015. How do modelled gestures influence preschool children’s spontaneous gesture production? Gesture 14(1). 1–25. DOI logoGoogle Scholar
Chu, Mingyuan & Sotaro Kita. 2011. The nature of gestures’ beneficial role in spatial problem solving. Journal of Experimental Psychology 140(1). 102–116. DOI logoGoogle Scholar
Chu, Mingyuan, Antje Meyer, Lucy Foulkes & Sotaro Kita. 2014. Individual differences in frequency and saliency of speech-accompanying gestures: The role of cognitive abilities and empathy. Journal of Experimental Psychology 143(2). 694–709. DOI logoGoogle Scholar
Church, R. Breckinridge & Susan Goldin-Meadow. 1986. The mismatch between gesture and speech as an index of transitional knowledge. Cognition 231. 43–71. DOI logoGoogle Scholar
Church, R. Breckinridge, Spencer D. Kelly & Kathryn Lynch. 2000. Immediate memory for mismatched speech and representational gesture across development. Journal of Nonverbal Behavior 24(2). 151–174. DOI logoGoogle Scholar
Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1). 37–46. DOI logoGoogle Scholar
Cook, Susan Wagner, Terina Kuangyi Yip & Susan Goldin-Meadow. 2012. Gestures, but not meaningless movements, lighten working memory load when explaining math. Language and Cognitive Processes 27(4). 594–610. DOI logoGoogle Scholar
Crawshaw, Camilla E., Friederike Kern, Ulrich Mertens & Katharina J. Rohlfing. 2020. Children’s narrative elaboration after reading a storybook versus viewing a video. Frontiers in Psychology 111. 569891. DOI logoGoogle Scholar
Debreslioska, Sandra & Marianne Gullberg. 2019. Discourse reference is bimodal: How information status in speech interacts with presence and viewpoint of gestures. Discourse Processes 56(1). 41–60. DOI logoGoogle Scholar
Demir, Özlem Ece, Susan C. Levine & Susan Goldin-Meadow. 2015. A tale of two hands: Children’s early gesture use in narrative production predicts later narrative structure in speech. Journal of Child Language 42(3). 662–681. DOI logoGoogle Scholar
Doskočilová, Hana & Zdeněk Miler. 1998/2013. Der Maulwurf und der Grüne Stern [The Mole and the Green Star]. Leipzig: LeiV.Google Scholar
Ehrlich, Stacy B., Susan C. Levine & Susan Goldin-Meadow. 2006. The importance of gesture in children’s spatial reasoning. Developmental Psychology 42(6). 1259–1268. DOI logoGoogle Scholar
Fischer, Kurt W. & Louise Silvern. 1985. Stages and individual differences in cognitive development. Annual Review of Psychology 36(1). 613–648. DOI logoGoogle Scholar
Flanagan, Dawn P. & Shauna G. Dixon. 2014. The Cattell-Horn-Carroll theory of cognitive abilities. In Cecil R. Reynolds, Kimberly J. Vannest & Elaine Fletcher-Janzen (eds.), Encyclopedia of Special education. New Jersey: Wiley. DOI logoGoogle Scholar
Frederiksen, Anne Therese. 2017. Separating viewpoint from mode of representation in iconic co-speech gestures: Insights from Danish narratives. Language and Cognition 9(4). 677–708. DOI logoGoogle Scholar
Furman, Reyhan, Alin C. Küntay & Asli Özyürek. 2014. Early language-specificity of children’s event encoding in speech and gesture: Evidence from caused motion in Turkish. Language, Cognition and Neuroscience 291. 620–634. DOI logoGoogle Scholar
Goldin-Meadow, Susan, Howard Nusbaum, Spencer D. Kelly & Susan Wagner. 2001. Explaining math: Gesturing lightens the load. Psychological Science 12(6). 516–522. DOI logoGoogle Scholar
Goldin-Meadow, Susan & Martha W. Alibali. 2013. Gesture’s role in speaking, learning, and creating language. Annual Review of Psychology 641. 257–283. DOI logoGoogle Scholar
Goodwyn, Susan W., Linda P. Acredolo & Catherine A. Brown. 2000. Impact of symbolic gesturing on early language development. Journal of Nonverbal Behavior 241. 80–104. DOI logoGoogle Scholar
Holler, Judith & Stephen C. Levinson. 2019. Multimodal language processing in human communication. Trends in Cognitive Sciences 23(8). 639–652. DOI logoGoogle Scholar
Hostetter, Autumn B. 2011. When do gestures communicate? A meta-analysis. Psychological Bulletin 137(2). 297–315. DOI logoGoogle Scholar
Hostetter, Autumn B. & Matha W. Alibali. 2007. Raise your hand if you’re spatial: Relations between verbal and spatial skills and gesture production. Gesture 7(1). 73–95. DOI logoGoogle Scholar
. 2008. Visible embodiment: Gestures as simulated action. Psychonomic Bulletin & Review 15(3). 495–514. DOI logoGoogle Scholar
. 2019. Gesture as simulated action: Revisiting the framework. Psychonomic Bulletin & Review 26(3). 721–752. DOI logoGoogle Scholar
Hostetter, Autumn B. & William D. Hopkins. 2002. The effect of thought structure on the production of lexical movements. Brain and Language 82(1). 22–29. DOI logoGoogle Scholar
Hostetter, Autumn B. & Catherine J. Skirving. 2011. The effect of visual vs. verbal stimuli on gesture production. Journal of Nonverbal Behavior 35(3). 205–223. DOI logoGoogle Scholar
Iverson, Jana M. & Thal, D. J. 1998. Communicative transitions. There’s more to the hand than meets the eye. In Amy M. Wetherby, Steven F. Warren & Joe Reichle (eds.), Transitions in prelinguistic communication, 59–86. Baltimore: Paul H. Brookes.Google Scholar
Kern, Friederike, Ulrich Boden, Anne Nemeth, Sofia Koutalidis, Olga Abramov, Stefan Kopp & Katharina J. Rohlfing. 2024. Preschool children’s discourse competence in different genres and how it relates to iconic gestures. Journal of Child Language 51(3). 656–680. DOI logoGoogle Scholar
Kidd, Evan, Seamus Donnelly & Morten H. Christiansen. 2018. Individual differences in language acquisition and processing. Trends in Cognitive Sciences 22(2). 154–169. DOI logoGoogle Scholar
Kita, Sotaro, Martha W. Alibali & Mingyuan Chu. 2017. How do gestures influence thinking and speaking? The gesture-for-conceptualization hypothesis. Psychological Review 124(3). 245–266. DOI logoGoogle Scholar
Landis, J. Richard & Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 33(1). 159. DOI logoGoogle Scholar
Laros, Jacob Arie & Peter Johannes Tellegen. 2015. Construction and validation of the SON-R 5 1/2–17, the Snijders-Oomen non-verbal intelligence test.
LeBaron, Curtis & Jürgen Streeck. 2000. Gestures, knowledge and the world. In David McNeill (ed.), Review of language and gesture: Window into thought and action, 118–138. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Lohman, David F. 1994. Spatial ability. In Robert J. Sternberg (ed.), Encyclopedia of human intelligence, 1000–1007. London: Macmillan.Google Scholar
Lüke, Carina, Ute Ritterfeld, Angela Grimminger, Katharina J. Rohlfing & Ulf Liszkowski. 2020. Integrated communication system: Gesture and language acquisition in typically developing children and children with LD and DLD. Frontiers in Psychology 111. 498187. DOI logoGoogle Scholar
Mandler, Jean M. 2000. Perceptual and conceptual processes in infancy. Journal of Cognition and Development 1(1). 3–36. DOI logoGoogle Scholar
Marstaller, Lars & Hana Burianová. 2013. Individual differences in the gesture effect on working memory. Psychonomic Bulletin & Review 20(3). 496–500. DOI logoGoogle Scholar
McNeill, David. 1992. Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.Google Scholar
Mickley, Manfred & Gerolf Renner. 2010. Intelligenztheorie für die Praxis: Auswahl, Anwendung und Interpretation deutschsprachiger Testverfahren für Kinder und Jugendliche auf Grundlage der CHCTheorie. Klinische Diagnostik und Evaluation 31. 447–466.Google Scholar
. 2019. Auswahl, Anwendung und Interpretation deutschsprachiger Intelligenztests für Kinder und Jugendliche auf Grundlage der CHC-Theorie: Update, Erweiterung und kritische Bewertung [Selection, use, and interpretation of German intelligence tests for children and adolescents based on CHC-theory: Update, extension, and critical discussion]. Praxis der Kinderpsychologie und Kinderpsychiatrie 68(4). 323–343. DOI logoGoogle Scholar
Mizuguchi, Takashi & Kuniaki Sugai. 2002. Object-related knowledge and the production of gestures with imagined objects by preschool children. Perceptual and Motor Skills 94(1). 71–79. DOI logoGoogle Scholar
Möhring, Wenke, Andrew D. Ribner, Robin Segerer, Melissa E. Libertus, Tobias Kahl, Larissa Maria Troesch & Alexander Grob. 2021. Developmental trajectories of children’s spatial skills: Influencing variables and associations with later mathematical thinking. Learning and Instruction 751. 101515. DOI logoGoogle Scholar
Morsella, Ezequiel & Robert M. Krauss. 2004. The role of gestures in spatial working memory and speech. The American Journal of Psychology 117(3). 411–7424. DOI logoGoogle Scholar
Mumford, Katherine H. & Sotaro Kita. 2014. Children use gesture to interpret novel verb meanings. Child Development 85(3). 1181–1189. DOI logoGoogle Scholar
Murgiano, Margherita, Yasamin Motamedi & Gabriella Vigliocco. 2021. Situating language in the real-world: The role of multimodal iconicity and indexicality. Journal of Cognition 4(1). 38. DOI logoGoogle Scholar
O’Reilly, Anne Watson, Kathleen M. Painter & Marc H. Bornstein. 1997. Relations between language and symbolic gesture development in early childhood. Cognitive Development 121. 185–197. DOI logoGoogle Scholar
Özer, Demet & Tilbe Göksun. 2020. Gesture use and processing: A review on individual differences in cognitive resources. Frontiers in Psychology 111. 573555. DOI logoGoogle Scholar
Özyürek, Asli. 2010. The role of iconic gestures in production and comprehension of language: Evidence from brain and behavior. In Stefan Kopp & Iple Wachsmuth (eds.), Gesture in embodied communication and human-computer interaction. New York: Springer. DOI logoGoogle Scholar
Parrill, Fey. 2010. Viewpoint in speech-gesture integration: Linguistic structure, discourse structure, and event structure. Language and Cognitive Processes 25(5). 650–668. DOI logoGoogle Scholar
Parrill, Fey & Kashmiri Stec. 2018. Seeing first person changes gesture but saying first person does not. Gesture 17(1). 158–175. DOI logoGoogle Scholar
Parrill, Fey, Brittany Lavanty, Austin Bennett, Alayna Klco & Ozlem Ece Demir-Lira. 2018. The relationship between character viewpoint gesture and narrative structure in children. Language and Cognition 10(3). 408–434. DOI logoGoogle Scholar
Perry, Michelle, R. Breckinridge Church & Susan Goldin-Meadow. 1992. Is gesture-speech mismatch a general index of transitional knowledge? Cognitive Development 7(1). 109–122. DOI logoGoogle Scholar
Quasthoff, Uta, Vivian Heller & Miriam Morek. 2017. On the sequential organization and genre-orientation of discourse units in interaction: An analytic framework. Discourse Studies 19(1). 84–110. DOI logoGoogle Scholar
Raven, John C. 1962. Advanced progressive matrices. London: Lewis.Google Scholar
Rettig, Heike. 2017. Mit Beispielen argumentieren. Die Analyse von Beispielverwendung in der verbalen Interaktion. In Iris Meißner & Eva Lia Wyss (eds.), Begründen — Erklären — Argumentieren: Konzepte und Modellierungen in der Angewandten Linguistik, 159–188. Tübingen: Stauffenburg Verlag.Google Scholar
Rohlfing, Katharina J. 2019. Language learning from the use of gestures. In Jessica Horst & Janne von Koss Torkildsen (eds.), International Handbook of Language Acquisition, 213–233. London: Routledge. DOI logoGoogle Scholar
Rohlfing, Katharina J., Ulrich Mertens, Sofia Koutalidis, Olga Abramov, Stefan Kopp & Friederike Kern. 2020. Through the lens of a communicative genre: Differences in use of iconic co-speech gestures by 4-years-old children. Poster presented at the online conference “Many Paths to Language”, Nijmegen.
Sassenberg, Uta, Manja Foth, Isabell Wartenburger & Elke Van Der Meer. 2011. Show your hands — Are you really clever? Reasoning, gesture production, and intelligence. Linguistics 49(1). 841. DOI logoGoogle Scholar
Sassenberg, Uta & Elke Van Der Meer. 2010. Do we really gesture more when it is more difficult? Cognitive Science 34(4). 643–664. DOI logoGoogle Scholar
Sekine, Kazuki, Catherine Wood & Sotaro Kita. 2018. Gestural depiction of motion events in narrative increases symbolic distance with age. Language, Interaction and Acquisition 9(1). 40–68. DOI logoGoogle Scholar
Selting, Margret, Peter Auer, Dagmar Barth-Weingarten, Jörg R. Bergmann, Pia Bergmann, Karin Birkner, Elizabeth Couper-Kuhlen, Arnulf Deppermann, Peter Gilles, Susanne Günthner, Martin Hartung, Friederike Kern, Christine Mertzlufft, Christian Meyer, Miriam Morek, Frank Oberzaucher, Jörg Peters, Uta Quastthoff, Willfried Schütte, Anja Stukenbrock & Susanne Uhmann. 2009. Gesprächsanalytisches Transkriptionssystem 2 (GAT 2). Gesprächsforschung — Online-Zeitschrift zur verbalen Interaktion 101. 353–402. [URL]
Smithson, Lisa & Elena Nicoladis. 2014. Lending a hand to imagery? The impact of visuospatial working memory interference upon iconic gesture production in a narrative task. Journal of Nonverbal Behavior 38(2). 247–258. DOI logoGoogle Scholar
Stec, Kashmiri. 2012. Meaningful shifts. Gesture 12(3). 327–360. DOI logoGoogle Scholar
Streeck, Jürgen. 2008. Depicting by gesture. Gesture 8(3). 280–302. DOI logoGoogle Scholar
Stites, Lauren J. & Şeyda Özçalışkan. 2017. Who did what to whom? Children track story referents first in gesture. Journal of Psycholinguistic Research 46(4). 1019–1032. DOI logoGoogle Scholar
Tellegen, Peter J., Jacob A. Laros, & Franz Petermann. 2007. SON-R 2½-7 Non-verbaler Intelligenztest. Testmanual mit deutscher Normierung und Validierung. Boston: Hogrefe.Google Scholar
Trofatter, Caroline, Carly Kontra, Sian Beilock & Susan Goldin-Meadow. 2015. Gesturing has a larger impact on problem-solving than action, even when action is accompanied by words. Language, Cognition and Neuroscience 30(3). 251–260. DOI logoGoogle Scholar
Vernon, Philip A. 1983. Speed of information processing and general intelligence. Intelligence 7(1). 53–70. DOI logoGoogle Scholar
Wood, David, Jerome S. Bruner & Gail Ross. 1976. The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, and Allied Disciplines 17(2). 89–100. DOI logoGoogle Scholar