Embodied artificial agents, like humanoid robots or virtual characters, can produce a variety of co-speech gestures in interactive settings. This enables a computational branch of gesture research that offers opportunities for (1) investigating production and comprehension processes through computational cognitive modeling, and (2) studying systematically the effects and functions of gesturing in human-agent interaction. In this chapter we review current approaches to synthesize gestures, and we discuss in detail findings from experiments on the effects of gestures produced by virtual characters or robots. Current evidence suggests that synthetic gesturing has considerable effects on how an artificial agent is perceived and attributed with human-like properties. In contrast, so far, there is little evidence showing that contemporary synthetic gesturing helps human addressees to better understand or retain the information communicated by an agent. We discuss what these findings imply for the potential functions that gesture may serve in improving human-agent interaction, and more generally for our understanding of gesture through computational modeling efforts.
Article outline
Introduction
Computational approaches to synthetic gesture
Realizing gestural movements
Determining content and form of co-speech gestures
The effects of gesture in human-agent interaction studies
Anderson, John R., Bothell, Daniel, Byrne, Michael D., Douglass, Scott, Lebiere, Christian, & Qin, Yulin
2004 “An integrated theory of the mind.” Psychological Review 111 (4): 1036–1060.
Bavelas, Janet B., Chovil, Nicole, Coates, Linda, & Roe, Lori
1995 “Gestures specialized for dialogue.” Personality and Social Psychology Bulletin 21: 394–405.
Bennewitz, Maren, Faber, Felix, Joho, Dominik, & Behnke, Sven
2007 “Fritz – a humanoid communication robot.” In Proceedings of IEEE Int. Symposium on Robot and Human Interactive Communication (RO-MAN 2007), 1072–1077.
Bergmann, Kirsten
2012The Production of Co-Speech Iconic Gestures: Empirical Study and Computational Simulation with Virtual Agents. PhD. Thesis, University of Bielefeld.
Bergmann, Kirsten, & Kopp, Stefan
2006 “Verbal or visual? How information is distributed across speech and gesture in spatial dialog).” In Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue, 90–97.
Bergmann, Kirsten, & Kopp, Stefan
2010 “Modeling the production of coverbal iconic gestures by learning Nayesian decision networks.” Applied Artificial Intelligence 24 (6): 530–551.
Bergmann, Kirsten, & Kopp, Stefan
2009 “Increasing expressiveness of virtual agents – autonomous generation of speech and gesture for spatial description tasks.” In Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2009), K. Decker, J. Sichman, G. Sierra, & C. Castelfranchi (eds), 361–368.
Bergmann, Kirsten, Eyssel, Friederike, & Kopp, Stefan
2012 “A second chance to make a first impression? How appearance and nonverbal behavior affect perceived warmth and competence of virtual agents over time.” In Proceedings of Intelligent Virtual Agents (IVA 2012), 126–138. Berlin, Heidelberg: Springer.
2010 “Individualized gesturing outperforms average gesturing – evaluating gesture production in virtual humans.” In Proceedings of Intelligent Virtual Agents (IVA 2010), 104–117. Berlin, Heidelberg: Springer.
Bickmore, Timothy, & Cassell, Justine
2005 “Social dialogue with embodied conversational agents.” In Advances in Natural Multimodal Dialogue Systems, J. C. J. van Kuppevelt, L. Dybkjær, & N. O. Bernsen, (eds) 23–54. New York: Springer.
Billard, Aude, Calinon, Sylvain, Dillmann, Rüdiger, & Schaal, Stefan
2008 “Robot programming by demonstration.” In Springer Handbook of Robotics, B. Siciliano & O. Khatib (eds) 1371–1394. Berlin, Heidelberg: Springer.
Blascovich, Jim, Loomis, Jack, Beall, Aandrew C., Swinth, Kimberly R., Hoyt, Crystal L., & Bailenson, Jeremy N.
2002 “Immersive virtual environment technology as a methodological tool for social psychology.” Psychological Inquiry 13 (2): 103–124.
Bohus, Dan, & Horvitz, Eric
2010 “Facilitating multiparty dialog with gaze, gesture, and speech.” In Proceedings of ICMI-MLMI'10, 1–8. Beijing, China.
2009 “Conversational gestures in human-robot interaction.” In Proceedings of the 2009 IEEE International Conference on Systems, Man and Cybernetics, 1645–1649.
Breslow, Leonard A., Harrison, Anthony M., & Trafton, J. Gregory
2010 “Linguistic spatial gestures.” In Proceedings of the 10th International Conference on Cognitive Modeling (ICCM), D. D. Salvucci & G. Gunzelmann (eds), 13–18.
Buisine, Stéphanie, & Martin, Jean-Claude
2007 “The effects of speech – gesture cooperation in animated agents’ behavior in multimedia presentations.” Interacting with Computers 19 (4): 484–493.
Cassell, Justine, & Thorisson, Kristinn R.
1999 “The power of a nod and a glance: envelope vs. emotional feedback in animated conversational agents.” Applied Artificial Intelligence: 1–20.
Cassell, Justine, Stone, Matthew, & Yan, Hao
2000 “Coordination and context-dependence in the generation of embodied conversation.” In Proceedings of Int. Conference on Natural Language Generation (INLG 2000), 1–8.
Chiu, Chung-Cheng, & Marsella, Stacy
2011 “How to train your avatar: A data driven approach to gesture generation.” In Proceedings of (IVA 2011), 127–140. Berlin, Heidelberg: Springer.
2011 “The dynamics of warmth and competence judgments, and their outcomes in organizations.” Research in Organizational Behavior 31: 73–98.
Dehn, Doris M., & Van Mulken, Susanne
2000 “The impact of animated interface agents: a review of empirical research.” International Journal of Human-Computer Studies 52 (1): 1–22.
DeSteno, David, Breazeal, Cynthia, Frank, Robert H., Pizarro, David, Baumann, Jolie, Dickens, Leah, & Lee, Jin Joo
2012 “Detecting trustworthiness of novel partners in economic exchange.” Psychological Science 23 (12): 1549–1556.
Gielniak, Michael J., & Thomaz, Andrea L.
2012 “Enhancing interaction through exaggerated motion synthesis.” In Proceedings of the 7th Annual ACM/IEEE International Conference on Human-Robot Interaction (HRI), 375–382. New York: ACM Press.
2006 “Implementing expressive gesture synthesis for embodied conversational agents.” In Gesture in Human-Computer Interaction and Simulation, S. Gibet, N. Courty, & J.-F. Kamp (eds), 188–199, Springer.
2004Gesture: Visible Action as Utterance. Cambridge University Press.
Kita, Sotaro, & Davies, Thomas S.
2009 “Competing conceptual representations trigger co-speech representational gestures.” Language and Cognitive Processes 24 (5): 761–775.
Kita, Sotaro, & Özyürek, Asli
2003 “What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking.” Journal of Memory and Language 48: 16–32.
Kopp, Stefan
2014 “Gestures, postures, gaze and movements in computer science: embodied agents.” In Body-Language-Communication: An International Handbook on Multimodality in Human Interaction, C. Müller, A. Cienki, E. Fricke, S. Ladewig, D. McNeill, & J. Bressem (eds), 1948–1954. Mouton De Gruyter.
Kopp, Stefan, & Wachsmuth, Ipke
2004 “Synthesizing multimodal utterances for conversational agents.” Computer Animation and Virtual Worlds 15 (1): 39–52.
Kopp, Stefan, Bergmann, Kirsten, & Kahl, Sebastian
2013 “A spreading-activation model of the semantic coordination of speech and gesture.” In Proceedings of the 35th Annual Meeting of the Cognitive Science Society (CogSci 2013), 823–828.
Kopp, Stefan
2017 “Using cognitive models to understand multimodal processes the case for speech and gesture production.” In Handbook of Multimodal-Multisensor Interfaces, S. Oviatt, B. Schuller, P. Cohen & A. Krueger (eds). ACM Books, Morgan Claypool.
Krämer, Nicole C., Tietz, Bernd, & Bente, Gary
2003Effects of embodied interface agents and their gestural activity (pp. 292–300). Proceedings of Intelligent Virtual Agents (IVA 2003), Berlin, Heidelberg: Springer.
Lee, Jina, & Marsella, Stacy
2006 “Nonverbal behavior generator for embodied conversational agents.” In Proceedings of Intelligent Virtual Agents (IVA 2006), 243–255. Berlin, Heidelberg: Springer.
2014 “Imitation of a pedagogical agent’s gestures enhances memory for words in second language.” Science Journal of Education 2 (5): 162–169.
Macedonia, Manuela, Müller, Karsten, & Friederici, Angela D.
2010 “The impact of iconic gestures on foreign language word learning and its neural substrate.” Human Brain Mapping 32 (6): 982–998.
Mayer, Richard E., & DaPra, C.Scott
2012 “An embodiment effect in computer-based learning with animated pedagogical agents.” Journal of Experimental Psychology: Applied 18 (3): 239–252.
Moshkina, Lilia, Trickett, Susan, & Trafton, J. Gregory
2014 “Social engagement in public places.” In Proceedings of ACM/IEEE International Conference on Human-Robot Interaction (HRI), 382–389. New York: ACM Press.
2008 “Gesture modeling and animation based on a probabilistic re-creation of speaker style.” ACM Transactions on Graphics 27 (1): 1–24.
Ng-Thow-Hing, Victor, Pengcheng Luo, & Okita, S.
2010 “Synchronized gesture and speech production for humanoid robots.” In Proceedings of the 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, 4617–4624.
2009 “Providing route directions.” In Proceedings of the 4th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 53–60. New York: ACM Press.
Rehm, Matthias, & André, Elisabeth
2007 “More than just a friendly phrase: Multimodal aspects of polite behavior in agents.” In Conversational Informatics, 69–84. Chichester, UK: John Wiley & Sons.
Ruttkay, Zsofia
2007 “Presenting in style by virtual humans.” In Proceedings of the COST Action International Conference on Verbal and Nonverbal Communication Behaviours, 23–36.
Salem, Maha, Eyssel, Friederike, Rohlfing, Katharina, Kopp, Stefan, & Joublin, Frank
2013 “To err is human(-like): Effects of robot gesture on perceived anthropomorphism and likability.” International Journal of Social Robotics 5 (3): 313–323.
Salem, Maha, Kopp, Stefan, Wachsmuth, Ipke, Rohlfing, Katharina, & Joublin, Frank
2012 “Generation and evaluation of communicative robot gesture.” International Journal of Social Robotics 4 (2): 201–217.
Sidner, Candace L. & Lee, Christopher
2003 “Engagement rules for human-robot collaborative interactions.” In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics (ICSMC), 3957–3962.
2004 “Speaking with hands: Creating animated conversational characters from recordings of human performance.” ACM Transactions on Graphics 23 (3): 506–513.
Winograd, Terry
1973 “A Process model of language understanding.” In Computer Models of Thought and Language, R. C. Schank & K. M. Colby (eds), 152–186. San Francisco.
Cited by
Cited by 4 other publications
Kelly, Spencer D. & Quang‐Anh Ngo Tran
2023. Exploring the Emotional Functions of Co‐Speech Hand Gesture in Language and Communication. Topics in Cognitive Science
Nirme, Jens, Magnus Haake, Agneta Gulz & Marianne Gullberg
2020. Motion capture-based animated characters for the study of speech–gesture integration. Behavior Research Methods 52:3 ► pp. 1339 ff.
Romero Andonegi, Asier, Irati de Pablo Delgado, Aintzane Etxebarria Lejarreta & Ainara Romero Andonegi
This list is based on CrossRef data as of 12 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.