The effects of culture and context on perceptions of robotic facial expressions
Casey C. Bennett | School of Informatics and Computing, Indiana University | Department of Informatics, Centerstone Research Institute | [email protected]
We report two experimental studies of human perceptions of robotic facial expressions while systematically varying context effects and the cultural background of subjects (n = 93). Except for Fear, East Asian and Western subjects were not significantly different in recognition rates, and, while Westerners were better at judging affect from mouth movement alone, East Asians were not any better at judging affect based on eye/brow movement alone. Moreover, context effects appeared capable of over-riding such cultural differences, most notably for Fear. The results seem to run counter to previous theories of cultural differences in facial expression based on emoticons and eye fixation patterns. We connect this to broader research in cognitive science – suggesting the findings support a dynamical systems view of social cognition as an emergent phenomenon. The results here suggest that, if we can induce appropriate context effects, it may be possible to create culture-neutral models of robots and affective interaction.
Adams, B., Breazeal, C., Brooks, R.A., & Scassellati, B. (2000). Humanoid robots: A new kind of tool. IEEE Intelligent Systems and their Applications, 15(4), 25–31.
Arizpe, J., Kravitz, D.J., Yovel, G., & Baker, C.I. (2012). Start position strongly influences fixation patterns during face processing: Difficulties with eye movements as a measure of information use. PLoS ONE, 7(2), e31106.
Asada, M., Hosoda, K., Kuniyoshi, Y., Ishiguro, H., Inui, T., Yoshikawa, Y., et al. (2009). Cognitive developmental robotics: A survey. IEEE Transactions on Autonomous Mental Development, 1(1), 12–34.
Auvray, M., Lenay, C., & Stewart, J. (2009). Perceptual interactions in a minimalist virtual environment. New Ideas in Psychology, 27(1), 32–47.
Barrett, L.F., Mesquita, B., & Gendron, M. (2011). Context in emotion perception. Current Directions in Psychological Science, 20(5), 286–290.
Barsalou, L.W., Breazeal, C., & Smith L.B. (2007). Cognition as coordinated non-cognition. Cognitive Processing, 8(2), 79–91.
Bartneck, C. & Okada, M. (2001). Robotic user interfaces.
Proceedings of the Human and Computer Conference (HC2001)
, 130–140.
Bartneck, C., Suzuki, T., Kanda, T., & Nomura, T. (2007). The influence of people’s culture and prior experiences with Aibo on their attitude towards robots. AI & Society, 21(1–2), 217–30.
Bartneck, D., Kulic, E., Croft, M., & Zoghbi, S. (2009). Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. International Journal of Social Robotics, 1(1), 71–81.
Bazo, D., Vaidyanathan, R., Lentz, A., & Melhuish, C. (2010). Design and testing of a hybrid expressive face for a humanoid robot.
Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
, 5317–5322.
Becker-Asano, C. & Ishiguro, H. (2011). Evaluating facial displays of emotion for the android robot Geminoid F.
Proceedings of the IEEE Workshop on Affective Computational Intelligence (WACI)
, 1–8.
Beer, R.D. (1995). A dynamical systems perspective on agent-environment interaction. Artificial Intelligence, 72(1–2), 173–215.
Beer, R.D. (2000). Dynamical approaches to cognitive science. Trends in Cognitive Sciences, 4(3), 91–99.
Bennett, C.C., & Šabanović, S. (2013). Perceptions of affective expression in a minimalist robotic face.
Proceedings of the ACM/IEEE Conference on Human-Robot Interaction (HRI)
, 81–82.
Bennett, C.C., & Šabanović, S. (2014). Deriving minimal features for human-like facial expressions in robotic faces. International Journal of Social Robotics, 6(3), 367–381.
Bennett, C.C., Šabanović, S., Fraune, M., & Shaw, K. (2014). Context congruency and robotic facial expressions: Do effects on human perceptions vary across culture?Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN)
, 465–470.
Biehl, M., Matsumoto, D., Ekman, P., Hearn, V., Heider, K., Kudoh, T., & Ton, V. (1997). Matsumoto and Ekman’s Japanese and Caucasian Facial Expressions of Emotion (JACFEE): Reliability data and cross-national differences. Journal of Nonverbal Behavior, 21(1), 3–21.
Blais, C., Roy, C., Fiset, D., Arguin, M., & Gosselin, F. (2012). The eyes are not the window to basic emotions. Neuropsychologia, 50(12), 2830–2838.
Breazeal, C. (2003). Emotion and sociable humanoid robots. International Journal of Human-Computer Studies, 59(1–2), 119–155.
Boiger, M., & Mesquita, B. (2012). The construction of emotion in interactions, relationships, and cultures. Emotion Review, 4(3), 221–229.
Bosse, T., Pontier, M., & Treur, J. (2010). A computational model based on Gross’ emotion regulation theory. Cognitive Systems Research, 11(3), 211–230.
Canamero, L. (2005). Emotion understanding from the perspective of autonomous robots research. Neural Networks, 18(4), 445–455.
Canamero, L., & Fredslund, J. (2001). I show you how I like you – can you read it in my face?IEEE Transactions on Systems Man and Cybernetics A: Systems and Humans, 31(5), 454–459.
Caporael, L.R. (1997). The evolution of truly social cognition: The core configuration model. Personality and Social Psychology Review, 1(4), 276–298.
Clark, A. (2013). Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 36(3), 181–204.
Cohn, J.F. (2010). Advances in behavioral science using automated facial image analysis and synthesis. IEEE Signal Process Magazine, 27(6), 128–133.
Darwin, C. (1872). The expression of the emotions in man and animals. London, UK: John Murray.
Davies, I.R., Sowden, P.T., Jerrett, D.T., Jerret,t T., & Corbett, G.G. (1998). A cross-cultural study of English and Setswana speakers on a colour triads task : A test of the Sapir-Whorf hypothesis. British Journal of Psychology, 89(1), 1–15.
De Gelder, B. (2009). Why bodies? Twelve reasons for including bodily expressions in affective neuroscience. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 364(1535), 3475–3484.
De Jaegher, H., Di Paolo, E., & Gallagher, S. (2010). Can social interaction constitute social cognition?Trends in Cognitive Sciences, 14(10), 441–447.
Ekman, P. (1971). Universals and cultural differences in facial expressions of emotion. Nebraska Symposium on Motivation, 191, 207–283.
Ekman, P. (2009). Darwin’s contributions to our understanding of emotional expressions. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 364(1535), 3449–3451.
Ekman, P., & Friesen, W.V. (2003). Unmasking the face: A guide to recognizing emotions from facial clues. Los Altos, CA, USA: Malor Books.
Elfenbein, H.A. (2013). Nonverbal dialects and accents in facial expressions of emotion. Emotion Review, 5(1), 90–96.
Embgen, S., Luber, M., Becker-Asano, C., Ragni, M., Evers, V., & Arras, K.O. (2012). Robot-specific social cues in emotional body language.
Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN)
, 1019–1025.
Friesen, W.V. (1973). Cultural differences in facial expressions in a social situation: An experimental test on the concept of display rules. Dissertation Abstracts International, 33(8-B), 3976–3977.
Froese, T., & Ziemke, T. (2009). Enactive artificial intelligence: Investigating the systemic organization of life and mind. Artificial Intelligence, 173(3–4), 466–500.
Geraci, R.M. (2006). Spiritual robots: Religion and our scientific view of the natural world. Theology and Science, 4(3), 229–246.
Gibson, J.J. (1979). The ecological approach to visual perception. Boston, MA, USA: Houghlin Mifflin.
Gross, J.J. & Levenson, R.W. (1995). Emotion elicitation using films. Cognition and Emotion, 9(1), 87–108.
Hall, E.T. (1977). Beyond culture. New York, NY, USA: Anchor Books.
Ikegami, T. & Suzuki, K. (2008). From a homeostatic to a homeodynamic self. BioSystems, 91(2), 388–400.
Jack, R.E., Blais, C., Scheepers, C., Schyns, P.G., & Caldara, R. (2009). Cultural confusions show that facial expressions are not universal. Current Biology, 19(18), 1543–1548.
Jack, R.E., Garrod, O.G., Yu, H., Caldara, R., & Schyns, P.G. (2012). Facial expressions of emotion are not culturally universal.
Proceeedings of the National Acadamy of Sciences USA
, 109(19), 7241–7244.
Kaplan, F. (2004). Who is afraid of the humanoid? Investigating cultural differences in the acceptance of robots. International Journal of Humanoid Robotics, 1(3), 465–480.
Kleinsmith, A., De Silva, P.R., & Bianchi-Berthouze, N. (2006). Cross-cultural differences in recognizing affect from body posture. Interacting with Computers, 18(6), 1371–89.
Kobayashi, H., Ichikawa, Y., Senda, M., & Shiiba, T. (2003). Realization of realistic and rich facial expressions by face robot.
Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
, 1123–1128.
Koda, T., Ruttkay, Z., Nakagawa, Y., & Tabuchi, K. (2010). Cross-cultural study on facial regions as cues to recognize emotions of virtual agents. In T. Ishida (Ed.), Lecture Notes in Computer Science (Culture and Computing) (pp. 16–27). Springer, Berlin.
Kozima, H., Michalowski, M.P., & Nakagawa, C. (2009). Keepon: A playful robot for research, therapy, and entertainment. International Journal of Social Robotics, 1(1), 3–18.
Krumhuber, E.G., Kappas, A., & Manstead, A.S.R. (2013). Effects of dynamic aspects of facial expressions: A review. Emotion Review, 5(1), 41–46.
Lee, H.R., & Sabanović, S. (2014). Culturally variable preferences for robot design and use in South Korea, Turkey, and the United States.
Proceedings of the ACM/IEEE Conference on Human-Robot Interaction (HRI)
, 17–24.
Lindquist, K.A. & Gendron, M. (2013). What’s in a word? Language constructs emotion perception. Emotion Review, 5(1), 66–71.
MacDorman, K.F., Vasudevan, S.K., & Ho, C.C. (2009). Does Japan really have robot mania? Comparing attitudes by implicit and explicit measures. AI & Society, 23(4), 485–510.
Matsumoto, D. (1992). American-Japanese cultural differences in the recognition of universal facial expressions. Journal of Cross-Cultural Psychology, 23(1), 72–84.
Matsumoto, N., Fujii, H., & Okada, M. (2006). Minimal design for human-agent communication. Artificial Life and Robotics, 10(1), 49–54.
McGann. M., De Jaegher, H., & Di Paolo, E. (2013). Enaction and psychology. Review of General Psychology, 17(2), 203–209.
Miwa, H., Itoh, K., Matsumoto, M., Zecca, M., Takanobu, H., Rocella, S., et al. (2004). Effective emotional expressions with expression humanoid robot we-4rii: Integration of humanoid robot hand rch-1.
Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
, 2203–2208.
Mori, M. (1970). Bukimi no tani[The uncanny valley]. Energy, 7(4), 33–35. [URL]
Nisbett, R.E., Peng, K., Choi, I., & Norenzayan, A. (2001). Culture and systems of thought: Holistic versus analytic cognition. Psychological Review, 108(2), 291–310.
Nisbett, R.E. (2003). The geography of thought: How Asians and Westerners think differently. New York, NY, USA: Free Press.
Nomura, T., & Kanda, T. (2003). On proposing the concept of robot anxiety and considering measurement of it.
Proceedings of 12th IEEE International Symposium on Robot and Human interactive Communication (RO-MAN)
, 373–378.
Pantic, M. (2009). Machine analysis of facial behaviour: Naturalistic and dynamic behaviour. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 364(1535), 3505–3513.
Perlovsky, L. (2009). Language and emotions: Emotional Sapir-Whorf hypothesis. Neural Networks, 22(5–6), 518–526.
Peterson, M.F., & Eckstein, M.P. (2012). Looking just below the eyes is optimal across face recognition tasks.
Proceeedings of the National Acadamy of Sciences USA
, 109(48), E3314–3323.
Picard, R.W. (1997). Affective computing. Boston, MA, USA: MIT Press.
Righart, R., & de Gelder, B. (2008). Recognition of facial expressions is influenced by emotional scene gist. Cognitive Affective & Behavavioral Neuroscience, 8(3), 264–272.
Russell, J.A., & Fernández-Dols, J.M. (1997). The psychology of facial expression. Cambridge, UK: Cambridge University Press.
Ruttkay, Z. (2009). Cultural dialects of real and synthetic emotional facial expressions. AI & Society, 24(3), 307–315.
Šabanović, S., Michalowski, M.P., & Simmons R. (2006). Robots in the wild: Observing human-robot social interaction outside the lab.
9th IEEE International Workshop on Advanced Motion Control
, 596–601.
Šabanović, S. (2010). Emotion in robot cultures: Cultural models of affect in social robot design.
Proceedings of the Conference on Design & Emotion (D&E2010)
, Chicago, IL.
Šabanović, S. (2014). Inventing Japan’s “robotics culture”: The repeated assembly of science, technology, and culture in social robotics. Social Studies of Science, 44(3), 342–367. Available online: [URL].
Saldien, J., Goris, K., Vanderborght, B., Vanderfaeillie, J., & Lefeber, D. 2010. Expressing emotions with the social robot Probo. International Journal of Social Robotics, 2(4), 377–389.
Scassellati, B. (2006). How developmental psychology and robotics complement each other. Technical report, Massachusetts Institute of Technology, CSAIL.
Shore, B. (1996). Culture in mind: Cognition, culture, and the problem of meaning. Oxford, UK: Oxford University Press.
Sosnowski, S., Bittermann, A., Kuhnlenz, K., & Buss, M. (2006). Design and evaluation of emotion-display EDDIE.
Proceedings of the 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
, 3113–3118.
Sterelny, K. (2007). Social intelligence, human intelligence and niche construction. Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1480), 719–730.
Yuki, M., Maddux, W.W., & Masuda, T. (2007). Are the windows to the soul the same in the East and West? Cultural differences in using the eyes and mouth as cues to recognize emotions in Japan and the United States. Journal of Experimental Social Psychology, 43(2), 303–311.
Zhang, J., & Sharkey, A.J. (2011). Contextual recognition of robot emotions. In R. Großet al. (Ed.), Towards autonomous robotic systems (pp. 78–89). Berlin, Germany: Springer.
Cited by (7)
Cited by seven other publications
Li, Mingming, Fu Guo, Chen Fang & Fengxiang Li
2023. Multisensory integration effect of humanoid robot appearance and voice on users’ affective preference and visual attention. Behaviour & Information Technology 42:14 ► pp. 2387 ff.
Penčić, Marko, Maja Čavić, Dragana Oros, Petar Vrgović, Kalman Babković, Marko Orošnjak & Dijana Čavić
2022. Anthropomorphic Robotic Eyes: Structural Design and Non-Verbal Communication Effectiveness. Sensors 22:8 ► pp. 3060 ff.
Robinson, Fraser & Goldie Nejat
2022. An analysis of design recommendations for socially assistive robot helpers for effective human-robot interactions in senior care. Journal of Rehabilitation and Assistive Technologies Engineering 9
Wullenkord, Ricarda, Jacqueline Bellon, Bruno Gransche, Sebastian Nähr-Wagener & Friederike Eyssel
2022. Social appropriateness in HMI. Interaction Studies. Social Behaviour and Communication in Biological and Artificial Systems 23:3 ► pp. 360 ff.
Ahn, Yunjeong & Jieun Lee
2021. The Role of Anthropomorphic Messengers in Sustainable Participatory Corporate Social Responsibility: Focusing on Messenger’s Facial Expression and Participation Effort. Sustainability 13:8 ► pp. 4365 ff.
Bennett, Casey
2021. Emergent Robotic Personality Traits via Agent-Based Simulation of Abstract Social Environments. Information 12:3 ► pp. 103 ff.
Chesher, Chris & Fiona Andreallo
2021. Robotic Faciality: The Philosophy, Science and Art of Robot Faces. International Journal of Social Robotics 13:1 ► pp. 83 ff.
This list is based on CrossRef data as of 8 january 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.