Article published in:
Gaze in Human-Robot Communication
Edited by Frank Broz, Hagen Lehmann, Bilge Mutlu and Yukiko Nakano
[Benjamins Current Topics 81] 2015
► pp. 47–70

Interactions between a quiz robot and multiple participants

Focusing on speech, gaze and bodily conduct in Japanese and English speakers

This paper reports on a quiz robot experiment in which we explore similarities and differences in human participant speech, gaze, and bodily conduct in responding to a robot’s speech, gaze, and bodily conduct across two languages. Our experiment involved three-person groups of Japanese and English-speaking participants who stood facing the robot and a projection screen that displayed pictures related to the robot’s questions. The robot was programmed so that its speech was coordinated with its gaze, body position, and gestures in relation to transition relevance places (TRPs), key words, and deictic words and expressions (e.g. this, this picture) in both languages. Contrary to findings on human interaction, we found that the frequency of English speakers’ head nodding was higher than that of Japanese speakers in human-robot interaction (HRI). Our findings suggest that the coordination of the robot’s verbal and non-verbal actions surrounding TRPs, key words, and deictic words and expressions is important for facilitating HRI irrespective of participants’ native language.

Keywords: conversation analysis, coordination of verbal and non-verbal actions, human-robot interaction (HRI), robot gaze comparison between English and Japanese, transition relevance place (TRP)
Published online: 16 December 2015
https://doi.org/10.1075/bct.81.04yam
References
Bavelas, J.B., Coates, L., & Johnson, T
(2002) Listener responses as a collaborative process: The role of gaze. Journal of Communication, 52(3), 566–580. DOI logoGoogle Scholar
CMU cross-cultural receptionist corpus
(2012) Retrieved from https://​github​.com​/maxipesfix​/receptionist​_corpus
Diginfonews
(2010) Museum Guide Robot: Diginfo [Video file]. Retrieved from https://​www​.youtube​.com​/watch​?v​=2cSCwMOxccs
Face API
Retrieved September 6, 2013. from http://​www​.seeingmachines​.com​/product​/faceapi/
Fox, B.A., Hayashi, M., & Jasperson, R
(1996) Resources and repair: A cross-linguistic study of syntax and repair. Studies in Interactional Sociolinguistics, 13, 185–237.Google Scholar
Garfinkel, H
(1967) Studies in ethnomethodology. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
Goodwin, C
(1981) Conversational organization: Interaction between speakers and hearers. Language, thought, and culture. New York: Academic Press.Google Scholar
(2000) Action and embodiment within situated human interaction. Journal of Pragmatics, 32(10), 1489–1522. DOI logoGoogle Scholar
Goodwin, M.H
(1990) He-said-she-said: Talk as social organization among black children (Vol. 618). Bloomington: Indiana University Press.Google Scholar
Hayashi, M
(2010) An overview of the question-response system in Japanese. Journal of Pragmatics, 42(10), 2685–2702. DOI logoGoogle Scholar
Heath, C., & Luff, P
(2000) Technology in action. Cambridge, UK: Cambridge University Press. DOI logoGoogle Scholar
Iwasaki, S
(2009) Initiating interactive turn spaces in Japanese conversation: Local projection and collaborative action. Discourse Processes, 46(2–3), 226–246. DOI logoGoogle Scholar
Jefferson, G
(1984) Transcript notation. In J.M. Atkinson & J. Hertiage (Eds.), Structures of social action. Studies in conversation analysis (pp. ix–xvi). Cambridge, UK: Cambridge University Press.Google Scholar
Kendon, A
(1967) Some functions of gaze-direction in social interaction. Acta psychologica, 26, 22–63. DOI logoGoogle Scholar
Knight, H., & Simmons, R
(2012) Estimating human interest and attention via gaze analysis. In Proceedings of the 12th International Conference on Intelligent Virtual Agents (IVA’12, pp. 245–251).
Kogure, M
(2007) Nodding and smiling in silence during the loop sequence of backchannels in Japanese conversation. Journal of Pragmatics, 39(7), 1275–1289. DOI logoGoogle Scholar
Lee, C., Lesh, N., Sidner, C.L., Morency, L.P., Kapoor, A., & Darrell, T
(2004) Nodding in conversations with a robot. In CHI’04 Extended Abstracts on Human Factors in Computing Systems (pp. 785–786). ACM.
Makatchev, M., Simmons, R., & Sakr, M
(2012) A cross-cultural corpus of annotated verbal and nonverbal behaviors in receptionist encounters. In Gaze in HRI: From Modeling to Communication workshop at 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI) , Boston, USA, March, 2012.
Maynard, S.K
(1990) Conversation management in contrast: Listener response in Japanese and American English. Journal of Pragmatics, 14(3), 397–412. DOI logoGoogle Scholar
Mutlu, B., Shiwa, T., Kanda, T., Ishiguro, H., & Hagita, N
(2009) Footing in human-robot conversations: how robots might shape participant roles using gaze cues. In Proceedings of the 4th ACM/IEEE International Conference on Human-Robot Interaction (pp. 61–68). ACM.
Nakano, Y., & Rehm, M
(2009) Multimodal corpus analysis as a method for ensuring cultural usability of embodied conversational agents. In Human centered design (pp. 521–530). Springer Berlin: Heidelberg. DOI logoGoogle Scholar
Nomura, T., Kanda, T., Suzuki, T., Han, J., Shin, N., Burke, J., & Kato, K
(2007) Implications on humanoid robots in pedagogical applications from cross-cultural analysis between Japan, Korea, and the USA. In Robot and Human Interactive Communication, 2007. RO-MAN 2007. The 16th IEEE International Symposium on (pp. 1052–1057). IEEE. DOI logo
Pitsch, K., Vollmer, A.L., & Muhlig, M
(2013) Robot feedback shapes the tutor’s presentation. How a robot’s online gaze strategies lead to micro-adaptation of the human’s conduct. Interaction Studies, 14(2), 268–296. DOI logoGoogle Scholar
Rossano, F
(2013) Gaze in conversation. In J. Sidnell & T. Stivers (Eds.), The handbook of conversation analysis (pp. 308–329). Malden, MA: Blackwell.Google Scholar
Rossano, F., Brown, P., & Levinson, S.C
(2009) Gaze, questioning and culture. In J. Sidnell (Ed.), Conversation analysis: Comparative perspectives (pp. 187–249). Cambridge, UK: Cambridge University Press. DOI logoGoogle Scholar
Sacks, H., Schegloff, E.A., & Jefferson, G
(1974) A simplest systematics for the organization of turn-taking for conversation. Language, 50(4), Part 1, 696–735. DOI logoGoogle Scholar
Sacks, H
(1987) On the preferences for agreement and contiguity in sequences in conversation. In G. Button & J.R.E. Lee (Eds.), Talk and social organization (pp. 54–69). Clevedon, UK: Multilingual Matters.Google Scholar
Schegloff, E.A
(2007) Sequence organization in interaction: A primer in conversation analysis (Vol. 1). Cambridge University Press. DOI logoGoogle Scholar
Suchman, L
(2006) Human-machine reconfigurations: Plans and situated actions. Cambridge, UK: Cambridge University Press. DOI logoGoogle Scholar
Stivers, T
(2008) Stance, alignment, and affiliation during storytelling: When nodding is a token of affiliation. Research on Language and Social Interaction, 41(1), 31–57. DOI logoGoogle Scholar
(2010) An overview of the question-response system in American English conversation. Journal of Pragmatics, 42(10), 2772–2781. DOI logoGoogle Scholar
Stivers, T., & Rossano, F
(2010) Mobilizing response. Research on Language and Social Interaction, 43(1), 3–31. DOI logoGoogle Scholar
Tanaka, H
(1999) Turn-taking in Japanese conversation: A study in grammar and interaction (Vol. 3). Amsterdam: John Benjamins. DOI logoGoogle Scholar
Traum, D., Aggarwal, P., Artstein, R., Foutz, S., Gerten, J., Katsamanis, A., Leuski, A., Noren, D., & Swartout, W
(2012) Ada and grace: Direct interaction with museum visitors. In Y. Nakano et al. (Eds.), Proceedings of the 12th International Conference on Intelligent Virtual Agents (IVA’12, pp. 245–251). Heidelberg: Springer.Google Scholar
Yamazaki, A., Yamazaki, K., Burdelski, M., Kuno, Y., & Fukushima, M
(2010) Coordination of verbal and non-verbal actions in human-robot interaction at museums and exhibitions. Journal of Pragmatics, 42(9), 2398–2414. DOI logoGoogle Scholar
Yamazaki, A., Yamazaki, K., Kuno, Y., Burdelski, M., Kawashima, M., & Kuzuoka, H
(2008) Precision timing in human-robot interaction: Coordination of head movement and utterance. In Proceedings of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems (CHI’ 08, pp. 131–140). ACM. DOI logo
Yamazaki, A., Yamazaki, K., Ohyama, T., Kobayashi, Y., & Kuno, Y
(2012) A techno-sociological solution for designing a museum guide robot: Regarding choosing an appropriate visitor. In Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction (HRI’ 12, pp. 309–316). ACM.
Yamazaki, K., Yamazaki, A., Okada, M., Kuno, Y., Kobayashi, Y., Hoshi, Y., & Heath, C
(2009) Revealing gauguin: Engaging visitors in robot guide’s explanation in an art museum. In Proceedings of the 27th International Conference on Human Factors in Computing Systems (CHI’ 09, pp. 1437–1446). ACM.