Cover not available
Article published In:
Interaction Studies
Vol. 21:2 (2020) ► pp.268–292

“Talk to you later”

Doing social robotics with conversation analysis. Towards the development of an automatic system for the prediction of disengagement

| Télécom Paris | Institut polytechnique de Paris | I3 CNRS
| Télécom Paris | Institut polytechnique de Paris | Université Paris-Saclay

This article presents an applied discussion of the possibility of integrating conversation analysis (CA) methodology into that of machine learning. The aim is to improve the detection of that which resembles disengagement in the interaction between a robot and a human. We offer a novel analytical assemblage at the heart of the two disciplines, and namely on the level of the annotation schemes provided by conversation analysis transcription methods. First, we demonstrate that the need for a stable structure in establishing an interaction scenario and in designing robot behaviours does not prevent the emergence of ordinariness or creativity among the participants engaged in this interaction. Secondly, based on an actual case, we emphasize the possibility of systematicness in CA transcription to support the choice (a) of the categories targeted by prediction methods and defined by the annotation scheme, and (b) of the verbal and non-verbal features used to create prediction models.

Keywords: social robotics, engagement, machine learning, multimodal features, annotation schemes, conversation analysis, transcription, closings
Article outline
  • 1.Interaction scenarios in social robotics and the notion of context in CA
    • a.Social robotics scenarios and the CA perspective
    • b.Emergence, spontaneity, stability
  • 2.Affective computing annotation of recordings in social robotics vs. CA transcription
    • a.Categories vs. emergence principle: Methodological comparison
    • b.Categories vs. emergence principles: Comparison of two types of production
    • c.Segmentation
  • 3.Conclusion and prospects
  • Acknowledgements
  • Notes
  • References
Published online: 20 May 2020
DOI logo
https://doi.org/10.1075/is.19001.roll
References
Bartneck, C., Belpaeme, T., Eyssel, F., Kanda, T., Keijsers, M., & Šabanović, S.
(2019) Human-robot interaction: An introduction. Cambridge: Cambridge University Press.Google Scholar
Beach, W. A.
(1993) Transitional regularities for ‘casual’ “Okay” usages. Journal of Pragmatics, 191, 325–352. DOI logoGoogle Scholar
Ben-Youssef, A., Clavel, C., Essid, S., Bilac, M., Chamoux, M., & Lim, A.
(2017) UE- HRI: a new dataset for the study of user engagement in spontaneous human- robot interactions. Paper presented at the 19th ACM International Conference on Multimodal Interaction, Glasgow, UK.
Breazeal, C.
(2003) Toward sociable robots. Robotics and Autonomous Systems, 421, 167–175. DOI logoGoogle Scholar
Button, G.
(1991) Conversation-in-a-series. In D. Boden & D. H. Zimmerman (Eds.), Talk and social structure. Studies in Ethnomethodology and Conversation Analysis (pp. 251–277). Cambridge: Polity Press.Google Scholar
Campano, S., Clavel, C., & Pélachaud, C.
(2015) “I like this painting too”: when an ECA shares appreciations to engage users. Paper presented at the 14th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’15), Istanbul, Turkey.
Cassell, J., Torres, O., & Prevost, S.
(1999) Turn taking vs. discourse structure: How best to model multimodal conversation. In Y. Wilks (Ed.), Machine Conversations (pp. 143–154). The Hague: Kluwer. DOI logoGoogle Scholar
Chapman, D.
(1992) Computer rules, conversational rules. Computational Linguistics, 18(4), 531–536.Google Scholar
Clavel, C., Vasilescu, I., & Devillers, L.
(2011) Fiction support for realistic portrayals of fear-type emotional manifestations. Computer Speech & Language, 25(1), 63–83. DOI logoGoogle Scholar
Clavel, C., Cafaro, A., Campano, S., & Pelachaud, C.
(2016) Fostering User Engagement in Face-to-Face Human-Agent Interactions: A Survey. In A. Esposito & L. C. Jain (Eds.), Toward Robotic Socially Believable Behaving Systems – Volume II: Modeling Social Signals (pp. 93–120). Cham: Springer International Publishing. DOI logoGoogle Scholar
Clavel, C., & Callejas, Z.
(2016) Sentiment analysis: from opinion mining to human-agent interaction. IEEE Transactions on affective computing, 7(1), 74–93. DOI logoGoogle Scholar
Cowie, R., & Cornelius, R. R.
(2003) Describing the emotional states that are expressed in speech. Speech communication, 40(1–2), 5–32. DOI logoGoogle Scholar
Dautenhahn, K.
(2007) Socially intelligent robots: dimensions of human-robot interaction. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 362(1480), 679–704. DOI logoGoogle Scholar
Dickerson, P., Robins, B., & Dautenhahn, K.
(2013) Where the action is: A conversation analytic perspective on interaction between a humanoid robot, a co-present adult and a child with an ASD. Interaction Studies, 14(2), 297–316. DOI logoGoogle Scholar
Duranti, A.
(1997) Linguistic anthropology. New York: Cambridge University Press. DOI logoGoogle Scholar
Duranti, A., & Goodwin, C.
(1992) Rethinking context: an introduction. In A. Duranti & C. Goodwin (Eds.), Rethinking Context, language as an interactive phenomenon (pp. 1–42). Cambridge University Press.Google Scholar
Fong, T., Nourbakhsh, I., & Dautenhahn, K.
(2003) A survey of socially interactive robots. Robotics and Autonomous Systems, 42(3–4), 143–166. DOI logoGoogle Scholar
Ford, C., & Thompson, S.
(1996) Interactional units in conversation: syntactic, intonational, and pragmatic resources for the management of turns. In E. Ochs, E. Schegloff, & S. Thompson (Eds.), Interaction and Grammar (134-184): Cambridge University Press. DOI logoGoogle Scholar
Garfinkel, H.
(1967) Studies in ethnomethodology. Engelwood Cliffs: Prentice-Hall.Google Scholar
Goffman, E.
(1973) La mise en scène de la vie quotidienne (Vol. 21. Les relations en public). Paris: Les Editions de minuit.Google Scholar
(1983) The interaction order. American Sociological Review, 48(1), pp: 1–17. DOI logoGoogle Scholar
Goodwin, C.
(1981) Conversational organization: interaction between speakers and hearers. New York: Academic Press.Google Scholar
Heritage, J.
(1991) L’Ethnométhodologie : une approche procédurale de l’action et de la communication. Réseaux CNET, 501, 89–130. DOI logoGoogle Scholar
Jefferson, G.
(1978) Sequential aspects of storytelling in conversation. In J. Schenkein (Ed.), Studies in the organization of conversational interaction (pp. 219–248). New York: New York Academic Press. DOI logoGoogle Scholar
Jones, R. A.
(2017) What makes a robot “social”? Social Studies of Science, 47(4), 556–579. DOI logoGoogle Scholar
Langlet, C., & Clavel, C.
(2014) Modelling user’s attitudinal reactions to the agent utterances: focus on the verbal content. Paper presented at the 5th International Workshop on Corpora for Research on Emotion, Sentiment & Social Signals (ES3 2014), Reykjavik, Iceland.
(2018) Detecting User’s Likes and Dislikes for a Virtual Negotiating Agent. Paper presented at the 20th ACM International Conference on Multimodal Interaction, Boulder, USA. DOI logo
Levinson, S. C.
(1983) Pragmatics. Cambridge University Press. DOI logoGoogle Scholar
Licoppe, C., & Figeac, J.
(2014) L’Organisation temporelle des engagements visuels dans des situations de multi-activité équipée en milieu urbain. Activités, 11(1).Google Scholar
Licoppe, C., & Rollet, N.
(2020, in press) « Je dois y aller ». Analyses de séquences de clôtures entre humains et robot. Réseaux. La Découverte. DOI logoGoogle Scholar
Mondada, L.
(2006) Participants’ online analysis and multimodal practices: projecting the end of the turn and the closing of the sequence. Discourse Studies, 8(1), pp. 117–129. DOI logoGoogle Scholar
(2008) Documenter l’articulation des ressources multimodales dans le temps : la transcription d’enregistrements vidéos d’interactions. In M. Bilger (Ed.), Données orales. Les enjeux de la transcription (Vol. 371), pp. 127–156. Perpignan: Presses Universitaires de Perpignan.Google Scholar
Mohri, M., Rostamizadeh, A., & Talwalkar, A.
(2012) Foundations of machine learning. MIT press.Google Scholar
Ochs, E.
(1979) Transcription as a theory. In B. B. Schieffelin (Ed.), Developmental Pragmatics (pp. 42–72). Academic Press.Google Scholar
Pelachaud, C., & Glas, N.
(2015a) Definitions of Engagement in Human-Agent Interaction. Paper presented at the International Workshop on Engagement in Human Computer Interaction (ENHANCE), Xi’an, China.
(2015b) Topic transition strategies for an information-giving agent. Paper presented at the 15th European Workshop on natural Language Generation, Brighton, UK.
Pelikan, H. R. M., & Broth, M.
(2016) Why That Nao?: How Humans Adapt to a Conventional Humanoid Robot in Taking Turns-at-Talk. Paper presented at the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, California, USA. DOI logo
Pitsch, K., Kuzuoka, H., Suzuki, Y., Süssenbach, L., Luff, P., & Heath, C.
(2009) “The first five seconds”: Contingent stepwise entry into an interaction as a means to secure sustained engagement in Human-Robot-Interaction. Paper presented at the IEEE International Symposium on Robot and Human Interactive Communication ROMAN 2009, Toyama, Japan.
Porcheron, M., Fischer, J. E., Reeves, S., & Sharples, S.
(2018) Voice Interfaces in Everyday Life. Montreal QC, Canada: Association for Computing Machinery. DOI logoGoogle Scholar
Rollet, N.
(2010) All the things you are. Activité multimodale, frontières et musiques improvisées en répétition In N. Andrieux-Reix (Ed.), Frontières. Du linguistique au sémiotique (pp. 279–302). Limoges: Lambert-Lucas.Google Scholar
(2013) “D’accord”. Approche conversationnelle et multimodale d’une forme située dans les appels au Samu-Centre 15. L’Information grammaticale, 1391.Google Scholar
Rollet, N., Jain, V., Licoppe, C., & Devillers, L.
(2017) Towards Interactional Symbiosis: Epistemic Balance and Co-presence in a Quantified Self Experiment. In L. Gamberini, A. Spagnolli, G. Jacucci, B. Blankertz, & J. Freeman (Eds.), Symbiotic Interaction: 5th International Workshop, Symbiotic 2016, Padua, Italy, September 29–30, 2016, Revised Selected Papers (pp. 143–154). Cham: Springer International Publishing. DOI logoGoogle Scholar
Sacks, H.
(1984) On doing ‘being ordinary’. In J. M. Atkinson & J. Heritage (Eds.), Structures of Social Action (pp. 413–429). Cambridge: CUP.Google Scholar
(1992) Lectures on conversation (Jefferson, G. ed.). Oxford: Blackwell.Google Scholar
Sacks, H., & Schegloff, E.
(1973) Opening up closings. Semiotica, 81, 289–326.Google Scholar
Sacks, H., Schegloff, E. A., & Jefferson, G.
(1974) A symplest systematics for the organization of turn-taking for conversation. Language, 501, 696–731. DOI logoGoogle Scholar
Sadazuka, K., Kuno, Y., Kawashima, M., & Yamazaki, K.
(2007) Museum Guide Robot with Effective Head Gestures. Paper presented at the International Conference on Control, Automation and Systems, Seoul, Korea. DOI logo
Schegloff, E.
(1987) Between micro and macro: contexts and other connections. In B. Giesen, J. C. Alexander, R. Münch, & N. J. Smelser (Eds.), The Micro-macro link (pp. 207–234). Los Angeles: University of California Press.Google Scholar
(2002) Accounts of conduct in interaction. Interruption, overlap, and turn-taking. In T. J. H. (Ed.), Handbook of sociological theory (pp. 287–321). New York: Kluwer Academic/Plenum publishers.Google Scholar
(2007) Sequence organization in interaction. A Primer in Conversation Analysis. (Vol. 11): Cambridge University Press. DOI logoGoogle Scholar
Sidner, C. L., & Dzikovska, M.
(2002 16–16 Oct. 2002) Human-robot interaction: engagement between humans and robots for hosting activities. Paper presented at the 4th IEEE International Conference on Multimodal Interfaces, Pittsburgh, USA.
Sidner, C. L., Lee, C., Kidd, C. D., & Rich, C.
(2005) Explorations in engagement for humans and robots. Artificial Intelligence, 1661, 140–164. DOI logoGoogle Scholar
Stivers, T.
(2008) Stance, Alignment, and Affiliation During Storytelling: When Nodding Is a Token of Affiliation. Research on Language & Social Interaction, 44(1), 31–57. DOI logoGoogle Scholar
(2015) Coding Social Interaction: A Heretical Approach in Conversation Analysis? Research on Language and Social Interaction, 48(1), 1–19. DOI logoGoogle Scholar
Šabanović, S., & Chang, W.-L.
(2016) Socializing robots: constructing robotic sociality in the design and use of the assistive robot PARO. AI and Society, 31(4), 537–551. DOI logoGoogle Scholar
Straub, I.
(2016) “It looks like a human!” The interrelation of social presence, interaction and agency ascription: a case study about the effects of an android robot on social agency ascription. AI and Society, 31(4), 553–571. DOI logoGoogle Scholar
Suchman, L.
(2007) Human-Machine Reconfigurations. Plans and situated actions, 2nd edition. New York: Cambridge University Press.Google Scholar
Yu, Z., Scherer, S., Devault, D., Gratch, J., Stratou, G., Morency, L., & Cassell, J.
(2013) Multimodal Prediction of Psychological Disorder: Learning Verbal and Nonverbal Commonality in Adjacency Pairs. Paper presented at the 17th Workshop Series on the Semantics and Pragmatics of Dialogue, Amsterdam, Netherland.
Zimmerman, D.
(2006) How closing matters in emergency telephone calls. Paper presented at the Annual meeting of the American Sociology Association, Montréal, Canada.
Cited by

Cited by 1 other publications

Kopp, Stefan & Nicole Krämer
2021. Revisiting Human-Agent Communication: The Importance of Joint Co-construction and Understanding Mental States. Frontiers in Psychology 12 DOI logo

This list is based on CrossRef data as of 15 april 2022. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.