Cover not available
Part of
Advances in Sign Language Corpus Linguistics
Edited by Ella Wehrmeyer
[Studies in Corpus Linguistics 108] 2023
► pp. 353–382

Chapter 13
Utterance unit annotation for the Japanese Sign Language Dialogue Corpus

Towards a method for detecting interactional boundaries in spontaneous sign language dialogue

| National Institute of Informatics | The Graduate University for Advanced Studies, SOKENDAI
| The Graduate University for Advanced Studies, SOKENDAI
| Tsukuba University of Technology
| Future University Hakodate
| The Graduate University for Advanced Studies, SOKENDAI
| The University of Tokyo
| Tsukuba University of Technology

This chapter defines ‘utterance units’ and describes their annotation in the Japanese Sign Language (JSL) dialogue corpus, first focusing on how human annotators – native signers of JSL – identify and annotate utterance units, before reporting on part of speech (POS) tagging for JSL and semi-automatic annotation of utterance units. The utterance unit is an original concept for segmenting and annotating movement features in sign language dialogue, based on signers’ native sense. We postulate a fundamental interaction-specific unit for understanding interactional mechanisms (such as turn-taking) in sign language social interactions from the perspectives of conversation analysis and multimodal interaction studies. We explain differences between sentence and utterance units, the corpus construction and composition, and the annotation scheme, before analyzing how JSL native annotators annotated the units. Finally, we show the application potential of this research by presenting two case studies, the first exploring POS annotations, and the second a first attempt at automatic annotation using OpenPose software.

Keywords: utterance units, Japanese Sign Language dialogue corpus, POS annotations, OpenPose software
Article outline
  • 1.Introduction
  • 2.Sentence versus utterance units
  • 3.The Colloquial Corpus of Japanese Sign Language
    • 3.1Tasks, areas and participants
    • 3.2Filming services and video clip editing
  • 4.Annotation of utterance units
    • 4.1Identifying an utterance unit
    • 4.2Annotation of utterance units on the individual level
    • 4.3Integration level
  • 5.Trial annotation of utterance units
    • 5.1Quantitative analysis
    • 5.2Qualitative analysis of utterance units
      • 5.2.1Utterance unit including mouthing
      • 5.2.2Utterance unit segmented by gaze shift
      • 5.2.3Utterance unit bounded by interlocutor’s actions
  • 6.The application potential of this research
    • 6.1Part of speech annotation for the utterance unit
      • 6.1.1Target data and annotation tool
      • 6.1.2POS annotation guidelines
      • 6.1.3Results, challenges and discussions
    • 6.2Automatic detection of utterance units
      • 6.2.1Detecting body-keypoint positions
      • 6.2.2Results
      • 6.2.2Discussion
  • 7.Conclusions
  • Notes
  • Acknowledgments
  • References
  • Appendix
Published online: 3 April 2023
DOI logo
https://doi.org/10.1075/scl.108.13bon
References
Bank, Richard, Crasborn, Onno & Van Hout, Roeland
2016 The prominence of spoken language elements in a sign language. Linguistics 54(6): 1281–1305. DOI logoGoogle Scholar
Bono, Mayumi, Kikuchi, Kouhei, Cibulka, Paul & Osugi, Yutaka
2014 A colloquial corpus of Japanese Sign Language: A design of language resources for observing sign language conversations. Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC 2014), Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard … Stelios Piperidis (eds), 1898–1904. Paris: ELRA. [URL] (27 August 2022).
Bono, Mayumi, Sakaida, Rui, Okada, Tomohiro, & Miyao, Yusuke
2020 Utterance-unit annotation for the JSL Dialogue Corpus: Toward a multimodal approach to corpus linguistics. In Proceedings of the 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives (LREC 2020), Eleni Efthimiou, Stavroula-Evita Fotinea, Thomas Hanke, Julie Hochgesang, Jette Kristofferson & Johanna Mesch (eds), 13–20. Paris: ELRA. [URL] (27 August 2022).
Boyes Braem, Penny & Sutton-Spence, Rachel
(eds) 2001 The Hands are the Head of the Mouth: The Mouth as Articulator in Sign Languages. Berlin: Signum.Google Scholar
Cao, Zhe, Simon, Tomas, Wei, Shih-En & Sheikh, Yaser
2017 Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 1302–1310. DOI logoGoogle Scholar
Cassidy, Stephen, Crasborn, Onno, Nieminen, Henri, Stoop, Wessel, Hulsbosch, Micha, Even, Susan, Komen, Erwin & Johnston, Trevor
2018 Signbank: Software to support web based dictionaries of sign language. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018), Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi … Takenobu Tokunaga (eds), 2359–2364. Paris: ELRA. [URL] (27 August 2022).
Celebi, Sait, Aydin, Ali, Temiz, Talha & Arici, Tarik
2013 Gesture recognition using skeleton data with weighted dynamic time warping. In Proceedings of the International Conference on Computer Vision Theory and Applications, Volume 1 (VISAPP2013), Sebastiano Battiato & José Braz (eds), 620–625. Institute for Systems and Technologies of Information, Control and Communication. DOI logoGoogle Scholar
Cormier, Kearsy, Quinto-Pozos, David, Sevcikova, Zed & Schembri, Adam
2012 Lexicalisation and de-lexicalisation processes in sign languages: Comparing depicting constructions and viewpoint gestures. Language & Communication 32(4): 329–348. DOI logoGoogle Scholar
Cormier, Kearsy, Smith, Sandra & Zwets, Martine
2013 Framing constructed action in British Sign Language narratives. Journal of Pragmatics 55: 119–139. DOI logoGoogle Scholar
Crasborn, Onno A.
2007 How to recognize a sentence when you see one. Sign Language & Linguistics 10(2): 103–111. DOI logoGoogle Scholar
Crasborn, Onno & Sloetjes, Han
2008 Enhanced ELAN functionality for sign language corpora. In Proceedings of the 3rd Workshop on the Representation and Processing of Sign Languages: Construction and Exploitation of Sign Language Corpora (LREC 2008), Onno Crasborn, Thomas Hanke, Eleni Efthimiou, Inge Zwitserlood & Ernst Thoutenhoofd (eds), 39–43. Paris: ELDA. [URL] (27 August 2022).
Crasborn, Onno, Zwitserlood, Inge, Van der Kooij, Els & Ormel, Ellen
2020 Global Signbank manual, version 2, Form–meaning units in sign languages: An inventory and studies of interpretation and use in Sign Language of the Netherlands (NGT). DOI logoGoogle Scholar
Den, Yasuharu, Koiso, Hanae, Maruyama, Takehiko, Maekawa, Kikuo, Takanashi, Katsuya, Enomoto, Mika & Yoshida, Nao
2010 Two-level annotation of utterance units in Japanese dialogs: An empirically emerged scheme. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC 2010), Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk … Daniel Tapias (eds) 2103–2110 Paris: ELRA. [URL] (28 August 2022).
Ford, Cecilia & Thompson, Sandra
1996 Interactional units in conversation: Syntactic, intonational, and pragmatic resources for the management of turns. In Interaction and Grammar [Studies in Interactional Sociolinguistics 13], Elinor Ochs, Emanuel Schegloff & Sandra Thompson, (eds), 134–184. Cambridge: CUP. DOI logoGoogle Scholar
Fragkiadakis, Manolis, Nyst, Victoria & Van der Putten, Peter
2020 Signing as input for a dictionary query: Matching signs based on joint positions of the dominant hand. In Proceedings of the 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives (LREC 2020), Eleni Efthimiou, Stavroula-Evita Fotinea, Thomas Hanke, Julie A. Hochgesang, Jette Kristoffersen & Johanna Mesch (eds), 69–74. Paris: ELRA. [URL] (30 August 2022).
Johnston, Trevor, Van Roekel, Jane & Schembri, Adam
2015 On the conventionalization of mouth actions in Australian Sign Language. Language and Speech 59(1): 1–40. DOI logoGoogle Scholar
Kendon, Adam
1967 Some functions of gaze-direction in social interaction. Acta Psychologica 26: 22–63. DOI logoGoogle Scholar
1970 Movement coordination in social interaction: Some examples described. Acta Psychologica 32(2): 101–125. DOI logoGoogle Scholar
2004 Gesture: Visible Action as Utterance. Cambridge: CUP. DOI logoGoogle Scholar
Kendrick, Kobin
2015 Other-initiated repair in English. Open Linguistics 1: 164–190. DOI logoGoogle Scholar
Kita, Sotaro & Ide, Sachiko
2007 Nodding, aizuchi, and final particles in Japanese conversation: How conversation reflects the ideology of communication and social relationships. Journal of Pragmatics 39: 1242–1254. DOI logoGoogle Scholar
Kitzinger, Celia
2013 Repair. In The Handbook of Conversation Analysis, Jack Sidnell & Tanya Stivers (eds), 229–256. Malden MA: Wiley Blackwell. DOI logoGoogle Scholar
Levinson, Stephen
1983 Pragmatics [Cambridge Textbooks in Linguistics Series]. Cambridge: CUP. DOI logoGoogle Scholar
Liddell, Scott
2003 Grammar, Gesture, and Meaning in American Sign Language. Cambridge: CUP. DOI logoGoogle Scholar
Maruyama, Takehiko, Den, Yasuharu & Koiso, Hanae
2020 Design and annotation of two-level utterance units in Japanese. In In Search of Basic Units of Spoken Language: A Corpus-Driven Approach [Studies in Corpus Linguistics 94], Shlomo Izre’el, Heliana Mello, Alessandro Panunzi & Tommaso Raso (eds), 155–180. Amsterdam: John Benjamins. DOI logoGoogle Scholar
McNeill, David
1992 Hand and Mind: What Gestures Reveal about Thought. Chicago IL: University of Chicago Press.Google Scholar
2005 Gesture and Thought. Chicago IL: University of Chicago Press. DOI logoGoogle Scholar
Mondada, Lorenza
2018 Multiple temporalities of language and body in interaction: Challenges for transcribing multimodality. Research on Language and Social Interaction 51: 85–106. DOI logoGoogle Scholar
Nivre, Joakim, De Marneffe, Marie-Catherine, Ginter, Filip, Hajič, Jan, Manning, Christopher, Pyysalo, Sampo, Schuster, Sebastian, Tyers, Francis & Zeman, Daniel
2020 Universal Dependencies v2: An ever-growing multilingual treebank collection. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC 2020), Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri … Stelios Piperidis (eds), 4034–4043. Paris: ELRA. [URL] (30 August 2022).
Östling, Robert, Börstell, Carl, Gärdenfors, Moa & Wirén, Mats
2017 Universal Dependencies for Swedish Sign Language. In Proceedings of the 21st Nordic Conference on Computational Linguistics, Jörg Tiedemann & Nina Tahmasebi (eds), 303–308. [URL] (30 August 2022).
Osugi, Yutaka & Bono, Mayumi
2015 Shuwa jinbungaku no kouchiku ni mukete (2): Shuwa gengo co-pasu project (Towards the construction of a humanities of sign language (2): Sign Language corpus project). In Shuwa, gengo, comyunike-shon, Nihon Shuwa Kenkyusho (ed.), 99–136. Kyoto: Bunrikaku.Google Scholar
Sacks, Harvey, Schegloff, Emanuel & Jefferson, Gail
1974 A simplest systematics for the organization of turn-taking for conversation. Language 50: 696–735. DOI logoGoogle Scholar
Schegloff, Emanuel, Jefferson, Gail & Sacks, Harvey
1977 The preference for self-correction in the organization of repair in conversation. Language 53(2): 361–382. DOI logoGoogle Scholar
Schegloff, Emanuel A.
2007 Sequence Organization in Interaction: A Primer in Conversation Analysis. Cambridge: CUP. DOI logoGoogle Scholar
Schneider, Pascal, Memmesheimer, Raphael, Kramer, Ivanna & Paulus, Dietrich
2019 Gesture recognition in RGB videos using human body keypoints and dynamic time warping. In RoboCup 2019: Robot World Cup XXIII [Lecture Notes in Computer Science 11531], Stephan Chalup, Tim Niemueller, Jackrit Suthakorn & Mary-Anne Williams (eds), 281–293. Singapore: Springer. DOI logoGoogle Scholar
Selting, Margret
2000 The construction of units in conversational talk. Language in Society 29(4): 477–517. DOI logoGoogle Scholar
Sidnell, Jack & Stivers, Tanya
(eds) 2013 The Handbook of Conversation Analysis. Malden MA: Wiley Blackwell. DOI logoGoogle Scholar
Torigoe, Takashi
1997 How does a conversation proceed in Japanese Sign Language? Focusing on overlapping of utterance Deaf dyads. Japanese Journal of Sign Linguistics 14(1): 13–20.Google Scholar