Cover not available
Part of
Advances in Sign Language Corpus Linguistics
Edited by Ella Wehrmeyer
[Studies in Corpus Linguistics 108] 2023
► pp. 353–382

Chapter 13
Utterance unit annotation for the Japanese Sign Language Dialogue Corpus

Towards a method for detecting interactional boundaries in spontaneous sign language dialogue

| National Institute of Informatics, Japan | The Graduate University for Advanced Studies, SOKENDAI, Japan
| The Graduate University for Advanced Studies, SOKENDAI, Japan
| Tsukuba University of Technology, Japan
| Future University Hakodate, Japan
| The Graduate University for Advanced Studies, SOKENDAI, Japan
| The University of Tokyo, Japan
| Tsukuba University of Technology, Japan
This chapter defines ‘utterance units’ and describes their annotation in the Japanese Sign Language (JSL) dialogue corpus, first focusing on how human annotators – native signers of JSL – identify and annotate utterance units, before reporting on part of speech (POS) tagging for JSL and semi-automatic annotation of utterance units. The utterance unit is an original concept for segmenting and annotating movement features in sign language dialogue, based on signers’ native sense. We postulate a fundamental interaction-specific unit for understanding interactional mechanisms (such as turn-taking) in sign language social interactions from the perspectives of conversation analysis and multimodal interaction studies. We explain differences between sentence and utterance units, the corpus construction and composition, and the annotation scheme, before analyzing how JSL native annotators annotated the units. Finally, we show the application potential of this research by presenting two case studies, the first exploring POS annotations, and the second a first attempt at automatic annotation using OpenPose software.
Keywords: utterance units, Japanese Sign Language dialogue corpus, POS annotations, OpenPose software
Article outline
  • 1.Introduction
  • 2.Sentence versus utterance units
  • 3.The Colloquial Corpus of Japanese Sign Language
    • 3.1Tasks, areas and participants
    • 3.2Filming services and video clip editing
  • 4.Annotation of utterance units
    • 4.1Identifying an utterance unit
    • 4.2Annotation of utterance units on the individual level
    • 4.3Integration level
  • 5.Trial annotation of utterance units
    • 5.1Quantitative analysis
    • 5.2Qualitative analysis of utterance units
      • 5.2.1Utterance unit including mouthing
      • 5.2.2Utterance unit segmented by gaze shift
      • 5.2.3Utterance unit bounded by interlocutor’s actions
  • 6.The application potential of this research
    • 6.1Part of speech annotation for the utterance unit
      • 6.1.1Target data and annotation tool
      • 6.1.2POS annotation guidelines
      • 6.1.3Results, challenges and discussions
    • 6.2Automatic detection of utterance units
      • 6.2.1Detecting body-keypoint positions
      • 6.2.2Results
      • 6.2.2Discussion
  • 7.Conclusions
  • Notes
  • Acknowledgments
  • References
  • Appendix
Published online: 3 April 2023
DOI logo
https://doi.org/10.1075/scl.108.13bon
References (38)
References
Bank, Richard, Crasborn, Onno & Van Hout, Roeland. 2016. The prominence of spoken language elements in a sign language. Linguistics 54(6): 1281–1305. DOI logoGoogle Scholar
Bono, Mayumi, Kikuchi, Kouhei, Cibulka, Paul & Osugi, Yutaka. 2014. A colloquial corpus of Japanese Sign Language: A design of language resources for observing sign language conversations. Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC 2014), Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard … Stelios Piperidis (eds), 1898–1904. Paris: ELRA. <[URL]> (27 August 2022).
Bono, Mayumi, Sakaida, Rui, Okada, Tomohiro, & Miyao, Yusuke. 2020. Utterance-unit annotation for the JSL Dialogue Corpus: Toward a multimodal approach to corpus linguistics. In Proceedings of the 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives (LREC 2020), Eleni Efthimiou, Stavroula-Evita Fotinea, Thomas Hanke, Julie Hochgesang, Jette Kristofferson & Johanna Mesch (eds), 13–20. Paris: ELRA. <[URL]> (27 August 2022).
Boyes Braem, Penny & Sutton-Spence, Rachel (eds). 2001. The Hands are the Head of the Mouth: The Mouth as Articulator in Sign Languages. Berlin: Signum.Google Scholar
Cao, Zhe, Simon, Tomas, Wei, Shih-En & Sheikh, Yaser. 2017. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 1302–1310. DOI logoGoogle Scholar
Cassidy, Stephen, Crasborn, Onno, Nieminen, Henri, Stoop, Wessel, Hulsbosch, Micha, Even, Susan, Komen, Erwin & Johnston, Trevor. 2018. Signbank: Software to support web based dictionaries of sign language. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018), Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi … Takenobu Tokunaga (eds), 2359–2364. Paris: ELRA. <[URL]> (27 August 2022).
Celebi, Sait, Aydin, Ali, Temiz, Talha & Arici, Tarik. 2013. Gesture recognition using skeleton data with weighted dynamic time warping. In Proceedings of the International Conference on Computer Vision Theory and Applications, Volume 1 (VISAPP2013), Sebastiano Battiato & José Braz (eds), 620–625. Institute for Systems and Technologies of Information, Control and Communication. DOI logoGoogle Scholar
Cormier, Kearsy, Quinto-Pozos, David, Sevcikova, Zed & Schembri, Adam. 2012. Lexicalisation and de-lexicalisation processes in sign languages: Comparing depicting constructions and viewpoint gestures. Language & Communication 32(4): 329–348. DOI logoGoogle Scholar
Cormier, Kearsy, Smith, Sandra & Zwets, Martine. 2013. Framing constructed action in British Sign Language narratives. Journal of Pragmatics 55: 119–139. DOI logoGoogle Scholar
Crasborn, Onno A. 2007. How to recognize a sentence when you see one. Sign Language & Linguistics 10(2): 103–111. DOI logoGoogle Scholar
Crasborn, Onno & Sloetjes, Han. 2008. Enhanced ELAN functionality for sign language corpora. In Proceedings of the 3rd Workshop on the Representation and Processing of Sign Languages: Construction and Exploitation of Sign Language Corpora (LREC 2008), Onno Crasborn, Thomas Hanke, Eleni Efthimiou, Inge Zwitserlood & Ernst Thoutenhoofd (eds), 39–43. Paris: ELDA. <[URL]> (27 August 2022).
Crasborn, Onno, Zwitserlood, Inge, Van der Kooij, Els & Ormel, Ellen. 2020. Global Signbank manual, version 2, Form–meaning units in sign languages: An inventory and studies of interpretation and use in Sign Language of the Netherlands (NGT). DOI logoGoogle Scholar
Den, Yasuharu, Koiso, Hanae, Maruyama, Takehiko, Maekawa, Kikuo, Takanashi, Katsuya, Enomoto, Mika & Yoshida, Nao. 2010. Two-level annotation of utterance units in Japanese dialogs: An empirically emerged scheme. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC 2010), Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk … Daniel Tapias (eds), 2103–2110. Paris: ELRA. <[URL]> (28 August 2022).
Ford, Cecilia & Thompson, Sandra. 1996. Interactional units in conversation: Syntactic, intonational, and pragmatic resources for the management of turns. In Interaction and Grammar [Studies in Interactional Sociolinguistics 13], Elinor Ochs, Emanuel Schegloff & Sandra Thompson, (eds), 134–184. Cambridge: CUP. DOI logoGoogle Scholar
Fragkiadakis, Manolis, Nyst, Victoria & Van der Putten, Peter. 2020. Signing as input for a dictionary query: Matching signs based on joint positions of the dominant hand. In Proceedings of the 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives (LREC 2020), Eleni Efthimiou, Stavroula-Evita Fotinea, Thomas Hanke, Julie A. Hochgesang, Jette Kristoffersen & Johanna Mesch (eds), 69–74. Paris: ELRA. <[URL]> (30 August 2022).
Johnston, Trevor, Van Roekel, Jane & Schembri, Adam. 2015. On the conventionalization of mouth actions in Australian Sign Language. Language and Speech 59(1): 1–40. DOI logoGoogle Scholar
Kendon, Adam. 1967. Some functions of gaze-direction in social interaction. Acta Psychologica 26: 22–63. DOI logoGoogle Scholar
. 1970. Movement coordination in social interaction: Some examples described. Acta Psychologica 32(2): 101–125. DOI logoGoogle Scholar
. 2004. Gesture: Visible Action as Utterance. Cambridge: CUP. DOI logoGoogle Scholar
Kendrick, Kobin. 2015. Other-initiated repair in English. Open Linguistics 1: 164–190. DOI logoGoogle Scholar
Kita, Sotaro & Ide, Sachiko. 2007. Nodding, aizuchi, and final particles in Japanese conversation: How conversation reflects the ideology of communication and social relationships. Journal of Pragmatics 39: 1242–1254. DOI logoGoogle Scholar
Kitzinger, Celia. 2013. Repair. In The Handbook of Conversation Analysis, Jack Sidnell & Tanya Stivers (eds), 229–256. Malden MA: Wiley Blackwell. DOI logoGoogle Scholar
Levinson, Stephen. 1983. Pragmatics [Cambridge Textbooks in Linguistics Series]. Cambridge: CUP. DOI logoGoogle Scholar
Liddell, Scott. 2003. Grammar, Gesture, and Meaning in American Sign Language. Cambridge: CUP. DOI logoGoogle Scholar
Maruyama, Takehiko, Den, Yasuharu & Koiso, Hanae. 2020. Design and annotation of two-level utterance units in Japanese. In In Search of Basic Units of Spoken Language: A Corpus-Driven Approach [Studies in Corpus Linguistics 94], Shlomo Izre’el, Heliana Mello, Alessandro Panunzi & Tommaso Raso (eds), 155–180. Amsterdam: John Benjamins. DOI logoGoogle Scholar
McNeill, David. 1992. Hand and Mind: What Gestures Reveal about Thought. Chicago IL: University of Chicago Press.Google Scholar
. 2005. Gesture and Thought. Chicago IL: University of Chicago Press. DOI logoGoogle Scholar
Mondada, Lorenza. 2018. Multiple temporalities of language and body in interaction: Challenges for transcribing multimodality. Research on Language and Social Interaction 51: 85–106. DOI logoGoogle Scholar
Nivre, Joakim, De Marneffe, Marie-Catherine, Ginter, Filip, Hajič, Jan, Manning, Christopher, Pyysalo, Sampo, Schuster, Sebastian, Tyers, Francis & Zeman, Daniel. 2020. Universal Dependencies v2: An ever-growing multilingual treebank collection. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC 2020), Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri … Stelios Piperidis (eds), 4034–4043. Paris: ELRA. <[URL]> (30 August 2022).
Östling, Robert, Börstell, Carl, Gärdenfors, Moa & Wirén, Mats. 2017. Universal Dependencies for Swedish Sign Language. In Proceedings of the 21st Nordic Conference on Computational Linguistics, Jörg Tiedemann & Nina Tahmasebi (eds), 303–308. <[URL]> (30 August 2022).
Osugi, Yutaka & Bono, Mayumi. 2015. Shuwa jinbungaku no kouchiku ni mukete (2): Shuwa gengo co-pasu project (Towards the construction of a humanities of sign language (2): Sign Language corpus project). In Shuwa, gengo, comyunike-shon, Nihon Shuwa Kenkyusho (ed.), 99–136. Kyoto: Bunrikaku.Google Scholar
Sacks, Harvey, Schegloff, Emanuel & Jefferson, Gail. 1974. A simplest systematics for the organization of turn-taking for conversation. Language 50: 696–735. DOI logoGoogle Scholar
Schegloff, Emanuel, Jefferson, Gail & Sacks, Harvey. 1977. The preference for self-correction in the organization of repair in conversation. Language 53(2): 361–382. DOI logoGoogle Scholar
Schegloff, Emanuel A. 2007. Sequence Organization in Interaction: A Primer in Conversation Analysis. Cambridge: CUP. DOI logoGoogle Scholar
Schneider, Pascal, Memmesheimer, Raphael, Kramer, Ivanna & Paulus, Dietrich. 2019. Gesture recognition in RGB videos using human body keypoints and dynamic time warping. In RoboCup 2019: Robot World Cup XXIII [Lecture Notes in Computer Science 11531], Stephan Chalup, Tim Niemueller, Jackrit Suthakorn & Mary-Anne Williams (eds), 281–293. Singapore: Springer. DOI logoGoogle Scholar
Selting, Margret. 2000. The construction of units in conversational talk. Language in Society 29(4): 477–517. DOI logoGoogle Scholar
Sidnell, Jack & Stivers, Tanya (eds). 2013. The Handbook of Conversation Analysis. Malden MA: Wiley Blackwell. DOI logoGoogle Scholar
Torigoe, Takashi. 1997. How does a conversation proceed in Japanese Sign Language? Focusing on overlapping of utterance Deaf dyads. Japanese Journal of Sign Linguistics 14(1): 13–20.Google Scholar