Cover not available

Article In: Sign Language & Linguistics: Online-First Articles

Machine-learning errors in Hong Kong Sign Language handshape recognition reflect markedness patterns attested in learning

 | The University of Hong Kong
 | The University of Hong Kong
ORCiD logo with link | The University of Hong Kong
 | The University of Hong Kong
 | The University of Hong Kong
This content is being prepared for publication; it may be subject to changes.
Abstract
Handshapes are a fundamental component of sign language phonology yet pose great difficulty for learners, who acquire handshapes according to physiological complexity, phonological features, and motor skills. Advances in machine-learning have led to handshape recognition models yielding high accuracy. Such models are designed for data processing, and it is unclear whether any errors made during training resemble those of humans. We present a handshape recognition model based on Hong Kong Sign Language, which was trained using input data equivalent to the average duration of study reported for beginner hearing adult learners (3 months, 968 signs, 62 handshapes). Unlike other models, ours was fed all frames surrounding each sign’s peak, analogous to human perception, yielding an accuracy of 62%. The error output and target output exhibit phonetic similarities. These errors resemble the production errors commonly made by human learners, which can be attributed to markedness. This indicates that markedness structure can also be inferred from sign perception.
Article outline
  • 1.Introduction
    • 1.1Machine learning approaches to handshape recognition
    • 1.2Background of HKSL
  • 2.Methods
    • 2.1Dataset
    • 2.2Preprocessing
    • 2.3Model
    • 2.4Quantifying articulatory distance with Go charts
    • 2.5Markedness and visual similarities
  • 3.Results
    • 3.1Handshape estimation performance
    • 3.2Descriptive error analysis
    • 3.3Representations of model’s handshape learning
    • 3.4Quantitative analysis of error types and articulatory distance
    • 3.5Markedness patterns in model errors
  • 4.Discussion
  • Acknowledgements
  • Notes
  • Author queries
  • References
References (89)
References
Ann, Jean. 1993. A linguistic investigation of the relationship between physiology and handshape. Tucson, AZ: University of Arizona doctoral dissertation. [URL]. (30 September, 2020).
Asadi-Aghbolaghi, Maryam, Albert Clapés, Marco Bellantonio, Hugo Jair Escalante, Víctor Ponce-López, Xavier Baró, Isabelle Guyon, Shohreh Kasaei & Sergio Escalera. 2017. Deep learning for action and gesture recognition in image sequences: A survey. In Sergio Escalera, Isabelle Guyon & Vassilis Athitsos (eds.), Gesture recognition, 539–578. Cham: Springer International Publishing. .Google Scholar logo with link to Google Scholar
Bonvillian, John D., Michael D. Orlansky, Lesley L. Novack, Raymond J. Folven & Pamela Holley-Wilcox. 1985. Language, cognitive, and cherological development: The first steps in sign language acquisition. In William C. Stokoe Jr. & Virginia Volterra (eds.), SLR ‘83: The III international symposium on sign language research, 10–22. Silver Spring, MD: Linstok.Google Scholar logo with link to Google Scholar
Boyes Braem, Penny. 1990. Acquisition of the handshape in American Sign Language: A preliminary analysis. In Virginia Volterra & Carol J. Erting (eds.), From gesture to language in hearing and deaf children, 107–127. Berlin & Heidelberg: Springer. .Google Scholar logo with link to Google Scholar
Brentari, Diane, Harry van der Hulst, Els van der Kooij & Wendy Sandler. 1996. [one] over [all]; [all] over [one]: a dependency phonology analysis of handshape in sign languages. Unpublished manuscript. Purdue University, University of Connecticut & Haifa University.Google Scholar logo with link to Google Scholar
Brentari, Diane. 1998. A prosodic model of sign language phonology. Cambridge, MA: MIT Press.Google Scholar logo with link to Google Scholar
. 2011. Sign language phonology. In John Goldsmith, Jason Riggle & Alan C. L. Yu (eds.), The handbook of phonological theory, 691–721. Oxford: John Wiley & Sons. .Google Scholar logo with link to Google Scholar
Bronstein, Michael, Evangelos Kalogerakis, Emanuele Rodola, Jonathan Masci & Davide Boscaini. 2016. Deep learning for shape analysis. In The 37th Annual Conference of the European Association for Computer Graphics: Tutorials (EG ’16), 1. Goslar: Eurographics Association. . (3 October, 2023).Google Scholar logo with link to Google Scholar
Camgoz, Necati Cihan, Simon Hadfield, Oscar Koller & Richard Bowden. 2017. SubUNets: End-to-end hand shape and continuous sign language recognition. In 2017 IEEE International Conference on Computer Vision (ICCV), 3075–3084. .Google Scholar logo with link to Google Scholar
Carbo, Alessa & Eric Nalisnick. 2025. Improving handshape representations for sign language processing: A graph neural network approach. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose & Violet Peng (eds.), The 2025 Conference on Empirical Methods in Natural Language Processing, 29122–29135. Suzhou, China: Association for Computational Linguistics. .Google Scholar logo with link to Google Scholar
Census and Statistics Department, Hong Kong Special Administrative Region. 2021. Social data collected via the general household survey: Special topics report — Report no.63 — Persons with disabilities and chronic diseases. Statistical Reports. Hong Kong: Census and Statistics Department, the Government of the Hong Kong Special Administrative Region. [URL]. (3 October, 2021).
Centre for Sign Linguistics and Deaf Studies, CUHK. Asian SignBank. [URL]. (27 June, 2024).
Chen Pichler, Deborah. 2009. Sign production by first-time hearing signers: A closer look at handshape accuracy. Cadernos de Saúde 21. 37–50. .Google Scholar logo with link to Google Scholar
. 2011. Sources of handshape error in first-time signers of ASL. In Gaurav Mathur & Donna Jo Napoli (eds.), Deaf around the world: The impact of language, 96–121. Oxford: Oxford University Press. .Google Scholar logo with link to Google Scholar
Conlin, Kimberly E., Gene R. Mirus, Claude Mauk & Richard P. Meier. 2000. The acquisition of first signs: Place, handshape, and movement. In Charlene Chamberlain, Jill P. Morford & Rachel I. Mayberry (eds.), Language acquisition by eye, 51–69. Mahwah, NJ: Lawrence Erlbaum.Google Scholar logo with link to Google Scholar
Deng, Xiaoming, Yinda Zhang, Shuo Yang, Ping Tan, Liang Chang, Ye Yuan & Hongan Wang. 2018. Joint hand detection and rotation estimation using CNN. IEEE Transactions on Image Processing 27(4). 1888–1900. .Google Scholar logo with link to Google Scholar
Eccarius, Petra. 2002. Finding common ground: A comparison of handshape across multiple sign languages. West Lafayette, IN: Purdue University MA thesis.Google Scholar logo with link to Google Scholar
. 2011. A constraint-based account of distributional differences in handshapes. In Rachel Channon & Harry van der Hulst (eds.), Formational units in sign languages, 261–284. Berlin: De Gruyter Mouton. . (27 June, 2024).Google Scholar logo with link to Google Scholar
Etxepare, Ricardo & Aritz Irurtzun. 2021. Gravettian hand stencils as sign language formatives. Philosophical Transactions of the Royal Society B: Biological Sciences. 376(1824)1. 20200205. .Google Scholar logo with link to Google Scholar
Fabiano-Smith, Leah & Jessica A. Barlow. 2010. Interaction in bilingual phonological acquisition: evidence from phonetic inventories. International Journal of Bilingual Education and Bilingualism 13(1). 81–97. .Google Scholar logo with link to Google Scholar
Fenlon, Jordan, Adam Schembri, Ramas Rentelis & Kearsy Cormier. 2013. Variation in handshape and orientation in British Sign Language: The case of the “1” hand configuration. Language & Communication 33(1). 69–91. .Google Scholar logo with link to Google Scholar
Ferreira, Pedro M., Jaime S. Cardoso & Ana Rebelo. 2019. On the role of multimodal learning in the recognition of sign language. Multimedia Tools and Applications 78(8). 10035–10056. .Google Scholar logo with link to Google Scholar
Fischer, Susan & Qunhu Gong. 2011. Marked hand configurations in Asian sign languages. In Rachel Channon & Harry van der Hulst (eds.), Formational units in sign languages, 19–42. Berlin: De Gruyter Mouton. .Google Scholar logo with link to Google Scholar
Forrest, Karen & Michele L. Morrisette. 1999. Feature analysis of segmental errors in children with phonological disorders. Journal of Speech, Language, and Hearing Research 42(1). 187–194. .Google Scholar logo with link to Google Scholar
Fukushima, Kunihiko. 1975. Cognitron: A self-organizing multilayered neural network. Biological Cybernetics 20(3). 121–136. .Google Scholar logo with link to Google Scholar
Good, Irving John. 1952. Rational decisions. Journal of the Royal Statistical Society. Series B (Methodological). 14(1). 107–114. [URL]. (3 October, 2023).
Guo, Hengkai, Guijin Wang, Xinghao Chen, Cairong Zhang, Fei Qiao & Huazhong Yang. 2017. Region ensemble network: Improving convolutional network for hand pose estimation. In 2017 IEEE International Conference on Image Processing (ICIP), 4512–4516. .Google Scholar logo with link to Google Scholar
Hanke, Thomas. 2004. HamNoSys — representing sign language data in language resources and language processing contexts. In The Fourth International Conference on Language Resources and Evaluation (LREC’04), 1–6. Lisbon: European Language Resources Association (ELRA).Google Scholar logo with link to Google Scholar
Hayes, Bruce. 2004. Phonological acquisition in Optimality Theory: The early stages. In René Kager, Joe Pater & Wim Zonneveld (eds.), Constraints in phonological acquisition, 158–203. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
He, Siming. 2019. Research of a sign language translation system based on deep learning. In 2019 International Conference on Artificial Intelligence and Advanced Manufacturing (AIAM), 392–396. Dublin, Ireland: IEEE. .Google Scholar logo with link to Google Scholar
HKSL Handshape Font. 2007. Chinese University of Hong Kong: Centre for Sign Linguistics and Deaf Studies.Google Scholar logo with link to Google Scholar
Ingram, David. 2016. Phonological acquisition. In Martyn Barrett (ed.), The development of language, 73–97. London: Psychology Press.Google Scholar logo with link to Google Scholar
Kakizaki, Manato, Abu Saleh Musa Miah, Koki Hirooka & Jungpil Shin. 2024. Dynamic Japanese Sign Language recognition throw hand pose estimation using effective feature extraction and classification approach. Sensors 24(3). .Google Scholar logo with link to Google Scholar
Kanda Kazuyuki [ 神田和幸]. 2010. 手話の言語的特性に関する研究: 手話電子化辞書のアーキテクチャ [A study of linguistic characteristics of Japanese Sign Language: Architecture of the electronic sign language dictionary]. Tokyo: 福村出版 [Fukumurashuppan]. [URL].
Karnopp, Lodenir Becker. 2002. Phonology acquisition in Brazilian Sign Language. In Bencie Woll & Gary Morgan (eds.), Directions in sign language acquisition, 29–53. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Kendon, Adam. 2004. Gesture: Visible action as utterance. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Kingma, Diederik P. & Jimmy Ba. 2017. Adam: A method for stochastic optimization. arXiv. . (3 October, 2023).Google Scholar logo with link to Google Scholar
Kohl, Patricia K. 1993. Early linguistic experience and phonetic perception: Implications for theories of developmental speech perception. Journal of Phonetics 211. 125–139. .Google Scholar logo with link to Google Scholar
Koller, Oscar, Hermann Ney & Richard Bowden. 2016. Deep Hand: How to train a CNN on 1 million hand images when your data is continuous and weakly labelled. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3793–3802. .Google Scholar logo with link to Google Scholar
Kothadiya, Deep R., Chintan M. Bhatt, Amjad Rehman, Faten S. Alamri & Tanzila Saba. 2023. SignExplainer: An explainable AI-enabled framework for sign language recognition with ensemble learning. IEEE Access 111. 47410–47419. .Google Scholar logo with link to Google Scholar
Koulierakis, Ioannis, Georgios Siolas, Eleni Efthimiou, Evita Fotinea & Andreas-Georgios Stafylopatis. 2020. Recognition of static features in sign language using key-points. In Eleni Efthimiou, Stavroula-Evita Fotinea, Thomas Hanke, Julie A. Hochgesang, Jette Kristoffersen & Johanna Mesch (eds.), The LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives, 123–126. Marseille: European Language Resources Association (ELRA). [URL]. (16 February, 2026).
Kuhl, Patricia K., Erica Stevens, Akiko Hayashi, Toshisada Deguchi, Shigeru Kiritani & Paul Iverson. 2006. Infants show a facilitation effect for native language phonetic perception between 6 and 12 months. Developmental Science 9(2). F13–F21. .Google Scholar logo with link to Google Scholar
Lim, Kian Ming, Alan Wee Chiat Tan, Chin Poo Lee & Shing Chiang Tan. 2019. Isolated sign language recognition using Convolutional Neural Network hand modelling and Hand Energy Image. Multimedia Tools and Applications 78(14). 19917–19944. Google Scholar logo with link to Google Scholar
Locke, John L. & Michael Studdert-Kennedy. 1983. Phonological acquisition and change. New York: Academic Press.Google Scholar logo with link to Google Scholar
Lugaresi, Camillo, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, et al. 2019. MediaPipe: A framework for building perception pipelines. arXiv. .Google Scholar logo with link to Google Scholar
Mak, Joe & Gladys Tang. 2011. Movement types, repetition, and feature organization in Hong Kong Sign Language. In Rachel Channon & Harry van der Hulst (eds.), Formational units in sign languages, 315–338. Berlin: De Gruyter Mouton.Google Scholar logo with link to Google Scholar
Marentette, Paula F. 1995. It’s in her hands: A case study of the emergence of phonology in American Sign Language. Montreal: McGill University PhD dissertation. [URL]. (26 June, 2024).
Marentette, Paula F. & Rachel I. Mayberry. 1999. Principles for an emerging phonological system: A case study of early ASL acquisition. In Charlene Chamberlain, Jill P. Morford & Rachel I. Mayberry (eds.), Language acquisition by eye, 71–90. New York, NY: Psychology Press.Google Scholar logo with link to Google Scholar
McIntire, Marina L. 1977. The acquisition of American Sign Language hand configurations. Sign Language Studies 161. 247–266. Google Scholar logo with link to Google Scholar
Meade, Gabriela, Brittany Lee, Natasja Massa, Phillip J. Holcomb, Katherine J. Midgley & Karen Emmorey. 2022. Are form priming effects phonological or perceptual? Electrophysiological evidence from American Sign Language. Cognition 2201. 104979. .Google Scholar logo with link to Google Scholar
Meier, Richard P., Claude Mauk, Gene R. Mirus & Kimberly E. Conlin. 1997. Motoric constraints on early sign acquisition. In Eve V. Clark (ed.), The Twenty-Ninth Annual Child Language Research Forum, 63–72. Stanford, CA: CSLI Press.Google Scholar logo with link to Google Scholar
Menn, Lise & Carol Stoel-Gammon. 1996. Phonological development. In Paul Fletcher & Brian MacWhinney (eds.), The handbook of child language, 335–360. Cambridge, MA: Blackwell. .Google Scholar logo with link to Google Scholar
Mertz, Justine, Chiara Annucci, Valentina Aristodemo, Beatrice Giustolisi, Doriane Gras, Giuseppina Turco, Carlo Geraci & Caterina Donati. 2022. Measuring sign complexity: Comparing a model-driven and an error-driven approach. Laboratory Phonology 24(1). .Google Scholar logo with link to Google Scholar
Miozzo, Michele & Francesca Peressotti. 2022. How the hand has shaped sign languages. Scientific Reports 12(1). 11980. .Google Scholar logo with link to Google Scholar
Oberweger, Markus, Paul Wohlhart & Vincent Lepetit. 2015. Training a feedback loop for hand pose estimation. In 2015 IEEE International Conference on Computer Vision (ICCV), 3316–3324. .Google Scholar logo with link to Google Scholar
Orlansky, Michael D. & John D. Bonvillian. 1988. Early sign acquisition. In Michael D. Smith & John L. Locke (eds.), The emergent lexicon: The child’s development of a linguistic vocabulary, 263–292. New York: Academic Press.Google Scholar logo with link to Google Scholar
Ortega, Gerardo. 2017. Iconicity and sign lexical acquisition: A review. Frontiers in Psychology 81. .Google Scholar logo with link to Google Scholar
Ortega, Gerardo & Gary Morgan. 2015a. Input processing at first exposure to a sign language. Second Language Research 31(4). 443–463. .Google Scholar logo with link to Google Scholar
. 2015b. Phonological development in hearing learners of a sign language: The influence of phonological parameters, sign complexity, and iconicity. Language Learning 65(3). 660–688. .Google Scholar logo with link to Google Scholar
Ortega, Gerardo, Annika Schiefner & Aslı Özyürek. 2019. Hearing non-signers use their gestures to predict iconic form-meaning mappings at first exposure to signs. Cognition 1911. 103996. .Google Scholar logo with link to Google Scholar
Paszke, Adam, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga & Adam Lerer. 2017. Automatic differentiation in PyTorch. In The 31st Conference on Neural Information Processing Systems (NIPS 2017). Long Beach, CA. [URL]. (3 October, 2023).
Pugeault, Nicolas & Richard Bowden. 2011. Spelling it out: Real-time ASL fingerspelling recognition. In 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), 1114–1119. .Google Scholar logo with link to Google Scholar
Rastgoo, Razieh, Kourosh Kiani & Sergio Escalera. 2018. Multi-modal deep hand sign language recognition in still images using restricted Boltzmann Machine. Entropy 20(11). 809. .Google Scholar logo with link to Google Scholar
Romani, Cristina, Andrew Olson, Carlo Semenza & Alessia Granà. 2002. Patterns of phonological errors as a function of a phonological versus an articulatory locus of impairment. Cortex 38(4). 541–567. .Google Scholar logo with link to Google Scholar
Sandler, Wendy. 1989. Phonological representation of the sign: Linearity and nonlinearity in American Sign Language. Dordrecht: Foris. Google Scholar logo with link to Google Scholar
. 2017. The challenge of sign language phonology. Annual Review of Linguistics 3(1). 43–63. .Google Scholar logo with link to Google Scholar
Sehyr, Zed Sevcikova, Naomi Caselli, Ariel M. Cohen-Goldberg & Karen Emmorey. 2021. The ASL-LEX 2.0 project: A database of lexical and phonological properties for 2,723 signs in American Sign Language. The Journal of Deaf Studies and Deaf Education 26(2). 263–277. .Google Scholar logo with link to Google Scholar
Siedlecki, Theodore Jr. 1991. The acquisition of American Sign Language phonology by young children of deaf parents. Lovingston, VA: University of Virginia PhD dissertation.Google Scholar logo with link to Google Scholar
Siedlecki, Theodore Jr. & John D. Bonvillian. 1993. Location, handshape & movement: Young children’s acquisition of the formational aspects of American Sign Language. Sign Language Studies 78(1). 31–52. Google Scholar logo with link to Google Scholar
. 1997. Young children’s acquisition of the handshape aspect of American Sign Language signs: Parental report findings. Applied Psycholinguistics 18(1). 17–39. .Google Scholar logo with link to Google Scholar
Siu, Wai Yan Rebecca. 2016. Location variation in Hong Kong Sign Language (HKSL). Asia-Pacific Language Variation 2(1). 4–47. .Google Scholar logo with link to Google Scholar
Stokoe, William C. Jr. 1960. Sign language structure: An outline of the visual communication system of the American deaf. Studies in Linguistics: Occasional Papers 81. Silver Spring, MD: Linstok Press. [URL]. (29 March, 2024).
Sutton-Spence, Rachel & Bencie Woll. 1999. The linguistics of British Sign Language: An introduction. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Sze, Felix, Connie Lo, Lisa Lo & Kenny Chu. 2013. Historical development of Hong Kong Sign Language. Sign Language Studies 13(2). 155–185. Google Scholar logo with link to Google Scholar
Tang, Gladys. 2007. Hong Kong Sign Language: A trilingual dictionary with linguistic descriptions. Hong Kong: Chinese University Press.Google Scholar logo with link to Google Scholar
. 2015. Hong Kong Sign Language. In William S-Y Wang & Chaofen Sun (eds.), The Oxford handbook of Chinese linguistics, 710–728. New York, NY: Oxford University Press.Google Scholar logo with link to Google Scholar
Tennant, Richard A. & Marianne Gluszak Brown. 2020. The American Sign Language handshape dictionary. Washington: Gallaudet University Press. [URL].
Thierfelder, Philip, Gillian Wigglesworth & Gladys Tang. 2020. Sign phonological parameters modulate parafoveal preview effects in deaf readers. Cognition 2011. 104286. .Google Scholar logo with link to Google Scholar
Thompson, Arthur Lewis, Wing Cheung Aaron Chik, Yu On Mavies Ngai, Pui Ching Rachel Chen, Chui Yin Judy Ng & Youngah Do. 2026. Iconicity and semantic transparency in Hong Kong Sign Language: Evidence from ratings and three guessing paradigms. Language and Cognition 181. e21. .Google Scholar logo with link to Google Scholar
Thompson, Arthur Lewis, Thomas Van Hoey, Aaron Wing Cheung Chik & Youngah Do. 2025. Iconic hand gestures from ideophones exhibit stability and emergent phonological properties: An iterated learning study. Cognitive Linguistics 36(2). 227–259. .Google Scholar logo with link to Google Scholar
Van der Kooij, Els. 2002. Phonological categories in sign language of the Netherlands: The role of phonetic implementation and iconicity. Utrecht: Universiteit Utrecht PhD dissertation. [URL].
Vihman, Marilyn. 2015. Perception and production in phonological development. In Brian MacWhinney & William O’Grady (eds.), The handbook of language emergence, 437–457. John Wiley & Sons. .Google Scholar logo with link to Google Scholar
Wadhawan, Ankita & Parteek Kumar. 2020. Deep learning-based sign language recognition system for static signs. Neural Computing and Applications 32(12). 7957–7968. .Google Scholar logo with link to Google Scholar
Werker, Janet F., H. Henny Yeung & Katherine A. Yoshida. 2012. How do infants become experts at native-speech perception? Current Directions in Psychological Science. SAGE Publications. . (27 June, 2024).Google Scholar logo with link to Google Scholar
Whitworth, Cecily. 2011. Features and natural classes in ASL handshapes. Sign Language Studies 12(1). 46–71. [URL]. (26 June, 2024).
Wong, Yuet On. 2008. Acquisition of handshape in Hong Kong Sign Language: A case study. Hong Kong: Chinese University of Hong Kong PhD dissertation.Google Scholar logo with link to Google Scholar
Zheng, Lihong, Bin Liang & Ailian Jiang. 2017. Recent advances of deep learning for sign language recognition. In 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA), 1–7. Sydney, NSW: IEEE. . (26 June, 2024).Google Scholar logo with link to Google Scholar
Zimmerman, Thomas G., Jaron Lanier, Chuck Blanchard, Steve Bryson & Young Harvill. 1986. A hand gesture interface device. In The SIGCHI/GI Conference on Human Factors in Computing Systems and Graphics Interface (CHI ’87), 189–192. New York, NY: Association for Computing Machinery. .Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue