Augmented reality coloring book: An interactive strategy for teaching children with autism to focus on specific nonverbal social cues to promote their social skills
I-Jui Lee | National Taipei University of Technology, Taipei Tech
Autism spectrum disorders (ASD) reduce one’s ability to act appropriately in social situations. Increasing
evidence indicates that children with ASD might ignore nonverbal social cues that usually aid social interaction because they do
not recognize or understand them. We asked children with ASD to color an augmented reality coloring book (ARCB) to teach them how
to recognize and understand some specific social signals and to ignore others. ARCB materials teach children to recognize and
understand social signals in various ways. They can, for example, view 3D animations of the ARCB materials on a tablet computer.
Thus, the ARCB can be used to help children with ASD focus their attention on the meaning and social value of nonverbal behaviors
in specific social situations. The ARCB has multiple functions: it extends the social features of the story, and it restricts
attention to the most important parts of the videos. Single-subject research with a multiple-baselines across-subject design was
used in this study. After five weeks of ARCB training intervention, all 3 participants’ scores rose significantly and dramatically
during the intervention phase (mean rate of correct answers improved from 14.24% to 47.33%), and remained significantly higher in
the maintenance phase than at baseline. We conclude that coloring pictures of social situations may help children with ASD
recognize and better understand these situations.
(2006) The complete guide to Asperger’s syndrome. Jessica Kingsley Publishers.
American Psychiatric Association
(2013) Diagnostic and statistical manual of mental disorders (DSM-5®). American Psychiatric Pub.
Baker, J. E.
(2003a) Social skills picture book: Teaching play, emotion, and communication to children with autism. Arlington, TX: Future Horizons.
Baker, J. E.
(2003b) Social skills training. Overland Park, KS: Autism Asperger Publishing.
Baird, G., & Norbury, C. F.
(2016) Social (pragmatic) communication disorders and autism spectrum disorder. Archives of Disease in Childhood, 101(8), 745–751.
Broadhead, P.
(2006) Developing an understanding of young children’s learning through play: the place of observation, interaction and reflection. British Educational Research Journal, 32(2), 191–207.
Castelli, F.
(2005) Understanding emotions from standardized facial expressions in autism and normal development. Autism, 9(4), 428–449.
Chen, C. H., Lee, I. J., & Lin, L. Y.
(2015) Augmented reality-based self-facial modeling to promote the emotional expression and social skills of adolescents with autism spectrum disorders. Research in developmental disabilities, 361, 396–403.
Chen, C. H., Lee, I. J., & Lin, L. Y.
(2016) Augmented reality-based video-modeling storybook of nonverbal facial cues for children with autism spectrum disorder to improve their perceptions and judgments of facial expressions and emotions. Computers in Human Behavior, 551, 477–485.
Clark, A., & Dünser, A.
(2012, March). An interactive augmented reality coloring book. In 3D User Interfaces (3DUI), 2012 IEEE Symposium on (pp. 7–10). IEEE.
Curry, N. A., & Kasser, T.
(2005) Can coloring mandalas reduce anxiety?Art Therapy, 22(2), 81–85.
Escobedo, L., Tentori, M., Quintana, E., Favela, J., & Garcia-Rosas, D.
(2014) Using augmented reality to help children with autism stay focused. IEEE Pervasive Computing, 13(1), 38–46.
Farr, W., Yuill, N., & Hinske, S.
(2012) An augmented toy and social interaction in children with autism. International Journal of Arts and Technology, 5(2–4), 104–125.
Fodstad, J. C., Matson, J. L., Hess, J., & Neal, D.
(2009) Social and communication behaviours in infants and toddlers with autism and pervasive developmental disorder-not otherwise specified. Developmental Neurorehabilitation, 12(3), 152–157.
Geschwind, D. H., & Levitt, P.
(2007) Autism spectrum disorders: developmental disconnection syndromes. Current opinion in neurobiology, 17(1), 103–111.
Gray, C. A.
(1998) Social stories and comic strip conversations with students with Asperger syndrome and high-functioning autism. Asperger syndrome or high-functioning autism?, 167–198.
Heath, M. A., Sheen, D., Leavy, D., Young, E., & Money, K.
(2005) Bibliotherapy: A resource to facilitate emotional healing and growth. School Psychology International, 26(5), 563–580.
Hedley, N. R.
(2003, August). Empirical evidence for advanced geographic visualization interface use. In International cartographic congress, Durban, South Africa.
Hobson, R. P., & Lee, A.
(1998) Hello and goodbye: A study of social engagement in autism. Journal of autism and developmental disorders, 28(2), 117–127.
Hornecker, E., & Buur, J.
(2006, April). Getting a grip on tangible interaction: a framework on physical space and social interaction. In Proceedings of the SIGCHI conference on Human Factors in computing systems (pp. 437–446). ACM.
Hoyek, N., Di Rienzo, F., Collet, C., Creveaux, T., & Guillot, A.
(2014) Hand mental rotation is not systematically altered by actual body position: Laterality judgment versus same–different comparison tasks. Attention, Perception, & Psychophysics, 76(2), 519–526.
Jarrold, C.
(2003) A review of research into pretend play in autism. Autism, 7(4), 379–390.
Kaufmann, H., & Schmalstieg, D.
(2003) Mathematics and geometry education with collaborative augmented reality. Computers & Graphics, 27(3), 339–345.
Klatzky, R. L., Wu, B., Shelton, D., & Stetten, G.
(2008) Effectiveness of augmented-reality visualization versus cognitive mediation for learning actions in near space. ACM Transactions on Applied Perception (TAP), 5(1), 1.
Koegel, L. K., Vernon, T. W., Koegel, R. L., Koegel, B. L., & Paullin, A. W.
(2012) Improving social engagement and initiations between children with autism spectrum disorder and their peers in inclusive settings. Journal of Positive Behavior Interventions, 14(4), 220–227.
Koldewyn, K., Weigelt, S., Kanwisher, N., & Jiang, Y.
(2013) Multiple objects tracking in autism spectrum disorders. Journal of autism and developmental disorders, 43(6), 1394–1405.
Krasny, L., Williams, B. J., Provencal, S., & Ozonoff, S.
(2003) Social skills interventions for the autism spectrum: essential ingredients and a model curriculum. Child and Adolescent Psychiatric Clinics of North America, 12(1), 107–122.
Lee, I. J., Chen, C. H., & Chang, K. P.
(2016) Augmented reality technology combined with three-dimensional holography to train the mental rotation ability of older adults. Computers in Human Behavior, 651, 488–500.
Lillard, A. S., Lerner, M. D., Hopkins, E. J., Dore, R. A., Smith, E. D., & Palmquist, C. M.
(2013) The impact of pretend play on children’s development: A review of the evidence. Psychological bulletin, 139(1), 1.
Lu, L., Petersen, F., Lacroix, L., & Rousseau, C.
(2010) Stimulating creative play in children with autism through sandplay. The Arts in Psychotherapy, 37(1), 56–64.
Marshall, P.
(2007, February). Do tangible interfaces enhance learning? In Proceedings of the 1st international conference on tangible and embedded interaction (pp. 163–170). ACM.
Martins, M. P., & Harris, S. L.
(2006) Teaching children with autism to respond to joint attention initiations. Child & Family Behavior Therapy, 28(1), 51–68.
Matson, J. L.
(2009) Applied behavior analysis for children with autism spectrum disorders. New York: Springer.
Magnenat, S., Ngo, D. T., Zünd, F., Ryffel, M., Noris, G., Rothlin, G., … & Sumner, R. W.
(2015) Live texturing of augmented reality characters from colored drawings. IEEE transactions on visualization and computer graphics, 21(11), 1201–1210.
McPartland, J. C., Webb, S. J., Keehn, B., & Dawson, G.
(2011) Patterns of visual attention to face and objects in autism spectrum disorder. Journal of Autism and Developmental Disorders, 41(2), 148–157.
Moser, T., & Martinsen, M. T.
(2010) The outdoor environment in Norwegian kindergartens as pedagogical space for toddlers’ play, learning and development. European Early Childhood Education Research Journal, 18(4), 457–471.
Mundy, P., Sigman, M., Ungerer, J., & Sherman, T.
(1986) Defining the social deficits of autism: the contribution of non-verbal communication measures. Journal of Child Psychology and Psychiatry, 27(5), 657–669.
Mynarikova, L.
(2012) Art-Based Program for Social and Emotional Development of Children. Online Submission.
QuiverVision
(2016) Quiver Augmented Reality used as art therapy for veterans. Retrieved from [URL]
Reynhout, G., & Carter, M.
(2006) Social Stories™ for children with disabilities. Journal of autism and developmental disorders, 36(4), 445–469.
Ryan, C., & Ni Charragain, C.
(2010) Teaching emotion recognition skills to children with autism. Journal of Autism and Developmental Disorders, 40(12), 1505–1511.
Shelton, B. E., & Hedley, N. R.
(2004) Exploring a cognitive basis for learning spatial relationships with augmented reality. Technology, Instruction, Cognition and Learning, 1(4), 323.
Sparrow, S. S.
(2011) Vineland adaptive behavior scales. In Encyclopedia of Clinical Neuropsychology (pp. 2618–2621). Springer New York.
Wraga, M., Thompson, W. L., Alpert, N. M., & Kosslyn, S. M.
(2003) Implicit transfer of motor strategies in mental rotation. Brain and Cognition, 52(2), 135–143.
Yuill, N., Strieth, S., Roake, C., Aspden, R., & Todd, B.
(2007) Brief Report: Designing a Playground for Children with Autistic Spectrum Disorders––Effects on Playful Peer Interactions. Journal of autism and developmental disorders, 37(6), 1192–1196.
Zünd, F., Ryffel, M., Magnenat, S., Marra, A., Nitti, M., Kapadia, M., … & Sumner, R. W.
(2015, November). Augmented creativity: bridging the real and virtual worlds to enhance creative play. In SIGGRAPH Asia 2015 Mobile Graphics and Interactive Applications (p. 21). ACM.
Cited by
Cited by 14 other publications
Bakır, Çiçek Nur, Sharif Omar Abbas, Egemen Sever, Aslıhan Özcan Morey, Herdem Aslan Genç & Tuba Mutluer
2023. Use of augmented reality in mental health-related conditions: A systematic review. DIGITAL HEALTH 9
2024. How flow and mindfulness interact with each other in mindfulness-based augmented reality mandala coloring activities. Frontiers in Psychology 14
Cheng, Yuchen & Loukia Bololia
2023. The Effects of Augmented Reality on Social Skills in Children with an Autism Diagnosis: A Preliminary Systematic Review. Journal of Autism and Developmental Disorders
2023. Leveraging Extended Reality for Autistic Individuals: A Scoping Review of Technical Features and Technology Affordances. IEEE Transactions on Learning Technologies 16:1 ► pp. 133 ff.
Hsu, Hsiu-Ting & I-Jui Lee
2020. Using Augmented Reality Technology with Serial Learning Framework to Develop a Serial Social Story Situation Board Game System for Children with Autism to Improve Social Situation Understanding and Social Reciprocity Skills. In Universal Access in Human-Computer Interaction. Applications and Practice [Lecture Notes in Computer Science, 12189], ► pp. 3 ff.
Huang, Yu-Chen & I-Jui Lee
2020. Using Augmented Reality and Concept Mapping to Improve Ability to Master Social Relationships and Social Reciprocity for Children with Autism Spectrum Disorder. In Universal Access in Human-Computer Interaction. Applications and Practice [Lecture Notes in Computer Science, 12189], ► pp. 19 ff.
Köse, Hasan & Nevin Güner-Yildiz
2021. Augmented reality (AR) as a learning material in special needs education. Education and Information Technologies 26:2 ► pp. 1921 ff.
Lee, I -Jui & Hsiu-Ting Hsu
2023. Applied the augmented reality technology combined with social stories strategies and computational thinking games to improve the social skills of children with ASD. Interactive Learning Environments► pp. 1 ff.
Lee, I-Jui
2020. How to use the Advantages of AR and VR Technique to Integrate Special Visual Training Strategies in Non-Verbal Communication Skills Training for Children with Autism. In Types of Nonverbal Communication [Working Title],
Lian, Xiaojie & Mohd Shahrizal Sunar
2021. Mobile Augmented Reality Technologies for Autism Spectrum Disorder Interventions: A Systematic Literature Review. Applied Sciences 11:10 ► pp. 4550 ff.
Lian, Xiaojie, Mohd Shahrizal Sunar, Qingqing Lian & Mohd Khalid Mokhtar
2023. Evaluating user interface of a mobile augmented reality coloring application for children with autism: An eye-tracking investigation. International Journal of Human-Computer Studies 178 ► pp. 103085 ff.
Neely, Leslie, Amarie Carnett, John Quarles, Se-Woong Park & Michelle Kelly
2023. Behavior Analytic Technologies Mediated via Augmented Reality for Autism: A Systematic Review. Journal of Developmental and Physical Disabilities
Newbutt, Nigel, Noah Glaser, Marc Sonley Francois, Matthew Schmidt & Sue Cobb
2023. How are Autistic People Involved in the Design of Extended Reality Technologies? A Systematic Literature Review. Journal of Autism and Developmental Disorders
Wang, Wei-Zhen & I-Jui Lee
2020. Social Intervention Strategy of Augmented Reality Combined with Theater-Based Games to Improve the Performance of Autistic Children in Symbolic Play and Social Skills. In HCI International 2020 – Late Breaking Papers: Universal Access and Inclusive Design [Lecture Notes in Computer Science, 12426], ► pp. 401 ff.
This list is based on CrossRef data as of 29 february 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.