Article published in:
Journal of Second Language Pronunciation
Vol. 4:1 (2018) ► pp. 129–153

Computer-assisted visual articulation feedback in L2 pronunciation instruction

A review

| University of Victoria | University of British Columbia
| University of British Columbia
| University of British Columbia | Haskins Laboratories
Language learning is a multimodal endeavor; to improve their pronunciation in a new language, learners access not only auditory information about speech sounds and patterns, but also visual information about articulatory movements and processes. With the development of new technologies in computer-assisted pronunciation training (CAPT) comes new possibilities for delivering feedback in both auditory and visual modalities. The present paper surveys the literature on computer-assisted visual articulation feedback, including direct feedback that provides visual models of articulation and indirect feedback that uses visualized acoustic information as a means to inform articulation instruction. Our focus is explicitly on segmental features rather than suprasegmental ones, with visual feedback conceived of as providing visualizations of articulatory configurations, movements, and processes. In addition to discussing types of visual articulation feedback, we also consider the criteria for effective delivery of feedback, and methods of evaluation.
Keywords: multimodality, visual feedback, articulation, CAPT, segmental features
Article outline
  • 1.Introduction
  • 2.Situating visual articulation feedback in the context of L2 learning
  • 3.Types of computer-assisted visual articulation feedback
    • 3.1Indirect feedback using visualized acoustic information
    • 3.2Direct feedback using visualized articulatory information
      • 3.2.1Ultrasound-based feedback
      • 3.2.2Feedback using intra-oral techniques
    • 3.3Simulation approaches
  • 4.Criteria for the effective use of visual articulation feedback
  • 5.Methods of evaluation and reporting
  • 6.Summary and conclusions
  • Acknowledgements
  • References
Published online: 31 May 2018
https://doi.org/10.1075/jslp.00006.bli
References
Abberton, E., & Fourcin, A. J.
(1975) Visual feedback and the acquisition of intonation. In E. H. Lenneberg & E. Lenneberg (Eds.). Foundations of language development: A multidisciplinary approach (Vol. 21, pp. 157–165). Paris: UNESCO.Google Scholar
Akahane-Yamada, R., McDermott, E., Adaichi, T., Kawahara, H., & Pruitt, J. S.
(1998) Computer-based second language production training by using spectrographic representation and HMM-based speech recognition scores. Paper presented at the 1998 International Conference on Spoken Language Processing, Sydney, Australia. Retrieved from http://​www​.mirlab​.org​/conference​_papers​/International_​Conference​/ICSLP%201998​/PDF​/AUTHOR​/SL980429​.PDF (16 December, 2015).
Aliaga-García, C., & Mora, J. C.
(2009) Assessing the effects of phonetic training on L2 sound perception and production. In M. A. Watkins, A. S. Rauber, & B. O. Baptista (Eds.), Recent research in second language phonetics/phonology: Perception and production (pp. 2–31). Newcastle upon Tyne: Cambridge Scholars.Google Scholar
Anderson, F.
(1960) An experimental pitch indicator for training deaf scholars. Journal of the Acoustical Society of America, 32(8), 1065–1074. CrossrefGoogle Scholar
Badin, P., Ben Youssef, A., Bailly, G., Elisei, F., & Hueber, T.
(2010) Visual articulatory feedback for phonetic correction in second language learning. Proceedings of the Workshop on Second Language Studies: Acquisition, Learning, Education, and Technology, 1–10.Google Scholar
Ballard, K. J., Smith, H. D., Paramatmuni, D., McCabe, P., Theodoros, D. G., & Murdoch, B. E.
(2012) Amount of kinematic feedback affects learning of speech motor skills. Motor Control, 161, 106–119. CrossrefGoogle Scholar
Bernhardt, B., Gick, B., Bacsfalvi, P., & Ashdown, J.
(2003) Speech habilitation of hard of hearing adolescents using electropalatography and ultrasound as evaluated by trained listeners. Clinical Linguistics & Phonetics, 17(3), 199–216. CrossrefGoogle Scholar
Bernhardt, B., Gick, B., Bacsfalvi, P., & Adler-Bock, M.
(2005) Ultrasound in speech therapy with adolescents and adults. Clinical Linguistics & Phonetics, 19(6/7), 605–617. CrossrefGoogle Scholar
Boersma, P.
(2001) Praat. A system for doing phonetics by computer. Glot International, 5(9/10), 341–345.Google Scholar
Bruce, C. J., Spittell, P. C., Montgomery, S. C., Bailey, K. R., Tajik, A. J., & Seward, J. B.
(2000) Ultrasound imager: Abdominal aortic aneurysm screening. Journal of the American Society of Echocardiography, 131, 674–679. CrossrefGoogle Scholar
Carey, M.
(2004) CALL visual feedback for pronunciation of vowels: Kay Sona-Match. CALICO Journal, 21(3), 571–601. CrossrefGoogle Scholar
Catford, J. C. & Pisoni, D. B.
(1970) Auditory versus articulatory training in exotic sounds. The Modern Language Journal, 54(7), 477–481.Google Scholar
Chun, D. M.
(1989) Teaching tone and intonation with microcomputers. CALICO Journal, 7(1), 21–46.Google Scholar
(1998) Signal analysis software for teaching discourse intonation. Language Learning & Technology, 2(1), 61–77.Google Scholar
(2002) Discourse intonation in L2: From theory and research to practice (Language Learning and Teaching 1). Amsterdam: John Benjamins. CrossrefGoogle Scholar
(2013) Computer-assisted pronunciation teaching. In C. Chapelle (ed). The encyclopedia of applied linguistics. Oxford: Blackwell.Google Scholar
Clarius
(2016) Wireless, handheld ultrasound for iOS and Android debuts. [Press release]. Retrieved from https://​www​.clarius​.me​/aium​-debut​-pr/.
Cleland, J., Scobbie, J. M., Nakai, S., & Wrench, A.
(2015) Helping children learn non-native articulations: the implications for ultrasound-based clinical intervention. Paper presented at the 2015 International Conference of Phonetic Sciences, Glasgow, Scotland. Retrieved from http://​www​.icphs2015​.info​/pdfs​/Papers​/ICPHS0698​.pdf (12 August, 2015).
Cook, V.
(Ed.) (1986) Experimental approaches to second language learning. Oxford: Pergamon.Google Scholar
de Bot, C. L. J.
(1980) The role of feedback and feedforward in the teaching of pronunciation. System, 81, 35–45. CrossrefGoogle Scholar
Demenko, G., Wagner, A., & Cylwik, N.
(2010) The use of speech technology in foreign language pronunciation training. Archives of Acoustics, 35(3), 309–329. CrossrefGoogle Scholar
Dowd, A., Smith, J., & Wolfe, J.
(1997) Learning to pronounce vowel sounds in a foreign language using acoustic measurements of the vocal tract as feedback in real time. Language and Speech, 41(1), 1–20. CrossrefGoogle Scholar
Engwall, O.
(2012) Analysis of and feedback on phonetic features in pronunciation training with a virtual teacher. Computer Assisted Language Learning, 25(1), 37–64. CrossrefGoogle Scholar
Gick, B., Bernhardt, B., Bacsfalvi, P., & Wilson, I.
(2008) Ultrasound imaging applications in second language acquisition. In J. G. Hansen Edwards & M. L. Zampini (Eds.), Phonology and second language acquisition (pp. 309–322). Amsterdam: John Benjamins. CrossrefGoogle Scholar
Hardison, D. M.
(2004) Generalization of computer-assisted prosody training: Quantitative and qualitative findings. Language Learning & Technology, 81, 34–52.Google Scholar
Hincks, R.
(2015) Technology and learning pronunciation. In M. Reed & J. M. Levis (Eds.), The handbook of English pronunciation (pp. 505–519). Hoboken, NJ: Wiley and Sons. CrossrefGoogle Scholar
Jenson, P. G., & Westermeier, F. X.
(1968) The effect of visual feedback on pronunciation in foreign language learning. Retrieved from http://​files​.eric​.ed​.gov​/fulltext​/ED015689​.pdf (29 August, 2015).
Kalikow, D. N., & Swets, J. A.
(1972) Experiments with computer-controlled displays in second-language learning. IEEE Transactions on Audio and Electroacoustics, AU-20(1), 23–28. CrossrefGoogle Scholar
Kartushina, N., Hervais-Adelman, A., Frauenfelder, U. H., & Golestani, N.
(2015) The effect of phonetic production training with visual feedback on the perception and production of foreign speech sounds. Journal of the Acoustical Society of America, 138(2), 817–832. CrossrefGoogle Scholar
Katz, W., Campbell, T., Wang, J., Farrar, E., Eubanks, J. C., Balasubramanian, A., Prabhakaran, B., & Rennaker, R.
(2014) Opti-Speech: A real-time, 3D visual feedback system for speech training. In Proceedings of Interspeech 2014, Singapore (pp.1174–1178). Retrieved from https://​www​.utdallas​.edu​/~wangjun​/paper​/Interspeech14​_opti​-speech​.pdf (22 January, 2016).
Katz, W. F., & Mehta, S.
(2015) Visual feedback of tongue movement for novel speech sound learning. Frontiers in Human Neuroscience, 91, 612. Crossref.Google Scholar
Kelsey, C. A., Minifie, F. D., & Hixon, T. J.
(1969) Applications of ultrasound in speech research. Journal of Speech, Language, and Hearing Research, 12(3), 564–575 CrossrefGoogle Scholar
Lambacher, S.
(1999) A CALL tool for improving second language acquisition of English consonants by Japanese learners. Computer Assisted Language Learning, 12(2), 137–156. CrossrefGoogle Scholar
Lee, J., Jang, J., & Plonksy, L.
(2015) The effectiveness of second language pronunciation instruction: A meta-analysis. Applied Linguistics, 36(3), 345–355. CrossrefGoogle Scholar
Léon, P. R., & Martin, P.
(1972) Applied linguistics and the teaching of intonation. The Modern Language Journal, 56(3), 139–144. CrossrefGoogle Scholar
Levis, J. M. & Pickering, L.
(2004) Teaching intonation in discourse using speech visualization technology. System, 32(4), 505–524. CrossrefGoogle Scholar
Levitt, J. S., & Katz, W. F.
(2007) Augmented visual feedback in second language learning: training Japanese post-alveolar flaps to American English speakers. Journal of the Acoustical Society of America, 122(5), 2996. CrossrefGoogle Scholar
Massaro, D. W., & Light, J.
(2003) Read my tongue movements: bimodal learning to perceive and produce non-native speech /r/ and /l/. Proceedings of the 8th European Conference on Speech Communication and Technology.Google Scholar
Mattheyses, W. & Verhelst, W.
(2015) Audiovisual speech synthesis: An overview of the state- of-the-art. Speech Communication, 661, 182–217. CrossrefGoogle Scholar
Moisik, S. R., Esling, J. H., Bird, S., & Lin, H.
(2011) Evaluating laryngeal ultrasound to study larynx state and height. In W. S. Lee & E. Zee (Eds.), Proceedings of the 17th International Congress of Phonetic Sciences Hong Kong (pp. 136–139).Google Scholar
Molholt, G.
(1988) Computer-assisted instruction in pronunciation for Chinese speakers of American English. TESOL Quarterly, 22(1), 91–111. CrossrefGoogle Scholar
(1990) Spectrographic analysis and patterns in pronunciation. Computers and the Humanities, 24(1/2), 81–92. CrossrefGoogle Scholar
Navarra, J., & Soto-Faraco, S.
(2007) Hearing lips in a second language: Visual articulatory information enables the perception of second language sounds. Psychological Research 711, 4–12. CrossrefGoogle Scholar
Neri, A., Cucchiarini, C., Strik, H., & Boves, L.
(2002) The pedagogy-technology interface in computer-assisted pronunciation training. Computer-Assisted Language Learning, 21(5), 393–408. CrossrefGoogle Scholar
Noguchi, M., Yamane, N., Tsuda, A., Kazama, M., Kim, B., & Gick, B.
(2015) Towards protocols for L2 pronunciation training using ultrasound imaging. Poster presentation at the 7th annual Pronunciation in Second Language Learning and Teaching (PSLLT) Conference. Dallas, TX, October 2015.Google Scholar
Olson, D. J.
(2014a) Phonetics and technology in the classroom: A practical approach to using speech analysis software in second-language pronunciation instruction. Hispania, 97(1), 47–68. CrossrefGoogle Scholar
(2014b) Benefits of visual feedback on segmental production in the L2 classroom. Language Learning and Technology, 18(3), 173–192.Google Scholar
Öster, A. -M.
(1997) Auditory and visual feedback in spoken L2 teaching. Reports from the Department of Phonetics, Umeå University (PHONUM), 41, 145–148.Google Scholar
Ouni, S.
(2014) Tongue control and its implication in pronunciation training. Computer Assisted Language Learning, 27(5), 439–453. CrossrefGoogle Scholar
Patten, I., & Edmonds, L. A.
(2015) Effect of training Japanese L1 speakers in the production of American English /r/ using spectrographic visual feedback. Computer Assisted Language Learning, 28(3), 241–259. CrossrefGoogle Scholar
Pillot-Loiseau, C., Kamiyama, T., & Kocjančič Antolík, T.
(2015) French /y/-/u/ contrast in Japanese learners with/without ultrasound feedback: vowels, non-words and words. Paper presented at the 2015 International Conference of Phonetic Sciences, Glasgow, Scotland. Retrieved on from http://​www​.icphs2015​.info​/pdfs​/Papers​/ICPHS0485​.pdf (12 August, 2015).
Quintana-Lara, M.
(2014) Effect of acoustic spectrographic instruction on production of English /i/ and /ɪ/ by Spanish pre-service English teachers. Computer Assisted Language Learning, 27(3), 207–227. CrossrefGoogle Scholar
R Core Team
(2014) R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.Google Scholar
Saito, K.
(2007) The influence of explicit pronunciation instruction on pronunciation in EFL settings: the case of English vowels and Japanese learners of English. The Linguistics Journal, 3(3), 16–40.Google Scholar
Schwartz, B.
(1993) On explicit and negative data effecting and affecting competence and linguistic behavior. Studies in Second Language Acquisition, 151, 147–163. CrossrefGoogle Scholar
Stone, M.
(2005) Preface to the special issue on ultrasound imaging of the tongue. Clinical Linguistics & Phonetics, 19(6–7), 453–454. CrossrefGoogle Scholar
Suemitsu, A., Dang, J., Ito, T., & Tiede, M.
(2015) A real-time articulatory visual feedback approach with target presentation for second language pronunciation learning. Journal of the Acoustical Society of America, 138(4), EL382–EL387. CrossrefGoogle Scholar
Tateishi, M., & Winters, S.
(2013) Does ultrasound training lead to improved perception of a non-native sound contrast? Evidence from Japanese learners of English. Paper presented at the 2013 meeting of the Canadian Linguistic Association, Victoria, BC, Canada. Retrieved from http://​homes​.chass​.utoronto​.ca​/~cla​-acl​/actes2013​/Tateishi​_and​_Winters​-2013​.pdf (12 August, 2015).
Thomson, R., & Derwing, T.
(2014) The effectiveness of L2 pronunciation instruction: A narrative review. Applied Linguistics, 36(3): 326–344. CrossrefGoogle Scholar
Tilsen, S., Das, D., & McKee, B.
(2015) Real-time articulatory biofeedback with electromagnetic articulography. Linguistics Vanguard, 1(1), 39–55. Crossref.Google Scholar
Truscott, J.
(2007) The effect of error correction on learners’ ability to write accurately. System, 161: 255–272.Google Scholar
Tsui, H. M.
(2012) Ultrasound speech training for Japanese adults learning English as a second language (Unpublished MSc thesis). University of British Columbia.Google Scholar
Vardanian, R. M.
(1964) Teaching English intonation through oscilloscope displays. Language Learning, 14(3–4), 109–117. CrossrefGoogle Scholar
Wilson, I.
(2014) Using ultrasound for teaching and researching articulation. Acoustical Science and Technology, 35(6), 285–289. CrossrefGoogle Scholar
Wilson, I., & Gick, B.
(2006) Ultrasound technology and second language acquisition research. In M. Grantham O’Brien, C. Shea, & J. Archibald (Eds.), Proceedings of the 8th Generative Approaches to Second Language Acquisition Conference (GASLA 2006) (pp. 148–152). Somerville, MA: Cascadilla Proceedings Project.Google Scholar
Wojtczak, J., & Bonadonna, P.
(2013) Pocket mobile smartphone system for the point-of-care submandibular ultrasonography. The American Journal of Emergency Medicine, 311, 573–577. CrossrefGoogle Scholar
Wu, Y., Gendrot, C., Hallé, P., & Adda-Decker, M.
(2015) On improving the pronunciation of French /r/ in Chinese learners by using real-time ultrasound visualization. Paper presented at the 2015 International Conference of Phonetic Sciences, Glasgow, Scotland. Retrieved from http://​www​.icphs2015​.info​/pdfs​/Papers​/ICPHS0786​.pdf (12 August, 2015).
Cited by

Cited by 18 other publications

Angelovska, Tanja
2021.  In English Pronunciation Instruction [AILA Applied Linguistics Series, 19], ► pp. 40 ff. Crossref logo
Barbier, Guillaume, Ryme Merzouki, Mathilde Bal, Shari R. Baum & Douglas M. Shiller
2021. Visual feedback of the tongue influences speech adaptation to a physical modification of the oral cavity. The Journal of the Acoustical Society of America 150:2 ► pp. 718 ff. Crossref logo
Bu, Yaohua, Tianyi Ma, Weijun Li, Hang Zhou, Jia Jia, Shengqi Chen, Kaiyuan Xu, Dachuan Shi, Haozhe Wu, Zhihan Yang, Kun Li, Zhiyong Wu, Yuanchun Shi, Xiaobo Lu & Ziwei Liu
2021.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, ► pp. 1 ff. Crossref logo
Chun, Dorothy M.
2019.  In The Encyclopedia of Applied Linguistics, ► pp. 1 ff. Crossref logo
Cibelli, Emily
2020. Training Non-Native Consonant Production with Perceptual and Articulatory Cues. Phonetica 77:1 ► pp. 1 ff. Crossref logo
Ciriza, María del Puy & Ahmed Rivera-Campos
2020. Teaching the Spanish trill to L1 English speakers using ultrasound instruction: a preliminary study on pronunciation pedagogy. Journal of Spanish Language Teaching 7:1 ► pp. 20 ff. Crossref logo
Demir, Yusuf & Galip Kartal
2022. MAPPING RESEARCH ON L2 PRONUNCIATION. Studies in Second Language Acquisition ► pp. 1 ff. Crossref logo
Hardison, Debra M.
2019.  In Encyclopedia of Educational Innovation, ► pp. 1 ff. Crossref logo
Hardison, Debra M.
2020.  In Encyclopedia of Educational Innovation, ► pp. 1 ff. Crossref logo
Johnson, Khia Anne, Gloria Madeleine Mellesmoen, Roger Yu-Hsiang Lo & Bryan Gick
2018. Prior Pronunciation Knowledge Bootstraps Word Learning. Frontiers in Communication 3 Crossref logo
Klaus, Adam, Daniel R. Lametti, Douglas M. Shiller & Tara McAllister
2019. Can perceptual training alter the effect of visual biofeedback in speech-motor learning?. The Journal of the Acoustical Society of America 145:2 ► pp. 805 ff. Crossref logo
Levis, John M.
2018. Plenary talk. Journal of Second Language Pronunciation 4:2 ► pp. 260 ff. Crossref logo
Levis, John M.
2018. Technology and second language pronunciation. Journal of Second Language Pronunciation 4:2 ► pp. 173 ff. Crossref logo
Li, Joanne Jingwen, Samantha Ayala, Daphna Harel, Douglas M. Shiller & Tara McAllister
2019. Individual predictors of response to biofeedback training for second-language production. The Journal of the Acoustical Society of America 146:6 ► pp. 4625 ff. Crossref logo
Lima, Edna F.
2020. The Supra Tutor. Journal of Second Language Pronunciation 6:1 ► pp. 39 ff. Crossref logo
Mozaffari, M. Hamed & Won-Sook Lee
2021.  In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), ► pp. 3043 ff. Crossref logo
Roon, Kevin D., Jaekoo Kang & D.H. Whalen
2020. Effects of Ultrasound Familiarization on Production and Perception of Nonnative Contrasts. Phonetica 77:5 ► pp. 350 ff. Crossref logo
Terband, Hayo, Aravind Namasivayam, Edwin Maas, Frits van Brenk, Marja-Liisa Mailend, Sanne Diepeveen, Pascal van Lieshout & Ben Maassen
2019. Assessment of Childhood Apraxia of Speech: A Review/Tutorial of Objective Measurement Techniques. Journal of Speech, Language, and Hearing Research 62:8S ► pp. 2999 ff. Crossref logo

This list is based on CrossRef data as of 23 april 2022. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.