Cover not available
Article published In:
Journal of Second Language Pronunciation
Vol. 4:1 (2018) ► pp.129–153

Computer-assisted visual articulation feedback in L2 pronunciation instruction

A review

| University of Victoria | University of British Columbia
| University of British Columbia
| University of British Columbia | Haskins Laboratories
Language learning is a multimodal endeavor; to improve their pronunciation in a new language, learners access not only auditory information about speech sounds and patterns, but also visual information about articulatory movements and processes. With the development of new technologies in computer-assisted pronunciation training (CAPT) comes new possibilities for delivering feedback in both auditory and visual modalities. The present paper surveys the literature on computer-assisted visual articulation feedback, including direct feedback that provides visual models of articulation and indirect feedback that uses visualized acoustic information as a means to inform articulation instruction. Our focus is explicitly on segmental features rather than suprasegmental ones, with visual feedback conceived of as providing visualizations of articulatory configurations, movements, and processes. In addition to discussing types of visual articulation feedback, we also consider the criteria for effective delivery of feedback, and methods of evaluation.
Keywords: multimodality, visual feedback, articulation, CAPT, segmental features
Article outline
  • 1.Introduction
  • 2.Situating visual articulation feedback in the context of L2 learning
  • 3.Types of computer-assisted visual articulation feedback
    • 3.1Indirect feedback using visualized acoustic information
    • 3.2Direct feedback using visualized articulatory information
      • 3.2.1Ultrasound-based feedback
      • 3.2.2Feedback using intra-oral techniques
    • 3.3Simulation approaches
  • 4.Criteria for the effective use of visual articulation feedback
  • 5.Methods of evaluation and reporting
  • 6.Summary and conclusions
  • Acknowledgements
  • References
Published online: 31 May 2018
DOI logo
https://doi.org/10.1075/jslp.00006.bli
References (67)
References
Abberton, E., & Fourcin, A. J. (1975). Visual feedback and the acquisition of intonation. In E. H. Lenneberg & E. Lenneberg (Eds.). Foundations of language development: A multidisciplinary approach (Vol. 21, pp. 157–165). Paris: UNESCO.Google Scholar
Akahane-Yamada, R., McDermott, E., Adaichi, T., Kawahara, H., & Pruitt, J. S. (1998). Computer-based second language production training by using spectrographic representation and HMM-based speech recognition scores. Paper presented at the 1998 International Conference on Spoken Language Processing, Sydney, Australia. Retrieved from <[URL]> (16 December, 2015).
Aliaga-García, C., & Mora, J. C. (2009). Assessing the effects of phonetic training on L2 sound perception and production. In M. A. Watkins, A. S. Rauber, & B. O. Baptista (Eds.), Recent research in second language phonetics/phonology: Perception and production (pp. 2–31). Newcastle upon Tyne: Cambridge Scholars.Google Scholar
Anderson, F. (1960). An experimental pitch indicator for training deaf scholars. Journal of the Acoustical Society of America, 32(8), 1065–1074. DOI logoGoogle Scholar
Badin, P., Ben Youssef, A., Bailly, G., Elisei, F., & Hueber, T. (2010). Visual articulatory feedback for phonetic correction in second language learning. Proceedings of the Workshop on Second Language Studies: Acquisition, Learning, Education, and Technology, 1–10.Google Scholar
Ballard, K. J., Smith, H. D., Paramatmuni, D., McCabe, P., Theodoros, D. G., & Murdoch, B. E. (2012). Amount of kinematic feedback affects learning of speech motor skills. Motor Control, 161, 106–119. DOI logoGoogle Scholar
Bernhardt, B., Gick, B., Bacsfalvi, P., & Ashdown, J. (2003). Speech habilitation of hard of hearing adolescents using electropalatography and ultrasound as evaluated by trained listeners. Clinical Linguistics & Phonetics, 17(3), 199–216. DOI logoGoogle Scholar
Bernhardt, B., Gick, B., Bacsfalvi, P., & Adler-Bock, M. (2005). Ultrasound in speech therapy with adolescents and adults. Clinical Linguistics & Phonetics, 19(6/7), 605–617. DOI logoGoogle Scholar
Boersma, P. (2001). Praat. A system for doing phonetics by computer. Glot International, 5(9/10), 341–345.Google Scholar
Bruce, C. J., Spittell, P. C., Montgomery, S. C., Bailey, K. R., Tajik, A. J., & Seward, J. B. (2000). Ultrasound imager: Abdominal aortic aneurysm screening. Journal of the American Society of Echocardiography, 131, 674–679. DOI logoGoogle Scholar
Carey, M. (2004). CALL visual feedback for pronunciation of vowels: Kay Sona-Match. CALICO Journal, 21(3), 571–601. DOI logoGoogle Scholar
Catford, J. C. & Pisoni, D. B. (1970). Auditory versus articulatory training in exotic sounds. The Modern Language Journal, 54(7), 477–481.Google Scholar
Chun, D. M. (1989). Teaching tone and intonation with microcomputers. CALICO Journal, 7(1), 21–46.Google Scholar
(1998). Signal analysis software for teaching discourse intonation. Language Learning & Technology, 2(1), 61–77.Google Scholar
(2002). Discourse intonation in L2: From theory and research to practice (Language Learning and Teaching 1). Amsterdam: John Benjamins. DOI logoGoogle Scholar
(2013). Computer-assisted pronunciation teaching. In C. Chapelle (ed). The encyclopedia of applied linguistics. Oxford: Blackwell.Google Scholar
Clarius. (2016). Wireless, handheld ultrasound for iOS and Android debuts. [Press release]. Retrieved from <[URL]>.
Cleland, J., Scobbie, J. M., Nakai, S., & Wrench, A. (2015). Helping children learn non-native articulations: the implications for ultrasound-based clinical intervention. Paper presented at the 2015 International Conference of Phonetic Sciences, Glasgow, Scotland. Retrieved from <[URL]> (12 August, 2015).
Cook, V. (Ed.). (1986). Experimental approaches to second language learning. Oxford: Pergamon.Google Scholar
de Bot, C. L. J. (1980). The role of feedback and feedforward in the teaching of pronunciation. System, 81, 35–45. DOI logoGoogle Scholar
Demenko, G., Wagner, A., & Cylwik, N. (2010). The use of speech technology in foreign language pronunciation training. Archives of Acoustics, 35(3), 309–329. DOI logoGoogle Scholar
Dowd, A., Smith, J., & Wolfe, J. (1997). Learning to pronounce vowel sounds in a foreign language using acoustic measurements of the vocal tract as feedback in real time. Language and Speech, 41(1), 1–20. DOI logoGoogle Scholar
Engwall, O. (2012). Analysis of and feedback on phonetic features in pronunciation training with a virtual teacher. Computer Assisted Language Learning, 25(1), 37–64. DOI logoGoogle Scholar
Gick, B., Bernhardt, B., Bacsfalvi, P., & Wilson, I. (2008). Ultrasound imaging applications in second language acquisition. In J. G. Hansen Edwards & M. L. Zampini (Eds.), Phonology and second language acquisition (pp. 309–322). Amsterdam: John Benjamins. DOI logoGoogle Scholar
Hardison, D. M. (2004). Generalization of computer-assisted prosody training: Quantitative and qualitative findings. Language Learning & Technology, 81, 34–52.Google Scholar
Hincks, R. (2015). Technology and learning pronunciation. In M. Reed & J. M. Levis (Eds.), The handbook of English pronunciation (pp. 505–519). Hoboken, NJ: Wiley and Sons. DOI logoGoogle Scholar
Jenson, P. G., & Westermeier, F. X. (1968). The effect of visual feedback on pronunciation in foreign language learning. Retrieved from <[URL]> (29 August, 2015).
Kalikow, D. N., & Swets, J. A. (1972). Experiments with computer-controlled displays in second-language learning. IEEE Transactions on Audio and Electroacoustics, AU-20(1), 23–28. DOI logoGoogle Scholar
Kartushina, N., Hervais-Adelman, A., Frauenfelder, U. H., & Golestani, N. (2015). The effect of phonetic production training with visual feedback on the perception and production of foreign speech sounds. Journal of the Acoustical Society of America, 138(2), 817–832. DOI logoGoogle Scholar
Katz, W., Campbell, T., Wang, J., Farrar, E., Eubanks, J. C., Balasubramanian, A., Prabhakaran, B., & Rennaker, R. (2014). Opti-Speech: A real-time, 3D visual feedback system for speech training. In Proceedings of Interspeech 2014, Singapore (pp.1174–1178). Retrieved from <[URL]> (22 January, 2016).
Katz, W. F., & Mehta, S. (2015). Visual feedback of tongue movement for novel speech sound learning. Frontiers in Human Neuroscience, 91, 612. DOI logo.Google Scholar
Kelsey, C. A., Minifie, F. D., & Hixon, T. J. (1969). Applications of ultrasound in speech research. Journal of Speech, Language, and Hearing Research, 12(3), 564–575 DOI logoGoogle Scholar
Lambacher, S. (1999). A CALL tool for improving second language acquisition of English consonants by Japanese learners. Computer Assisted Language Learning, 12(2), 137–156. DOI logoGoogle Scholar
Lee, J., Jang, J., & Plonksy, L. (2015). The effectiveness of second language pronunciation instruction: A meta-analysis. Applied Linguistics, 36(3), 345–355. DOI logoGoogle Scholar
Léon, P. R., & Martin, P. (1972). Applied linguistics and the teaching of intonation. The Modern Language Journal, 56(3), 139–144. DOI logoGoogle Scholar
Levis, J. M. & Pickering, L. (2004). Teaching intonation in discourse using speech visualization technology. System, 32(4), 505–524. DOI logoGoogle Scholar
Levitt, J. S., & Katz, W. F. (2007). Augmented visual feedback in second language learning: training Japanese post-alveolar flaps to American English speakers. Journal of the Acoustical Society of America, 122(5), 2996. DOI logoGoogle Scholar
Massaro, D. W., & Light, J. (2003). Read my tongue movements: bimodal learning to perceive and produce non-native speech /r/ and /l/. Proceedings of the 8th European Conference on Speech Communication and Technology.Google Scholar
Mattheyses, W. & Verhelst, W. (2015). Audiovisual speech synthesis: An overview of the state- of-the-art. Speech Communication, 661, 182–217. DOI logoGoogle Scholar
Moisik, S. R., Esling, J. H., Bird, S., & Lin, H. (2011). Evaluating laryngeal ultrasound to study larynx state and height. In W. S. Lee & E. Zee (Eds.), Proceedings of the 17th International Congress of Phonetic Sciences Hong Kong (pp. 136–139).Google Scholar
Molholt, G. (1988). Computer-assisted instruction in pronunciation for Chinese speakers of American English. TESOL Quarterly, 22(1), 91–111. DOI logoGoogle Scholar
(1990). Spectrographic analysis and patterns in pronunciation. Computers and the Humanities, 24(1/2), 81–92. DOI logoGoogle Scholar
Navarra, J., & Soto-Faraco, S. (2007). Hearing lips in a second language: Visual articulatory information enables the perception of second language sounds. Psychological Research 711, 4–12. DOI logoGoogle Scholar
Neri, A., Cucchiarini, C., Strik, H., & Boves, L. (2002). The pedagogy-technology interface in computer-assisted pronunciation training. Computer-Assisted Language Learning, 21(5), 393–408. DOI logoGoogle Scholar
Noguchi, M., Yamane, N., Tsuda, A., Kazama, M., Kim, B., & Gick, B. (2015). Towards protocols for L2 pronunciation training using ultrasound imaging. Poster presentation at the 7th annual Pronunciation in Second Language Learning and Teaching (PSLLT) Conference. Dallas, TX, October 2015.Google Scholar
Olson, D. J. (2014a). Phonetics and technology in the classroom: A practical approach to using speech analysis software in second-language pronunciation instruction. Hispania, 97(1), 47–68. DOI logoGoogle Scholar
(2014b). Benefits of visual feedback on segmental production in the L2 classroom. Language Learning and Technology, 18(3), 173–192.Google Scholar
Öster, A. -M. (1997). Auditory and visual feedback in spoken L2 teaching. Reports from the Department of Phonetics, Umeå University (PHONUM), 41, 145–148.Google Scholar
Ouni, S. (2014). Tongue control and its implication in pronunciation training. Computer Assisted Language Learning, 27(5), 439–453. DOI logoGoogle Scholar
Patten, I., & Edmonds, L. A. (2015). Effect of training Japanese L1 speakers in the production of American English /r/ using spectrographic visual feedback. Computer Assisted Language Learning, 28(3), 241–259. DOI logoGoogle Scholar
Pillot-Loiseau, C., Kamiyama, T., & Kocjančič Antolík, T. (2015). French /y/-/u/ contrast in Japanese learners with/without ultrasound feedback: vowels, non-words and words. Paper presented at the 2015 International Conference of Phonetic Sciences, Glasgow, Scotland. Retrieved on from <[URL]> (12 August, 2015).
Quintana-Lara, M. (2014). Effect of acoustic spectrographic instruction on production of English /i/ and /ɪ/ by Spanish pre-service English teachers. Computer Assisted Language Learning, 27(3), 207–227. DOI logoGoogle Scholar
R Core Team (2014). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.Google Scholar
Saito, K. (2007). The influence of explicit pronunciation instruction on pronunciation in EFL settings: the case of English vowels and Japanese learners of English. The Linguistics Journal, 3(3), 16–40.Google Scholar
Schwartz, B. (1993). On explicit and negative data effecting and affecting competence and linguistic behavior. Studies in Second Language Acquisition, 151, 147–163. DOI logoGoogle Scholar
Stone, M. (2005). Preface to the special issue on ultrasound imaging of the tongue. Clinical Linguistics & Phonetics, 19(6–7), 453–454. DOI logoGoogle Scholar
Suemitsu, A., Dang, J., Ito, T., & Tiede, M. (2015). A real-time articulatory visual feedback approach with target presentation for second language pronunciation learning. Journal of the Acoustical Society of America, 138(4), EL382–EL387. DOI logoGoogle Scholar
Tateishi, M., & Winters, S. (2013). Does ultrasound training lead to improved perception of a non-native sound contrast? Evidence from Japanese learners of English. Paper presented at the 2013 meeting of the Canadian Linguistic Association, Victoria, BC, Canada. Retrieved from <[URL]> (12 August, 2015).
Thomson, R., & Derwing, T. (2014). The effectiveness of L2 pronunciation instruction: A narrative review. Applied Linguistics, 36(3): 326–344. DOI logoGoogle Scholar
Tilsen, S., Das, D., & McKee, B. (2015). Real-time articulatory biofeedback with electromagnetic articulography. Linguistics Vanguard, 1(1), 39–55. DOI logo.Google Scholar
Truscott, J. (2007). The effect of error correction on learners’ ability to write accurately. System, 161: 255–272.Google Scholar
Tsui, H. M. (2012). Ultrasound speech training for Japanese adults learning English as a second language (Unpublished MSc thesis). University of British Columbia.Google Scholar
Vardanian, R. M. (1964). Teaching English intonation through oscilloscope displays. Language Learning, 14(3–4), 109–117. DOI logoGoogle Scholar
Wilson, I. (2014). Using ultrasound for teaching and researching articulation. Acoustical Science and Technology, 35(6), 285–289. DOI logoGoogle Scholar
Wilson, I., & Gick, B. (2006). Ultrasound technology and second language acquisition research. In M. Grantham O’Brien, C. Shea, & J. Archibald (Eds.), Proceedings of the 8th Generative Approaches to Second Language Acquisition Conference (GASLA 2006) (pp. 148–152). Somerville, MA: Cascadilla Proceedings Project.Google Scholar
Wojtczak, J., & Bonadonna, P. (2013). Pocket mobile smartphone system for the point-of-care submandibular ultrasonography. The American Journal of Emergency Medicine, 311, 573–577. DOI logoGoogle Scholar
Wu, Y., Gendrot, C., Hallé, P., & Adda-Decker, M. (2015). On improving the pronunciation of French /r/ in Chinese learners by using real-time ultrasound visualization. Paper presented at the 2015 International Conference of Phonetic Sciences, Glasgow, Scotland. Retrieved from <[URL]> (12 August, 2015).
Cited by (26)

Cited by 26 other publications

Kocjančič, Tanja, Tomáš Bořil & Susanna Hofmann
2024. Acoustic and Articulatory Visual Feedback in Classroom L2 Vowel Remediation. Language and Speech DOI logo
Shahid, Muhammad Suhaib, Andrew P French, Michel F Valstar & Gleb E Yakubov
2024. Research in methodologies for modelling the oral cavity. Biomedical Physics & Engineering Express 10:3 ► pp. 032001 ff. DOI logo
Sun, Yan
2024. The Application of Intelligent Speech Recognition in the Teaching of Spoken English in Colleges and Universities. Applied Mathematics and Nonlinear Sciences 9:1 DOI logo
CENGİZ, Behice Ceyda
2023. Computer-Assisted Pronunciation Teaching: An Analysis of Empirical Research. Participatory Educational Research 10:3 ► pp. 72 ff. DOI logo
Feng, Qiang & M Grazia Busà
2023. acquisition of Italian /r-l/ contrast by L1-Chinese learners. Journal of Monolingual and Bilingual Speech 4:3 DOI logo
Schuhmann, Katharina S., Sarah Schaech & Cain Catto
2023. Multisensory Pronunciation Training in a Video Conference-Based Foreign Language Classroom. In Advances in Second/Foreign Language Acquisition, ► pp. 19 ff. DOI logo
Chun, Dorothy M. & Yan Jiang
2022. Using Technology to Explore L2 Pronunciation. In Second Language Pronunciation, ► pp. 129 ff. DOI logo
Demir, Yusuf & Galip Kartal
2022. MAPPING RESEARCH ON L2 PRONUNCIATION. Studies in Second Language Acquisition 44:5 ► pp. 1211 ff. DOI logo
Kühnert, Barbara & Claire Pillot-Loiseau
2022. Teaching Pronunciation with Direct Visual Articulatory Feedback: Pedagogical Considerations for the Use of Ultrasound in the Classroom. Recherches anglaises et nord-américaines :55 ► pp. 9 ff. DOI logo
Angelovska, Tanja
2021. Chapter 3. When a psycholinguist enters the multilingual classroom. In English Pronunciation Instruction [AILA Applied Linguistics Series, 19], ► pp. 40 ff. DOI logo
Barbier, Guillaume, Ryme Merzouki, Mathilde Bal, Shari R. Baum & Douglas M. Shiller
2021. Visual feedback of the tongue influences speech adaptation to a physical modification of the oral cavity. The Journal of the Acoustical Society of America 150:2 ► pp. 718 ff. DOI logo
Bu, Yaohua, Tianyi Ma, Weijun Li, Hang Zhou, Jia Jia, Shengqi Chen, Kaiyuan Xu, Dachuan Shi, Haozhe Wu, Zhihan Yang, Kun Li, Zhiyong Wu, Yuanchun Shi, Xiaobo Lu & Ziwei Liu
2021. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, ► pp. 1 ff. DOI logo
Mozaffari, M. Hamed & Won-Sook Lee
2021. 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), ► pp. 3043 ff. DOI logo
Cibelli, Emily
2020. Training Non-Native Consonant Production with Perceptual and Articulatory Cues. Phonetica 77:1 ► pp. 1 ff. DOI logo
Ciriza, María del Puy & Ahmed Rivera-Campos
2020. Teaching the Spanish trill to L1 English speakers using ultrasound instruction: a preliminary study on pronunciation pedagogy. Journal of Spanish Language Teaching 7:1 ► pp. 20 ff. DOI logo
Lima, Edna F.
2020. The Supra Tutor. Journal of Second Language Pronunciation 6:1 ► pp. 39 ff. DOI logo
Roon, Kevin D., Jaekoo Kang & D.H. Whalen
2020. Effects of Ultrasound Familiarization on Production and Perception of Nonnative Contrasts. Phonetica 77:5 ► pp. 350 ff. DOI logo
Chun, Dorothy M.
2019. Computer‐Assisted Pronunciation Teaching. In The Encyclopedia of Applied Linguistics, ► pp. 1 ff. DOI logo
Hardison, Debra M.
2019. Technology-Based Communication Success for Second-Language Learners. In Encyclopedia of Educational Innovation, ► pp. 1 ff. DOI logo
Hardison, Debra M.
2020. Technology-Based Communication Success for Second-Language Learners. In Encyclopedia of Educational Innovation, ► pp. 1 ff. DOI logo
Klaus, Adam, Daniel R. Lametti, Douglas M. Shiller & Tara McAllister
2019. Can perceptual training alter the effect of visual biofeedback in speech-motor learning?. The Journal of the Acoustical Society of America 145:2 ► pp. 805 ff. DOI logo
Li, Joanne Jingwen, Samantha Ayala, Daphna Harel, Douglas M. Shiller & Tara McAllister
2019. Individual predictors of response to biofeedback training for second-language production. The Journal of the Acoustical Society of America 146:6 ► pp. 4625 ff. DOI logo
Terband, Hayo, Aravind Namasivayam, Edwin Maas, Frits van Brenk, Marja-Liisa Mailend, Sanne Diepeveen, Pascal van Lieshout & Ben Maassen
2019. Assessment of Childhood Apraxia of Speech: A Review/Tutorial of Objective Measurement Techniques. Journal of Speech, Language, and Hearing Research 62:8S ► pp. 2999 ff. DOI logo
Johnson, Khia Anne, Gloria Madeleine Mellesmoen, Roger Yu-Hsiang Lo & Bryan Gick
2018. Prior Pronunciation Knowledge Bootstraps Word Learning. Frontiers in Communication 3 DOI logo
Levis, John M.
2018. Plenary talk. Journal of Second Language Pronunciation 4:2 ► pp. 260 ff. DOI logo
Levis, John M.
2018. Technology and second language pronunciation. Journal of Second Language Pronunciation 4:2 ► pp. 173 ff. DOI logo

This list is based on CrossRef data as of 15 september 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.