Cover not available
Article published In:
ITL - International Journal of Applied Linguistics
Vol. 170:1 (2019) ► pp.24–52

Lexical aspects of comprehensibility and nativeness from the perspective of native-speaking English raters

| Waseda University
| Concordia University
| Birkbeck, University of London
| University College London
| Western University
This study analyzed the contribution of lexical factors to native-speaking raters’ assessments of comprehensibility and nativeness in second language (L2) speech. Using transcribed samples to reduce non-lexical sources of bias, 10 naïve L1 English raters evaluated speech samples from 97 L2 English learners across two tasks (picture description and TOEFL integrated). Subsequently, the 194 transcripts were analyzed through statistical software (e.g., Coh-metrix, VocabProfile) for 29 variables spanning various lexical dimensions. For the picture description task, separation in lexical correlates of the two constructs was found, with distinct lexical measures tied to comprehensibility and nativeness. In the TOEFL integrated task, comprehensibility and nativeness were largely indistinguishable, with identical sets of lexical variables, covering dimensions of diversity and range. Findings are discussed in relation to the acquisition, assessment, and teaching of lexical properties in L2 speech.
Keywords: second language speech, vocabulary, comprehensibility, nativeness
Article outline
  • Research on L2 speaking
  • L2 comprehensibility versus nativeness
  • The current study
  • Method
    • Speakers
    • Speaking tasks
    • Materials
    • Raters
    • Rating procedure
  • Lexical analysis
  • Coh-metrix measures
  • VocabProfile measures
  • Results
    • Comprehensibility and nativeness ratings
    • Lexical variables and rated constructs
    • Lexical predictors of comprehensibility and nativeness
  • Discussion
    • Lexical correlates of speech ratings
    • Comprehensibility versus nativeness
    • Task effects
  • Implications
  • Limitations and future research
  • Conclusion
  • Notes
  • References
Published online: 5 April 2019
DOI logo
https://doi.org/10.1075/itl.17026.app
References (55)
References
Abercrombie, D. (1949). Teaching pronunciation. ELT Journal, 31, 113–122. DOI logoGoogle Scholar
Adams, M. L. (1980). Five co-occurring factors in speaking proficiency. In J. R. Firth (Ed.), Measuring spoken language proficiency (pp. 1–6). Washington, DC: Georgetown University Press.Google Scholar
Anderson, R. C., & Freebody, P. (1981). Vocabulary knowledge. In J. T. Gutherie (Ed.) Comprehension and teaching: Researching reviews (pp. 77–117). Newark, DE. International Reading Association.Google Scholar
Baayen, R. H., Piepenbrock, R., & Gulikers, L. (1995). CELEX. Philadelphia, PA: Linguistic Data Consortium.Google Scholar
Browne, C., Culligan, B., & Phillips, J. (2013). The new general service list. Retrieved from <[URL]>
Cobb, T. (2016). Compleat Lexical Tutor [computer program]. <[URL]> (15 January 2016).
Crossley, S. A., Cobb, T., & McNamara, D. (2013). Comparing count-based and band-based indices of word frequency: Implications for active vocabulary research and pedagogical applications. System, 411, 965–981. DOI logoGoogle Scholar
Crossley, S. A., & McNamara, D. (2013). Applications of text analysis tools for spoken response grading. Language Learning & Technology, 171, 171–192.Google Scholar
Crossley, S. A., Salsbury, T., & McNamara, D. S. (2009). Measuring L2 lexical growth using hypernymic relationships. Language Learning, 591, 307–334. DOI logoGoogle Scholar
(2010). The development of polysemy and frequency use in English second language speakers. Language Learning, 601, 573–605. DOI logoGoogle Scholar
(2015). Assessing lexical proficiency using analytic ratings: A case for collocation accuracy. Applied Linguistics, 361, 570–590.Google Scholar
Crossley, S. A., Salsbury, T., McNamara, D. S., & Jarvis, S. (2011). What is lexical proficiency? Some answers from computational models of speech data. TESOL Quarterly, 451, 182–193. DOI logoGoogle Scholar
Crossley, S. A., Yang, H. S., & McNamara, D. S. (2014). What’s so simple about simplified texts? A computational and psycholinguistic investigation of text comprehension and text processing. Reading in a Foreign Language, 261, 92–113.Google Scholar
Crowther, D., Trofimovich, P., Isaacs, T., & Saito, K. (2015). Does a speaking task affect second language comprehensibility? The Modern Language Journal, 991, 80–95. DOI logoGoogle Scholar
Derwing, T. M., & Munro, M. J. (2015). Pronunciation fundamentals: Evidence-based perspectives for L2 teaching and research. Amsterdam: John Benjamins. DOI logoGoogle Scholar
(2013). The development of L2 oral language skills in two L1 groups: A seven-year study. Language Learning, 631, 163–185. DOI logoGoogle Scholar
Derwing, T. M., Rossiter, M. J., & Ehrensberger-Dow, M. (2002). They speaked and wrote real good: Judgements of non-native and native grammar. Language Awareness, 111, 84–99. DOI logoGoogle Scholar
Derwing, T. M., Rossiter, M. J., Munro, M. J., & Thomson, R. I. (2004). L2 fluency: Judgements on different tasks. Language Learning, 541, 655–679. DOI logoGoogle Scholar
Educational Testing Services (2004). Independent Speaking Scoring Rubrics. Retrieved from <[URL]>
Field, A. (2009). Discovering statistics using SPSS (3rd ed.). London: Sage.Google Scholar
Ginther, A., Dimova, S., & Yang, R. (2010). Conceptual and empirical relationships between temporal measures of fluency and oral English proficiency with implications for automated scoring. Language Testing, 271, 379–399. DOI logoGoogle Scholar
Graesser, A. C., McNamara, D. S., & Kulikowich, J. M. (2011). Coh-Metrix providing multilevel analyses of text characteristics. Educational Researcher, 401, 223–234. DOI logoGoogle Scholar
Granena, G. (2014). Individual differences in sequence learning ability and second language acquisition in early childhood and adulthood. Language Learning, 631, 665–703. DOI logoGoogle Scholar
Iwashita, N., Brown, A., McNamara, T., & O’Hagan, S. (2008). Assessed levels of second language speaking proficiency: How distinct? Applied Linguistics, 291, 24–49. DOI logoGoogle Scholar
Jarvis, S., & Daller, M. (2013). Vocabulary knowledge: Human ratings and automated measures. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Kang, O., Rubin, D., Pickering, L. (2010). Suprasegmental measures of accentedness and judgements of English language learner proficiency in oral English. The Modern Language Journal, 941, 554–566. DOI logoGoogle Scholar
Koizumi, R., & In’nami, Y. (2012). Effects of text length on lexical diversity measures: Using short texts with less than 200 tokens. System, 401, 554–564. DOI logoGoogle Scholar
Kormos, J., & Dénes, M. (2004). Exploring measures and perceptions of fluency in the speech of second language learners. System, 321, 145–164. DOI logoGoogle Scholar
Kuiken, F., & Vedder, I. (2014). Raters’ decisions, rating procedures and rating scales. Language Testing, 311, 279–284. DOI logoGoogle Scholar
Kyle, K., & Crossley, S. (2015). Automatically assessing lexical sophistication: Indices, tools, findings, and applications. TESOL Quarterly, 491, 757–786. DOI logoGoogle Scholar
Larson-Hall, J. (2010). A guide to doing statistics in second language research using SPSS. New York: Routledge.Google Scholar
Levis, J. (2005). Changing contexts and shifting paradigms in pronunciation teaching. TESOL Quarterly, 391, 369–377. DOI logoGoogle Scholar
Lu, X. (2012). The relationship of lexical richness to the quality of ESL learners’ oral narratives. The Modern Language Review, 961, 190–208. DOI logoGoogle Scholar
McNamara, D. S., Graesser, A. C., McCarthy, P., & Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
McNamara, T. F. (1990). Item response theory and the validation of an ESP test for health professionals. Language Testing, 71, 52–75. DOI logoGoogle Scholar
Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., & Miller, K. J. (1990). Introduction to WordNet: An on-line lexical database. International Journal of Lexicography, 31, 235–244. DOI logoGoogle Scholar
Moyer, A. (2013). Foreign accent: The phenomenon of non-native speech. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Munro, M. J., & Derwing, T. M. (2006). The functional load principle in ESL pronunciation instruction: An exploratory study. System, 341, 520–531. DOI logoGoogle Scholar
(2009). Comprehensibility as a factor in listener interaction preferences: Implications for the workplace. The Canadian Modern Language Review, 661, 181–202. DOI logoGoogle Scholar
Munro, M. J., & Derwing, T. M., Burgess, C. S. (2010). Detection of nonnative speaker status from content-masked speech. Speech Communication, 521, 626–637. DOI logoGoogle Scholar
Nation, I. S. P. (2012). The BNC/COCA word family lists (17 September 2012). Unpublished paper. Available at <[URL]>
Neufeld, S., & Billuroğlu, A. (2005). In search of the critical lexical mass: How ‘general’ is the GSL? How ‘academic’ is the AWL? Available at <[URL]>
Oppenheimer, D. M. (2008). The secret life of fluency. Trends in Cognitive Sciences, 121, 237–241. DOI logoGoogle Scholar
Perfetti, C. A., Landi, N., & Oakhill, J. The acquisition of reading comprehension skill. In M. J. Snowling & C. Hulme (Eds.), The science of reading: A handbook (pp. 227–247). Oxford: Blackwell.
Plonsky, L., & Oswald, F. L. (2014). How big is “big”? Interpreting effects sizes in L2 research. Language Learning, 641, 878–912. DOI logoGoogle Scholar
Read, J. (2004). Plumbing the depths. In P. Bogaards & B. Laufer (Eds.), Vocabulary in second language (pp. 209–227). Amsterdam: John Benjamins. DOI logoGoogle Scholar
Révész, A., Michel, M., & Gilabert, R. (2015). Measuring cognitive task demands using dual-task methodology, subjective self-ratings, and expert judgements. Studies in Second Language Acquisition, 38(4), 703–737. DOI logoGoogle Scholar
Robinson, P. (2005). Cognitive complexity and task sequencing: Studies in a componential framework for second language task design. International Review of Applied Linguistics in Language Teaching, 431, 1–32. DOI logoGoogle Scholar
Saito, K., Trofimovich, P., Isaacs, T. (2016). Second language speech production: Investigating linguistic correlates of comprehensibility and accentedness for learners at different ability levels. Applied Psycholinguistics, 371, 217–240. DOI logoGoogle Scholar
Saito, K., Webb, S., Trofimovich, P., & Isaacs, T. (2015). Lexical profiles of comprehensible second language speech. Studies in Second Language Acquisition, 381, 677–701. DOI logoGoogle Scholar
Salsbury, T., Crossley, S. A., & McNamara, D. S. (2011). Psycholinguistic word information in second language oral discourse. Second Language Research, 271, 343–360. DOI logoGoogle Scholar
Trofimovich, P., & Isaacs, T. (2012). Disentangling accent from comprehensibility. Bilingualism: Language and Cognition, 151, 905–916. DOI logoGoogle Scholar
Wen, Q., Wang, L., & Liang, M. (2005). Spoken and written English corpus of Chinese learners. Beijing: Foreign Language Teaching and Research Press.Google Scholar
Wilson, M. D. (1988). The MRC psycholinguistic database: Machine readable dictionary. Behavioural Research Methods, Instruments and Computers, 201, 6–11. DOI logoGoogle Scholar
Wray, A. (2002). Formulaic language and the lexicon. New York, NY: Cambridge University Press. DOI logoGoogle Scholar
Cited by (10)

Cited by ten other publications

Díez-Ortega, María & Kristopher Kyle
2024. Measuring the development of lexical richness of L2 Spanish: A longitudinal learner corpus study. Studies in Second Language Acquisition 46:1 ► pp. 169 ff. DOI logo
Tsunemoto, Aki & Pavel Trofimovich
2024. Coherence and Comprehensibility in Second Language Speakers’ Academic Speaking Performance. Studies in Second Language Acquisition ► pp. 1 ff. DOI logo
Wang-Taylor, Yixin, Jon Clenton & Yinna Ren
2024. Investigating the Impact of Dialogic and Trialogic Interactive Factors on Chinese Advanced L2 learners’ Vocabulary Use in Spoken Contexts. Languages 9:8 ► pp. 266 ff. DOI logo
Shao, Yujie, Kazuya Saito & Adam Tierney
2023. How Does Having a Good Ear Promote Instructed Second Language Pronunciation Development? Roles of Domain‐General Auditory Processing in Choral Repetition Training. TESOL Quarterly 57:1 ► pp. 33 ff. DOI logo
Wang-Taylor, Yixin & Jon Clenton
2022. Exploring the relationship between L2 vocabulary size and academic speaking. System 107 ► pp. 102822 ff. DOI logo
Saito, Kazuya
2021. What Characterizes Comprehensible and Native‐like Pronunciation Among English‐as‐a‐Second‐Language Speakers? Meta‐Analyses of Phonological, Rater, and Instructional Factors. TESOL Quarterly 55:3 ► pp. 866 ff. DOI logo
Saito, Kazuya, Yui Suzukida, Mai Tran & Adam Tierney
2021. Domain‐General Auditory Processing Partially Explains Second Language Speech Learning in Classroom Settings: A Review and Generalization Study. Language Learning 71:3 ► pp. 669 ff. DOI logo
Suzukida, Yui & Kazuya Saito
2021. Which segmental features matter for successful L2 comprehensibility? Revisiting and generalizing the pedagogical value of the functional load principle. Language Teaching Research 25:3 ► pp. 431 ff. DOI logo
Noreillie, Ann-Sophie, Piet Desmet & Elke Peters
2020. Factors Predicting Low-Intermediate French Learners’ Vocabulary Use in Speaking Tasks. The Canadian Modern Language Review 76:3 ► pp. 194 ff. DOI logo
Saito, Kazuya, Mai Tran, Yui Suzukida, Hui Sun, Viktoria Magne & Meltem Ilkan
2019. HOW DO SECOND LANGUAGE LISTENERS PERCEIVE THE COMPREHENSIBILITY OF FOREIGN-ACCENTED SPEECH?. Studies in Second Language Acquisition 41:5 ► pp. 1133 ff. DOI logo

This list is based on CrossRef data as of 18 october 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.