Cover not available
Part of
The Evolution of Pronunciation Teaching and Research: 25 years of intelligibility, comprehensibility, and accentedness
Edited by John M. Levis, Tracey M. Derwing and Murray J. Munro
[Benjamins Current Topics 121] 2022
► pp. 125–151

Reactions to second language speech

Influences of discrete speech characteristics, rater experience, and speaker first language background

ORCID logo | University College London
| Brock University

This study investigates how Mandarin and Slavic language speakers’ comprehensibility, accentedness, and fluency ratings, as assigned by experienced teacher-raters and novice raters, align with discrete linguistic measures, and raters’ accounts of influences on their scoring. In addition to examining mean ratings in relation to rater experience and speaker first language background, we correlated ratings with segmental, prosodic, and temporal measures. Introspective reports were segmented, coded, enumerated, and submitted to loglinear analysis to elucidate influences on ratings. Results showed that ratings were strongly correlated with prosodic goodness and moderately correlated with segmental errors, implying the importance of both segmentals and prosody in L2 speech ratings. Experienced teacherraters provided lengthier reports than novice raters, producing more comments for all coded categories where an error was identified except for pausing (a disfluency marker). This may be because novice raters observed little else about the speech or struggled to pinpoint or articulate other features.

Keywords: accent, comprehensibility, English as a second language, fluency, pronunciation assessment, raters, rating scales, speech perception
Article outline
  • 1.Introduction
  • 2.The current study
  • 3.Method
    • 3.1Research design
    • 3.2L2 speakers
    • 3.3Speech elicitation and data preparation
    • 3.4Raters
    • 3.5Rating sessions
    • 3.6Rating scale normalization
    • 3.7Deriving discrete linguistic measures from the L2 speech samples
    • 3.8Analysis of introspective reports
  • 4.Results
    • 4.1Preliminary analyses
    • 4.2Rater experience and speaker L1 in relation to global ratings and discrete measures
    • 4.3Analysis of the factors that raters reportedly take notice of when rating L2 speech
  • 5.Discussion
    • 5.1Rater experience
    • 5.2Speaker L1 background
    • 5.3Concluding remarks
  • Acknowledgments
  • Note
  • References
Published online: 1 June 2022
DOI logo
https://doi.org/10.1075/bct.121.07isa
References
Bannigan, K., & Watson, R.
(2009) Reliability and validity in a nutshell. Journal of Clinical Nursing, 18(23), 3237–3243. DOI logoGoogle Scholar
Bongaerts, T., Mennen, S., & van der Slik, F.
(2000) Authenticity of pronunciation in naturalistic second language acquisition: The case of very advanced late learners of Dutch as a second language. Studia Linguistica, 54(2), 298–308. DOI logoGoogle Scholar
Bongaerts, T., van Summeren, C., Planken, B., & Schils, E.
(1997) Age and ultimate attainment in the pronunciation of a foreign language. Studies in Second Language Acquisition, 19(4), 447–465. DOI logoGoogle Scholar
Bradlow, A., Clopper, C., Smiljanic, R., & Walter, M. A.
(2010) A perceptual phonetic similarity space for languages: Evidence from five native language listener groups. Speech Communication, 52(11–12), 930–942. DOI logoGoogle Scholar
Browne, K., & Fulcher, G.
(2017) Pronunciation and intelligibility in assessing spoken fluency. In T. Isaacs & P. Trofimovich (Eds.), Second language pronunciation: Interdisciplinary perspectives (pp. 37–53). Bristol, UK: Multilingual Matters.Google Scholar
Chalhoub-Deville, M.
(1995) Deriving oral assessment scales across different tests and rater groups. Language Testing, 12(1), 62–70. DOI logoGoogle Scholar
Creswell, J. W., & Plano Clark, V. L.
(2017) Designing and conducting mixed methods research (3rd ed.). Thousand Oaks, CA: SAGE.Google Scholar
Crowther, D., Trofimovich, P., Saito, K., & Isaacs, T.
(2015) Second language comprehensibility revisited: Investigating the effects of learner background. TESOL Quarterly, 49(4), 814–837. DOI logoGoogle Scholar
de Boer, M., & Heeren, W.
(2019) The speaker-specificity of filled pauses: A cross-linguistic study. Proceedings of the International Congress of Phonetic Sciences (ICPhS) 2019 (pp. 607–611). Melbourne, Australia: Australasian Speech Science and Technology Association.Google Scholar
Derwing, T. M., & Munro, M. J.
(1997) Accent, intelligibility, and comprehensibility: Evidence from four L1s. Studies in Second Language Acquisition, 19(1), 1–16. DOI logoGoogle Scholar
(2009) Comprehensibility as a factor in listener interaction preferences: Implications for the workplace. Canadian Modern Language Review, 66(2), 181–202. DOI logoGoogle Scholar
(2013) The development of L2 oral language skills in two L1 groups: A 7-year study. Language Learning, 63(2), 163–185. DOI logoGoogle Scholar
Derwing, T. M., Rossiter, M. J., Munro, M. J., & Thomson, R. I.
(2004) Second language fluency: Judgments on different tasks. Language Learning, 54(4), 665–679. DOI logoGoogle Scholar
Derwing, T. M., Thomson, R. I., & Munro, M. J.
(2006) English pronunciation and fluency development in Mandarin and Slavic speakers. System, 34(2), 183–193. DOI logoGoogle Scholar
Douglas, D.
(1994) Quantity and quality in speaking test performance. Language Testing, 11(1), 125–144. DOI logoGoogle Scholar
Ericsson, K. A., & Simon, H. A.
(1993) Protocol analysis: Verbal reports as data (Rev. ed.). Cambridge, MA: MIT Press. DOI logoGoogle Scholar
Foote, J. A., Isaacs, T., & Trofimovich, P.
(2013, June 3–5). Developing a teacher-friendly assessment tool for L2 comprehensibility. Canadian Association of Applied Linguistics (ACLA/CAAL) conference, Calgary, AB.
Foote, J. A., & Trofimovich, P.
(2018) Is it because of my language background? A study of language background influence on comprehensibility judgments. Canadian Modern Language Review, 74(2), 253–278. DOI logoGoogle Scholar
Gass, S. M., & Mackey, A.
(2000) Stimulated recall methodology in second language research. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
Hahn, L. D.
(2004) Primary stress and intelligibility: Research to motivate the teaching of suprasegmentals. TESOL Quarterly, 38(2), 201–233. DOI logoGoogle Scholar
Isaacs, T., & Harding, L.
(2017) Research timeline: Pronunciation assessment. Language Teaching, 50(3), 347–366. DOI logoGoogle Scholar
Isaacs, T., & Thomson, R. I.
(2013) Rater experience, rating scale length, and judgments of L2 pronunciation: Revisiting research conventions. Language Assessment Quarterly, 10(2), 135–159. DOI logoGoogle Scholar
Isaacs, T., & Trofimovich, P.
(2012) Deconstructing comprehensibility: Identifying the linguistic influences on listeners’ L2 comprehensibility ratings. Studies in Second Language Acquisition, 34(3), 475–505. DOI logoGoogle Scholar
Isaacs, T., Trofimovich, P., Yu, G., & Chereau, B. M.
(2015) Examining the linguistic aspects of speech that most efficiently discriminate between upper levels of the revised IELTS pronunciation scale. IELTS research reports online series, 4.Google Scholar
Kang, O., & Moran, M.
(2014) Functional loads of pronunciation features in nonnative speakers’ oral assessment. TESOL Quarterly, 48(1), 176–187. DOI logoGoogle Scholar
Lennon, P.
(1990) Investigating fluency in EFL: A quantitative approach. Language Learning, 40(3), 387–417. DOI logoGoogle Scholar
Lumley, T.
(2005) Assessing second language writing: The rater’s perspective. Frankfurt: Peter Lang.Google Scholar
McAndrews, M. M., & Thomson, R. I.
(2017) Establishing an empirical basis for priorities in pronunciation teaching. Journal of Second Language Pronunciation, 3(2), 267–287. DOI logoGoogle Scholar
Munro, M. J.
(2018) Dimensions of pronunciation. In O. Kang, R. Thomson, & J. Murphy. The Routledge handbook of contemporary English pronunciation (pp. 413–431). New York: Routledge.Google Scholar
Munro, M. J., & Derwing, T. M.
(2006) The functional load principle in ESL pronunciation instruction: An exploratory study. System, 34(4), 520–531. DOI logoGoogle Scholar
O’Brien, M. G.
(2016) Methodological choices in rating speech samples. Studies in Second Language Acquisition, 38(3), 587–605. DOI logoGoogle Scholar
Pawlikowska-Smith, G.
(2000) Canadian Language Benchmarks 2000: Theoretical framework. Ottawa, ON: Centre for Canadian Language Benchmarks.Google Scholar
Rajadurai, J.
(2007) Intelligibility studies: A consideration of empirical and ideological issues. World Englishes, 26(1), 87–98. DOI logoGoogle Scholar
Riggenbach, H.
(1991) Toward an understanding of fluency: A microanalysis of non-native speaker conversations. Discourse Processes, 14(4), 423–441. DOI logoGoogle Scholar
Rose, H., & Galloway, N.
(2019) Global Englishes for language teaching. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Rossiter, M. J.
(2009) Perceptions of L2 fluency by native and non-native speakers of English. Canadian Modern Language Review, 65(3), 395–412. DOI logoGoogle Scholar
Saito, K., Trofimovich, P., & Isaacs, T.
(2016) Second language speech production: Investigating linguistic correlates of comprehensibility and accentedness for learners at different ability levels. Applied Psycholinguistics, 37(2), 217–240. DOI logoGoogle Scholar
Saito, K., Trofimovich, P., Isaacs, T., & Webb, S.
(2017) Re-examining phonological and lexical correlates of second language comprehensibility: The role of rater experience. In T. Isaacs & P. Trofimovich (Eds.), Second language pronunciation assessment: Interdisciplinary perspectives (pp. 131–146). Bristol, UK: Multilingual Matters.Google Scholar
Saito, K., Webb, S., Trofimovich, P., & Isaacs, T.
(2016) Lexical correlates of comprehensibility versus accentedness in second language speech. Bilingualism: Language and Cognition, 19(3), 597–609. DOI logoGoogle Scholar
Schiavetti, N.
(1992) Scaling procedures for the measurement of speech intelligibility. In R. D. Kent (Ed.), Intelligibility in speech disorders (pp. 11–34). Amsterdam: John Benjamins. DOI logoGoogle Scholar
Stevens, J. P.
(2009) Applied multivariate statistics for the social sciences (5th ed.). New York: Taylor & Francis.Google Scholar
Suzukida, Y., & Saito, K.
(2019) Which segmental features matter for successful L2 comprehensibility? Revisiting and generalizing the pedagogical value of the Functional Load principle. Language Teaching Research. Advance online publication. DOI logoGoogle Scholar
Thomson, R. I., & Isaacs, T.
(2009) Within-category variation in L2 English vowel learning. Canadian Acoustics, 37, 138–139.Google Scholar
Thompson, I.
(1991) Foreign accents revisited: The English pronunciation of Russian immigrants. Language Learning, 41(2), 177–204. DOI logoGoogle Scholar
Upshur, J. A., & Turner, C. E.
(1999) Systematic effects in the rating of second-language speaking ability: Test method and learner discourse. Language Testing, 16(1), 82–111. DOI logoGoogle Scholar
Winke, P., Gass, S., & Myford, C.
(2013) Raters’ L2 background as a potential source of bias in rating oral performance. Language Testing, 30(2), 231–252. DOI logoGoogle Scholar
Zielinski, B. W.
(2008) The listener: No longer the silent partner in reduced intelligibility. System, 36(1), 69–84. DOI logoGoogle Scholar