Cover not available
Article published In:
Journal of Second Language Pronunciation
Vol. 6:2 (2020) ► pp.236–264

Testing the malleability of teachers’ judgments of second language speech

ORCID logo | Concordia University, Canada
| University of Calgary, Canada
| Concordia University, Canada
| University of Calgary, Canada

This study examined whether a negative social bias can influence how teachers evaluate second language (L2) speech. Twenty-eight teachers of L2 German from Western Canada – 14 native speakers (NSs) and 14 proficient non-native speakers (NNSs) – rated recordings of 24 adult L2 learners of German across five speech dimensions (accentedness, comprehensibility, vowel/consonant accuracy, intonation, flow) using 1,000-point scales. Immediately before rating, half of NS and NNS teachers heard critical comments about undergraduate German students’ language skills, while the other half heard no biasing comments. Under negative bias, while the NNS teachers provided favorable evaluations across all five measures, NS teachers followed suit for only intonation and flow, downgrading L2 speakers’ accentedness, comprehensibility, and vowel/consonant accuracy. Findings call into question the relative stability of L2 speech ratings and highlight the importance of social context and teacher status as native versus non-native speakers of the target language in assessments of L2 speaking performance.

Keywords: German, social attitudes, speech rating, accent bias, linguistic stereotyping, second language, speech assessment, teacher status, accentedness, comprehensibility, fluency
Article outline
  • 1.Introduction
  • 2.Background literature
    • 2.1Listener biases and their origins
    • 2.2Listener-specific factors
    • 2.3The current study
  • 3.Method
    • 3.1Teacher raters
    • 3.2Speech materials
    • 3.3Rating procedure
    • 3.4Data analysis
  • 4.Results
  • 5.Discussion
    • 5.1L2 speech ratings and listener bias
    • 5.2Sources of teacher bias
    • 5.3Relationship between teacher status and negative bias varies across speech dimensions
    • 5.4Uneven baselines in the absence of negative bias
  • 6.Implications and conclusion
  • Acknowledgements
  • Notes
  • References
Published online: 9 March 2020
DOI logo
https://doi.org/10.1075/jslp.19015.tay
References
Anderson-Hsieh, J., Johnson, R., & Koehler, K.
(1992) The relationship between native speaker judgments of nonnative pronunciation and deviance in segmentals, prosody, and syllable structure. Language Learning, 421, 529–555. DOI logoGoogle Scholar
Beukeboom, C. J., & Semin, G. R.
(2006) How mood turns on language. Journal of Experimental Social Psychology, 421, 553–566. DOI logoGoogle Scholar
Bodenhausen, G. V., Mussweiler, T., Gabriel, S., & Moreno, K. N.
(2001) Affective influences on stereotyping and intergroup relations. In J. P. Forgas (Ed.), Handbook of affect and social cognition (pp. 319–343). Mahwah, NJ: Lawrence Erlbaum.Google Scholar
Bongaerts, T., van Summeren, C., Planken, B., & Schils, E.
(1997) Age and ultimate attainment in the pronunciation of a foreign language. Studies in Second Language Acquisition, 191, 447–465. DOI logoGoogle Scholar
Boughton, Z.
(2006) When perception isn’t reality: Accent identification and perceptual dialectology in French. Journal of French Language Studies, 161, 277–304. DOI logoGoogle Scholar
Bradlow, A. R., & Bent, T.
(2008) Perceptual adaptation to non-native speech. Cognition, 1061, 707–729. DOI logoGoogle Scholar
Brown, A.
(1995) The effect of rater variables in the development of an occupation-specific language performance test. Language Testing, 121, 1–15. DOI logoGoogle Scholar
Bryk, A. S., & Schneider, B.
(2003) Trust in schools: A core resource for school reform. Educational Leadership, 601, 40–45.Google Scholar
Buckingham, L.
(2014) Attitudes to English teachers’ accents in the Arabian Gulf. International Journal of Applied Linguistics, 241, 50–73. DOI logoGoogle Scholar
Carey, M. D., Mannell, R. H., & Dunn, P. K.
(2011) Does a rater’s familiarity with a candidate’s pronunciation affect the rating in oral proficiency interviews? Language Testing, 281, 201–219. DOI logoGoogle Scholar
Cargile, A. C., & Giles, H.
(1997) Understanding language attitudes: Exploring listener affect and identity. Language & Communication, 171, 195–217. DOI logoGoogle Scholar
Chiba, R., Matsuura, H., & Yamamoto, A.
(1995) Japanese attitudes toward English accents. World Englishes, 141, 77–86. DOI logoGoogle Scholar
Crowther, D., Trofimovich, P., & Isaacs, T.
(2016) Linguistic dimensions of second language accent and comprehensibility. Journal of Second Language Pronunciation, 21, 160–182. DOI logoGoogle Scholar
D’Onofrio, A.
(2018) Controlled and automatic perceptions of a sociolinguistic marker. Language Variation and Change, 301, 261–285. DOI logoGoogle Scholar
forthcoming). Sociolinguistic signs as cognitive representations. In L. Hall-Lew, E. Moore, & R. J. Podesva Eds. Social meaning and linguistic variation Cambridge, UK Cambridge University Press
Dailey-O’Cain, J.
(1999) The perception of post-unification German regional speech. In D. R. Preston (Ed.), Handbook of perceptual dialectology (Vol 11, pp. 227–242). Philadelphia, PA: John Benjamins. DOI logoGoogle Scholar
Dalton-Puffer, C., Kaltenböck, G., & Smit, U.
(1997) Learner attitudes and L2 pronunciation in Austria. World Englishes, 161, 115–128. DOI logoGoogle Scholar
Derwing, T. M., & Munro, M.
(2013) The development of L2 oral language skills in two L1 groups: A 7-year study. Language Learning, 631, 163–185. DOI logoGoogle Scholar
Derwing, T. M., & Munro, M. J.
(2015) Pronunciation fundamentals: Evidence-based perspectives for L2 teaching and research. Philadelphia, PA: John Benjamins. DOI logoGoogle Scholar
Derwing, T. M., Rossiter, M. J., Munro, M. J., & Thomson, R. I.
(2004) Second language fluency: Judgments on different tasks. Language Learning, 541, 655–679. DOI logoGoogle Scholar
Eskenazi, M.
(2009) An overview of spoken language technology for education. Speech Communication, 511, 832–844. DOI logoGoogle Scholar
Fayer, J. M., & Krasinski, E.
(1987) Native and nonnative judgments of intelligibility and irritation. Language Learning, 371, 313–326. DOI logoGoogle Scholar
Fazio, R. H., Sanbonmatsu, D. M., Powell, M. C., & Kardes, F. R.
(1986) On the automatic activation of attitudes. Journal of Personality and Social Psychology, 501, 229–238. DOI logoGoogle Scholar
Foote, J. A., & Trofimovich, P.
(2018) Is it because of my language background? A study of language background influence on comprehensibility judgments. Canadian Modern Language Review, 741, 253–278. DOI logoGoogle Scholar
Ford, C. E.
(1984) The influence of speech variety on teachers’ evaluation of students with comparable academic ability. TESOL Quarterly, 181, 25–40. DOI logoGoogle Scholar
Galinsky, A. D., & Moskowitz, G. B.
(2000) Perspective-taking: Decreasing stereotype expression, stereotype accessibility, and in-group favoritism. Journal of Personality and Social Psychology, 781, 708–724. DOI logoGoogle Scholar
Goethe Institut
(2004) Einstufungstest [Placement test]. [URL]
Hansen Edwards, J. G., Zampini, M. L., & Cunningham, C.
(2018) The accentedness, comprehensibility, and intelligibility of Asian Englishes. World Englishes, 371, 538–557. DOI logoGoogle Scholar
(2019) Listener judgments of speaker and speech traits of varieties of Asian English. Journal of Multilingual and Multicultural Development, 401, 691–706. DOI logoGoogle Scholar
Hansen, K., Rakić, T., & Steffens, M. C.
(2014) When actions speak louder than words: Preventing discrimination of nonstandard speakers. Journal of Language and Social Psychology, 331, 68–77. DOI logoGoogle Scholar
He, D., & Miller, L.
(2011) English teacher preference: The case of China’s non-English-major students. World Englishes, 301, 428–443. DOI logoGoogle Scholar
Hu, G., & Lindemann, S.
(2009) Stereotypes of Cantonese English, apparent native/non-native status, and their effect on non-native English speakers’ perception. Journal of Multilingual and Multicultural Development, 301, 253–269. DOI logoGoogle Scholar
Hu, G., & Su, J.
(2015) The effect of native/non-native information on non-native listeners’ comprehension. Language Awareness, 241, 273–281. DOI logoGoogle Scholar
Hundt, M., Zipp, L., & Huber, A.
(2015) Attitudes in Fiji towards varieties of English. World Englishes, 341, 688–707. DOI logoGoogle Scholar
Isaacs, T., & Thomson, R. I.
(2013) Rater experience, rating scale length, and judgments of L2 pronunciation: Revisiting research conventions. Language Assessment Quarterly, 101, 135–159. DOI logoGoogle Scholar
Kang, O.
(2012) Impact of rater characteristics and prosodic features of speaker accentedness on ratings of international teaching assistants’ oral performance. Language Assessment Quarterly, 91, 249–269. DOI logoGoogle Scholar
Kang, O., & Rubin, D. L.
(2009) Reverse linguistic stereotyping: Measuring the effect of listener expectations on speech evaluation. Journal of Language and Social Psychology, 281, 441–456. DOI logoGoogle Scholar
Kawakami, K., Phills, C. E., Greenwald, A. G., Simard, D., Pontiero, J., Brnjas, A., & Dovidio, J. F.
(2012) In perfect harmony: Synchronizing the self to activated social categories. Journal of Personality and Social Psychology, 1021, 562–575. DOI logoGoogle Scholar
Lambert, W. E., Hodgson, R. C., Gardner, R. C., & Fillenbaum, S.
(1960) Evaluational reactions to spoken languages. The Journal of Abnormal and Social Psychology, 601, 44–51. DOI logoGoogle Scholar
Lappin-Fortin, K., & Rye, B. J.
(2014) The use of pre-/posttest and self-assessment tools in a French pronunciation course. Foreign Language Annals, 471, 300–320. DOI logoGoogle Scholar
Lindemann, S.
(2003) Koreans, Chinese or Indians? Attitudes and ideologies about non-native English speakers in the United States. Journal of Sociolinguistics, 71, 348–364. DOI logoGoogle Scholar
Major, R. C., Fitzmaurice, S. F., Bunta, F., & Balasubramanian, C.
(2002) The effects of non-native accents on listening comprehension: Implications for ESL assessment. TESOL Quarterly, 361, 173–190. DOI logoGoogle Scholar
Montgomery, C.
(2007) Northern English dialects: A perceptual approach (Unpublished PhD dissertation). University of Sheffield, Sheffield, United Kingdom.Google Scholar
Munro, M. J., Derwing, T. M., & Morton, S. L.
(2006) The mutual intelligibility of L2 speech. Studies in Second Language Acquisition, 281, 111–131. DOI logoGoogle Scholar
Norton, B., & Toohey, K.
(2011) Identity, language learning, and social change. Language Teaching, 441, 412–446. DOI logoGoogle Scholar
O’Brien, M. G.
(2014) L2 learners’ assessments of accentedness, fluency, and comprehensibility of native and nonnative German speech. Language Learning, 641, 715–748. DOI logoGoogle Scholar
Pantos, A. J., & Perkins, A. W.
(2013) Measuring implicit and explicit attitudes toward foreign accented speech. Journal of Language and Social Psychology, 321, 3–20. DOI logoGoogle Scholar
Preston, D.
(1999) Handbook of perceptual dialectology 11. Philadelphia, PA: John Benjamins. DOI logoGoogle Scholar
Reid, K. T., Trofimovich, P., & O’Brien, M. G.
(2019) Social attitudes and speech ratings: Effects of positive and negative bias on multiage listeners’ judgments of second language speech. Studies in Second Language Acquisition, 411, 419–442. DOI logoGoogle Scholar
Rose, R. L.
(2017) Differences in second language speech fluency ratings: Native versus nonnative listeners. In L. Degand et al. (Eds.), Proceedings of the International Conference “Fluency & Disfluency Across Languages and Language Varieties” (pp. 101–103). Louvain-la-Neuve, Belgium: Université catholique de Louvain.Google Scholar
Rossiter, M. J.
(2009) Perceptions of L2 fluency by native and non-native speakers of English. Canadian Modern Language Review, 651, 395–412. DOI logoGoogle Scholar
Rubin, D. L.
(1992) Nonlanguage factors affecting undergraduates’ judgments of nonnative English-speaking teaching assistants. Research in Higher Education, 331, 51–531. DOI logoGoogle Scholar
Ryan, E. B., Carranza, M. A., & Moffie, R. W.
(1977) Reactions toward varying degrees of accentedness in the speech of Spanish-English bilinguals. Language and Speech, 201, 267–273. DOI logoGoogle Scholar
Ryan, E. B., Hewstone, M., & Giles, H.
(1984) Language and intergroup attitudes. In J. R. Eiser (Ed.), Attitudinal judgment (pp. 135–158). New York: Springer. DOI logoGoogle Scholar
Sokolov, C.
(2014) Self-evaluation of rater bias in written composition assessment. Linguistica, 541, 261–275. DOI logoGoogle Scholar
Spillane, J. P.
(2006) Distributed leadership. San Francisco, CA: Jossey-Bass. DOI logoGoogle Scholar
Spillane, J. P., Hallett, T., & Diamond, J. B.
(2003) Forms of capital and the construction of leadership: Instructional leadership in urban elementary schools. Sociology of Education, 761, 1–17. DOI logoGoogle Scholar
Steele, C. M., & Aronson, J.
(1995) Stereotype threat and the intellectual test performance of African Americans. Journal of Personality and Social Psychology, 691, 797–811. DOI logoGoogle Scholar
Stewart, M. A., Ryan, E. G., & Giles, H.
(1985) Accent and social class effects on status and solidarity evaluations. Personality and Social Psychology Bulletin, 111, 98–105. DOI logoGoogle Scholar
Tajfel, H.
(1972) Experiments in a vacuum. In J. Israel & H. T. Triandis (Eds.), The context of social psychology: A critical assessment (pp. 69–119). London, UK: Academic Press.Google Scholar
Tan, Y.-Y., & Castelli, C.
(2013) Intelligibility and attitudes: How American and Singapore English are perceived around the world. English World-Wide, 341, 177–201. DOI logoGoogle Scholar
Isaacs, T., & Trofimovich, P.
(2012) Deconstructing comprehensibility: Identifying the linguistic influences on listeners’ L2 comprehensibility ratings. Studies in Second Language Acquisition, 341, 475–505. DOI logoGoogle Scholar
van Bezooijen, R.
(2002) Aesthetic evaluation of Dutch: Comparisons across dialects, accents and languages. In D. Long & D. Preston (Eds.), Handbook of perceptual dialectology (Vol. 21, pp. 13–30). Amsterdam, The Netherlands: John Benjamins. DOI logoGoogle Scholar
Visser, P. S., & Cooper, J.
(2007) Attitude change. The Sage handbook of social psychology (pp. 197–218). Thousand Oaks, CA: Sage. DOI logoGoogle Scholar
Vissers, C. T. W. M., Virgillito, D., Fitzgerald, D. A., Speckens, A. E. M., Tendolkar, I., Van Oostrom, I., & Chwilla, D. J.
(2010) The influence of mood on the processing of syntactic anomalies: Evidence from P600. Neuropsychologia, 481, 3521–3531. DOI logoGoogle Scholar
Watson, K., & Clark, L.
(2015) Exploring listeners’ real-time reactions to regional accents. Language Awareness, 241, 38–59. DOI logoGoogle Scholar
Wheelan, S., & Kaeser, R. M.
(1997) The influence of task type and designated leaders on developmental patterns in groups. Small Group Research, 281, 94–121. DOI logoGoogle Scholar
Winke, P.
(2013) Rating oral language. In C. A. Chapelle (Ed.), The encyclopedia of applied linguistics (pp. 1–7). Hoboken, NJ: Blackwell. DOI logoGoogle Scholar
Winke, P., & Gass, S.
(2013) The influence of second language experience and accent familiarity on oral proficiency rating: A qualitative investigation. TESOL Quarterly, 471, 762–789. DOI logoGoogle Scholar
Winke, P., Gass, S., & Myford, C.
(2013) Raters’ L2 background as a potential source of bias in rating oral performance. Language Testing, 301, 231–252. DOI logoGoogle Scholar
Xu, W., Wang, Y., & Case, R. E.
(2010) Chinese attitudes towards varieties of English: A pre-Olympic examination. Language Awareness, 191, 249–260. DOI logoGoogle Scholar
Yao, Z., Saito, K., Trofimovich, P., & Isaacs, T.
(2013) Z-Lab [Computer software]. [URL]
Cited by

Cited by 1 other publications

Trofimovich, Pavel, Oguzhan Tekin & Kim McDonough
2021. Task engagement and comprehensibility in interaction. Journal of Second Language Pronunciation 7:3 ► pp. 435 ff. DOI logo

This list is based on CrossRef data as of 23 april 2022. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.