Rater variation in the assessment of speech acts

Naoko Taguchi


This study addresses variability among native speaker raters who evaluated pragmatic performance of learners of English as a foreign language. Using a five-point rating scale, four native English speakers of mixed cultural background (one African American, one Asian American, and two Australians) assessed the appropriateness of two types of speech acts (requests and opinions) produced by 48 Japanese EFL students. To explore norms and the reasoning behind the raters’ assessment practice, individual introspective verbal interviews were conducted. Eight students’ speech act productions (64 speech acts in total) were selected randomly, and the raters were asked to rate each speech act and then explain their rating decision. Interview data revealed similarities and differences in their use of pragmatic norms and social rules in evaluating appropriateness.

Quick links
A browser-friendly version of this article is not yet available. View PDF
Austin, J.L
(1962) How to do things with words. Cambridge: Harvard University Press.Google Scholar
Bachman, Lyle, and Adrian Palmer
(1996) Language Testing in Practice: Designing and Developing Useful Language Tests. Oxford: Oxford University Press.Google Scholar
Brown, Annie
(2000) An investigation of rater's orientation in awarding scores in the IELTS interview. In R. Tulloch (ed.), IELTS Research Report 3: 49-84.Google Scholar
(2003) Interviewer variation and the co-construction of speaking proficiency. Language Testing 20: 1-25. CrossrefGoogle Scholar
(2005) Interviewer Variability in Oral Proficiency Interviews. Frankfurt am Main: Peter Lang.Google Scholar
Brown, James D
(2001) Six types of pragmatics tests in two different contexts. In K. Rose & G. Kasper (eds.), Pragmatics in Language Teaching. Cambridge: Cambridge University Press, pp. 301-325. CrossrefGoogle Scholar
Clark, H.H
(1979) Responding to indirect speech acts. Cognitive Psychology 11: 430-477. CrossrefGoogle Scholar
Cohen, Andrew
(1994) Assessing Language Ability in the Classroom. Rowley, MS: Newbury House.Google Scholar
Cohen, Andrew, and Elite Olshtain
(1981) Developing a measure of socio-cultural competence: The case of apology. Language Learning 31: 113-134. CrossrefGoogle Scholar
Davies, Alan
(2003) The native speaker in applied linguistics. In A. Davies & K. Elder (eds.), Handbook of Applied Linguistics. New York: Blackwell, pp. 431-450.Google Scholar
Ducassee, Ana M
(2009) Assessing paired orals: Raters' orientation to interaction. Language Testing 26: 423-443. CrossrefGoogle Scholar
Green, A
(1998) Verbal Protocol Analysis in Language Testing Research: A Handbook (vol.5). Cambridge: Cambridge University Press.Google Scholar
Hoffman-Hicks, Sheila
(1992) Linguistic and pragmatic competence: Their relationship in the overall competence of the language learner. In L.F. Bouton & Y. Kachru (eds.), Pragmatics and Language Learning Monograph Series Vol.3. Urbana-Champaign, IL: University of Illinois, pp. 66-80.Google Scholar
Hudson, Thom, Emily Detmer, and James D. Brown
(1995) Developing Prototypic Measures of Cross- Cultural Pragmatics (Technical Report No.7). Honolulu, HI: University of Hawai'I at Manoa, Second Language Teaching & Curriculum Center.Google Scholar
Johnson, Jeff, and Gad Lim
(2009) The influence of rater language background on writing performance assessment. Language Testing 26: 485-505. CrossrefGoogle Scholar
Levinson, Stephen
(1983) Pragmatics. Cambridge: Cambridge University Press.  BoP. CrossrefGoogle Scholar
Liu, Jianda
(2006) Measuring Interlanguage Pragmatic Knowledge of EFL Learners. Frankfurt: Peter Lang.Google Scholar
May, Lyn
(2007) Interaction in a paired speaking test: The rater's perspective. Ph.D. dissertation: The University of Melbourne.
(2009) Co-constructed interaction in a paired speaking test: The rater's perspective. Language Testing 26: 397-421. CrossrefGoogle Scholar
McNamara, Tim, and Tom Lumley
(1997) The effect of interlocutor and assessment mode variables in overseas assessments of speaking skills in occupational settings. Language Testing 14: 140-156. CrossrefGoogle Scholar
Mey, Jacob
(1993) Pragmatics: An Introduction. Oxford: Blackwell.  BoPGoogle Scholar
Roever, Carston
(2005) Testing EFL Pragmatics. Frankfurt: Peter Lang. CrossrefGoogle Scholar
O'Loughlin, Kieran
(2002) The impact of gender in oral proficiency testing. Language Testing 19: 169-192. CrossrefGoogle Scholar
Poliitt, Alastair, and Neil Murray
(1996) What do raters really pay attention to? In M. Milanovic & N. Saville (eds.), Performance Testing Cognition and Assessment: Selected papers from the 15th Language Testing Research Colloquium. Cambridge: Cambridge University Press, pp. 74-91.Google Scholar
Taguchi, Naoko
(2006) Analysis of appropriateness in a speech act of request in L2 English. Pragmatics 16: 513-535.  BoP CrossrefGoogle Scholar
Thomas, Jenny
(1995) Meaning in Interaction: An Introduction to Pragmatics. London: Longman.  BoPGoogle Scholar
Sasaki, Miyuki
(1998) Investigating EFL students' production of speech acts: A comparison of production questionnaires and role-plays. Journal of Pragmatics 30: 457-484. Crossref  BoPGoogle Scholar
Walter, Scott
(2007) A conversation-analytic hermeneutic rating protocol to assess L2 oral pragmatic competence. Language Testing 24: 155-183. CrossrefGoogle Scholar
Yamashita, Sayoko
(1996) Six Measures of JSL Pragmatics (Technical report #14). Honolulu, HI: University of Hawai'i at Manoa, Second Language Teaching & Curriculum Center.Google Scholar
(2001) Using pictures for research in pragmatics-eliciting pragmatic strategies by picture response tests. In T. Hudson & J.D. Brown (eds.), A Focus on Language Test Development: Expanding the Language Proficiency Construct Across a Variety of Tests. (Technical Report #21). University of Hawaii, Second language Teaching and Curriculum Center, pp. 35-56.Google Scholar
Yoshitake, S
(1997) Measuring interlanguage pragmatic competence of Japanese students of English as a foreign language: A multi-test framework evaluation. Unpublished Ph.D. thesis, Columbia Pacific University, Novata, CA.