Article published In:
Dutch Journal of Applied LinguisticsVol. 3:1 (2014) ► pp.101–108
Raters’ social considerations in the essay rating process
The case of Chinese assessors of high-stake exam essays written in English
The social cognitive view on essay rating process argues that human essay rating is constrained by a series of the measures issued by the test institution. However, studies of the institutional constraints all focused on text quality relevant factors interfering into the rating process. Text quality irrelevant social factors remain uninvestigated. Taking the TEM8 (a test for English majors in China) as an example, the present study explores into those factors. Raters’ think aloud protocols when rating essays and the follow up interviews identified a number of text quality irrelevant social factors raters bring into consideration when scoring essays, such as institutional awareness, test knowledge, test taker expectation, knowledge of rating system, ethical consideration and physical condition. The way those factors influence the rating process is further discussed. The results are meaningful both for the understanding of the essay rating construct and human essay rating practice.
Keywords: essay rating, raters, social factors
References
Cumming, A., Kantor, R., & Powers, D. (
2002) Decision making while rating ESL/EFL writing tasks: A descriptive framework. Modern Language Journal, 861, 67–96. 
Davies, A. (
1997) Introduction: The limits of ethics in language testing. Language Testing, 141, 235–241.
DeRemer, M. (
1998) Writing assessment: Raters’ elaboration of the rating task. Assessing Writing, 51, 7–29.
Eckes, T. (
2008) Rater types in writing performance assessments: A classification approach to rater variability. Language Testing, 25(2), 155–185.
Freedman, S.W. (
1981) Influences of evaluation of expository essays: Beyond the text. Research in the Teaching of English, 151, 245–255.
Freedman, S.W., & Calfee, R.C. (
1983) Holistic assessment of writing: Experimental design and cognitive theory. In P. Mosenthal, L. Tamor, & S.A. Walmsley (Eds.), Research on writing: Principles and methods (pp. 75–98). New York: Longman.
Hunt, K. (
1965) Grammatical structures written at three grade levels. (NCTE Research report No. 3). Champaign, IL, USA: NCTE.
Jin, Y., & Fan, J. (
2011) Test for English Majors (TEM) in China. Language Testing, 28(4), 589–596.
Knoch, U. (
2009) Diagnostic assessment of writing: A comparison of two rating scales. Language Testing, 26(2), 275–304.
Lim, G.S. (
2011) The development and maintenance of rating quality in performance writing assessment: A longitudinal study of new and experienced raters. Language Testing, 281, 543–560.
Lumley, T. (
2002) Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19(3), 246–276.
2005) Assessing second language writing: The rate’s perspective. Frankfurt am Main: Peter Lang.
Neilsen, L., & Piche, G. (
1981) The influence of headed nominal complexity and lexical choice on teachers’ evaluation of writing. Research in the Teaching of English, 151, 65–74.
O’Sullivan, B., & Rignall, M. (
2007) Assessing the value of bias analysis feedback to raters for the IELTS writing module. In L. Taylor, & P. Falvey (Eds.), IELTS collected papers (pp. 446–476). Cambridge: Cambridge University Press.
Sakyi, A. (
2000) Validation of holistic scoring for ESL writing assessment: A study of how raters evaluate ESL compositions on a holistic scale. In A.J. Kunnan (Ed.), Fairness and validation in language assessment (pp. 130–153). Cambridge: Cambridge University Press.
Shohamy, E. (
2001) The power of tests: A critical perspective on the uses of language tests. London: Longman.
Stewart, M., & Grobe, C. (
1979) Syntactic maturity, mechanics of writing, and teachers’ quality ratings. Research in the Teaching of English, 131, 207–215.
Vaughan, C. (
1991) Holistic assessment: What goes on in the rater’s mind? In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 111–125). Norwood, NJ: Ablex.
Veal, L.R. (
1974) Syntactic measures and rated quality in the writing of young children. (Studies in Language Education, Report No. 8). Athens: University of Georgia. (ERIC Document Reproduction Service No. 090 55).
Zou, S. (
2011) On enhancing test fairness: The case of the TEM4 and TEM8. Foreign Language Testing and Teaching, (1), 42–50.