Article published In:
ITL - International Journal of Applied LinguisticsVol. 127/128 (2000) ► pp.37–51
Vol. 127/128 (2000) ► pp.37–51
Assessing Second Language Writing
The Relationship between Computerized Analysis and Rater Evaluation
This article examines the relationship between two kinds of methods used to assess the quality of second language writing : 1) objective computerised text analysis focusing on the linguistic features of written texts, and 2) subjective evaluation performed by human raters using a combination of holistic and analytical scoring procedures. In particular, it attempts to explore the potentials and possible limitations of using computerised programs as research tools in second language writing research.
The written sample consisted of a total of 132 short essays written b y ESL students enrolled in various academic programs at an American university. The first method used computerized programs to assess the written texts in terms of syntactic complexity, lexical complexity and grammatical accuracy, whereas in the second method, two ESL raters evaluated the same sample of texts by first assigning a holistic score to each piece of writing, then applying an analytical scheme to assess linguistic features at the syntactic, lexical and grammatical level as well as textual and rhetorical features at the discourse level. A series of correlation analyses were performed using the scores obtained from these two kinds of assessment procedures at the correspondent levels. The results show that a significant correlation was consistently found between these two kinds of scores at the level of grammatical accuracy, yet n o significant correlation was found in any of the other categories. The results also indicate a high level of internal consistency in t he computerized analysis.
References
BIBER, D.
(1988) : Variations across speech and writing. Cambridge : Cambridge University Press. 
(1996) : Corpus-based investigation of language use. Annual Review of Applied Linguistics 161 : 115–136.
COHEN, A.D.
(1994) : Assessing language ability in the classroom. Boston, Mass : Heinle & Heinle Publishers.
CONNOR-LlNTON, J.
(1995): Looking behind the curtain : What do L2 composition ratings really mean? TESOL Quarterly 29, 4 : 762–765.
CONNOR, U.
(1991): Linguistic/rhetorical measures for evaluating ESL writing. In L. Hamp-Lyons (Ed.), assessing second language writing in academic contexts (pp. 215–25). Norwood, NJ : Ablex.
ELBOW, P.
(1996) : Writing assessment in the 21st century : A Utopian view. In L.Z. Bloom, D.A. Daiker, and E.M. White (eds) Composition in the 21st century : Crisis and change (pp. 83–100). Carbon-dale : Southern Illinois University Press.
FERRIS, D.
(1993) : The design of an automatic analysis program for L2 text research : Necessity and feasibility. Journal of Second Language Writing 2, 2:119–129.
(1994): Lexical and syntactic features of ESL writing by students at different levels of L2 proficiency. TESOL Quarterly, 28, 2:414–20.
HAMP-LYONS, L.
(1991) : Scoring procedures for ESL contexts. In L. Hamp-Lyons (Ed.), Aassessing second language writing in academic contexts (pp. 241–66). Norwood, NJ : Ablex.
(1995) : Rating non-native writing : The trouble with holistic scoring. TESOL Quarterly 29, 4758–762.
HAMP-LYONS, L. & B. KROLL.
(1996) : Issues in ESL writing assessment : An overview. College ESL 61, 11:52–72.
HUNT, K.
(1965) : Grammatical structures written at three grade levels. Champaign, IL : National Council of Teachers of English.
KROLL, B.
(1998) : Assessing Writing Abilities. Annual Review of Applied Linguistics, 181: 219–240
LAUFER, B. & NATION, P.
(1995) : Vocabulary size and use : Lexical richness in L2 written production. Applied Linguistics, 161, 41:307–321.
POLIO, CG.
(1997) : Measures of linguistic accuracy in second language writing research. Language Learning 47, 1:101–143.
SCOTT, M.
(1996): Wordsmith Tools. Cambridge University Press.
VAUGHAN, C.
(1991) : Holistic assessment: What goes on in the raters' minds? In L. Hamp-Lyons (Ed.), assessing second language writing in academic contexts (pp.111–26). Norwood, NJ : Ablex.
Cited by
Cited by 2 other publications
Kim, Ha Ram, Melissa Bowles, Xun Yan & Sun Joo Chung
2018. Examining the comparability between paper- and computer-based versions of an integrated writing placement test. Assessing Writing 36 ► pp. 49 ff. 
Silva, Tony & Colleen Brice
2004. 4. RESEARCH IN TEACHING WRITING. Annual Review of Applied Linguistics 24
This list is based on CrossRef data as of 1 april 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.