Cover not available
Article published In:
Chinese Language and Discourse
Vol. 11:2 (2020) ► pp.169–197

Incoherence in L2 writing

A comparison of expert insider and non-expert outsider ratings

| Arizona State University
Current knowledge of the functional dimensions (e.g., coherence) of L2 written performance is mainly based on expert readers’ views. Non-expert native readers’ perspectives of L2 written productions are not often examined, which prevents a comprehensive and objective understanding of how an L2 writer’s performance may be perceived and evaluated by the target discourse community. Studies to date also lack clear findings of both the types of incoherence phenomena that may exist in L2 texts and the factors that may contribute to such incoherencies. The current study investigates how expert and non-expert native readers evaluate incoherence in L2 Chinese argumentative essays, as well as how their evaluations may differ quantitatively or qualitatively. The findings reveal that although expert readers marked incoherence in L2 writing significantly more frequently than non-expert readers did, expert and non-expert readers displayed similar patterns and tendencies in their judgments pertaining to the frequency with which incoherence instances appeared in the essays and in the severity of the incoherence.
Keywords: coherence, cohesion, argumentative essay, L2 Chinese, expert rater, non-expert rater
Article outline
  • 1.Introduction
  • 2.Literature review
    • 2.1Conceptualizations of L2 written coherence
    • 2.2Expert and non-expert raters’ judgments of L2 written coherence
    • 2.3Development of L2 written coherence
  • 3.The present study
    • 3.1Dataset
    • 3.2Raters
    • 3.3Rating
    • 3.4Analysis
  • 4.Results
    • 4.1Judgments of frequency and severity of incoherence by the expert and non-expert raters
    • 4.2Reasons for incoherence by the expert and non-expert raters
    • 4.3Inter-rater relationships
  • 5.Discussion
    • 5.1Judgments of frequency and severity of incoherence by the expert and non-expert raters
    • 5.2Reasons for incoherence by the expert and non-expert raters
  • 6.Conclusion and implications
  • Reference
Published online: 24 November 2020
DOI logo
https://doi.org/10.1075/cld.20018.lia
References (45)
Reference
Bardovi-Harlig, K. 1990. Pragmatic word order in English composition. In U. Connor & A. Johns (Eds.), Coherence in writing: Research and pedagogical perspectives (pp. 43–65). Alexandria, VA: TESOL.Google Scholar
Carrell, P. 1982. Cohesion is not coherence. TESOL Quarterly, 161, 479–488. DOI logoGoogle Scholar
Chen, Y., & Baker, P. 2016. Investigating criterial discourse features across second language development: Lexical bundles in rated learner essays, CEFR B1, B2 and C1. Applied Linguistics, 371, 849–880.Google Scholar
Chiang, S. 2003. The importance of cohesive conditions to perceptions of writing quality at the early stages of foreign language learning. System, 311, 471–484. DOI logoGoogle Scholar
Connor, U. 1990. Linguistic/rhetorical measures for international persuasive student writing. Research in the Teaching of English, 241, 67–87.Google Scholar
Crossley, S., Kyle, K., & McNamara, D. 2016a. The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion. Behavior Research Methods, 481, 1227–1237. DOI logoGoogle Scholar
2016b. The development and use of cohesive devices in L2 writing and their relations to judgments of essay quality. Journal of Second Language Writing, 321, 1–16. DOI logoGoogle Scholar
Crossley, S., & McNamara, D. 2011. Shared features of L2 writing: Intergroup homogeneity and text classification. Journal of Second Language Writing, 201, 271–285. DOI logoGoogle Scholar
2012. Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading, 35, 2, 115–135. DOI logoGoogle Scholar
Crossley, S., Yang, H. S., & McNamara, D. 2014. What’s so simple about simplified texts? A computational and psycholinguistic investigation of text comprehension and text processing. Reading in a Foreign Language, 26(1), 92–113.Google Scholar
Cumming, A. 1990. Expertise in evaluating second language compositions. Language Testing, 71, 31–51. DOI logoGoogle Scholar
Eckes, T. 2008. Rater types in writing performance assessments: A classification approach to rater variability. Language Testing, 25(2), 155–185. DOI logoGoogle Scholar
Ferris, D. 1994. Lexical and syntactic features of ESL writing by students at different levels of L2 proficiency. TESOL Quarterly, 281, 414–420. DOI logoGoogle Scholar
Guo, L., Crossley, S., & McNamara, D. 2013. Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 181, 218–238. DOI logoGoogle Scholar
Halliday, M., & Hasan, R. 1976. Cohesion in English. London, UK: Longman.Google Scholar
Harman, R. 2013. Literary intertextuality in genre-based pedagogies: Building lexical cohesion in fifth-grade L2 writing. Journal of Second Language Writing, 221, 125–140. DOI logoGoogle Scholar
Hyland, K. 2005. Metadiscourse: Exploring interaction in writing. London, UK: Continuum.Google Scholar
Jiang, W. 2013. Measurements of development in L2 written production: The case of L2 Chinese. Applied Linguistics, 341, 1–24. DOI logoGoogle Scholar
Kang, Y. J. 2005. Written narratives as an index of L2 competence in Korean EFL learners. Journal of Second Language Writing, 141, 259–279. DOI logoGoogle Scholar
Kennedy, C., & Thorp, D. 2007. A corpus-based investigation of linguistic responses to an IELTS academic writing task. In L. Taylor & P. Falvey (Eds), IELTS collected papers: Research in speaking and writing assessment (pp. 316–377). Cambridge, UK: Cambridge University Press.Google Scholar
Knoch, U. 2007. ‘Little coherence, considerable strain for reader’: A comparison between two rating scales for the assessment of coherence. Assessing Writing, 12(2), 108–128. DOI logoGoogle Scholar
Kormos, J. 2011. Task complexity and linguistic and discourse features of narrative writing performance. Journal of Second Language Writing, 201, 148–161. DOI logoGoogle Scholar
Kuiken, F., & Vedder, I. 2014. Rating written performance: What do raters do and why? Language Testing, 311, 329–348. DOI logoGoogle Scholar
2017. Functional adequacy in L2 writing: Towards a new rating scale. Language Testing, 341, 321–336. DOI logoGoogle Scholar
Kuiken, F., Vedder, I., & Gilabert, R. 2010. Communicative adequacy and linguistic complexity in L2 writing. In I. Bartning, M. Martin & I. Vedder (Eds.), Communicative proficiency and linguistic development: Intersections between SLA and language testing research (pp. 81–99). Eurosla: European Second Language Association.Google Scholar
Li, S. 2014. The gap in the use of lexical cohesive devices in writing between native Chinese speakers and second language users. Journal of the Chinese Language Teachers Association, 49(3), 25–47.Google Scholar
Li, T., & Wharton, S. 2012. Metadiscourse repertoire of L1 Mandarin undergraduates writing in English: A cross-contextual, cross-disciplinary study. Journal of English for Academic Purposes, 111, 345–356. DOI logoGoogle Scholar
Liu, M., & Braine, G. 2005. Cohesive features in argumentative writing produced by Chinese undergraduates. System, 331, 623–636. DOI logoGoogle Scholar
McCulley, G. 1985. Writing quality, coherence, and cohesion. Research in the Teaching of English, 191, 269–282.Google Scholar
McCutchen, D. 1996. A capacity theory of writing: Working memory in composition. Educational Psychology Review, 81, 299–325. DOI logoGoogle Scholar
McNamara, D. 2001. Reading both high and low coherence texts: Effects of text sequence and prior knowledge. Canadian Journal of Experimental Psychology, 551, 51–62. DOI logoGoogle Scholar
McNamara, D., Crossley, S., & McCarthy, P. 2010. Linguistic features of writing quality. Written Communication, 271, 57–86. DOI logoGoogle Scholar
McNamara, D., Crossley, S., & Roscoe, R. 2013. Natural language processing in an intelligent writing strategy tutoring system. Behavior Research Methods, 451, 499–515. DOI logoGoogle Scholar
McNamara, D., & Kintsch, W. 1996. Learning from texts: Effects of prior knowledge and text coherence. Discourse Processes, 22(3), 247–288. DOI logoGoogle Scholar
McNamara, D., Kintsch, E., Songer, N., & Kintsch, W. 1996. Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 141, 1–43. DOI logoGoogle Scholar
Neuner, J. 1987. Cohesive ties and chains in good and poor freshman essays. Research in the Teaching of English, 211, 92–105.Google Scholar
O’Reilly, T., & McNamara, D. 2007. Reversing the reverse cohesion effect: Good texts can be better for strategic, high-knowledge readers. Discourse Processes, 431, 121–152. DOI logoGoogle Scholar
Reid, J. 1992. A computer text analysis of four cohesion devices in English discourse by native and nonnative writers. Journal of Second Language Writing, 11, 79–107. DOI logoGoogle Scholar
Schoonen, R. 2005. Generalizability of writing scores: an application of structural equation modeling. Language Testing, 22(1), 1–30. DOI logoGoogle Scholar
Schoonen, R., Vergeer, M., & Eiting, M. 1997. The assessment of writing ability: Expert readers versus lay readers. Language Testing, 14(2), 157–184. DOI logoGoogle Scholar
Tannen, D. 1984. Spoken and written narrative in English and Greek. In D. Tannen (Ed.). Coherence in spoken and written discourse (pp. 21–41). Norwood, NJ: Ablex.Google Scholar
Weigle, S. C. 2007. Teaching writing teachers about assessment. Journal of Second Language Writing, 16(3), 194–209. DOI logoGoogle Scholar
Witte, S., & Faigley, L. 1981. Coherence, cohesion, and writing ability. College Composition and Communication, 321, 189–204. DOI logoGoogle Scholar
Yang, C. 2013. Textual conjunctives and topic-fronting devices in CFL learners’ written summaries. Journal of the Chinese Language Teachers Association, 48(1), 71–89.Google Scholar
Yang, W., & Sun, Y. 2012. The use of cohesive devices in argumentative writing by Chinese EFL learners at different proficiency levels. Linguistics and Education, 231, 31–48. DOI logoGoogle Scholar