Chapter published in:
Understanding L2 Proficiency: Theoretical and meta-analytic investigations
Edited by Eun Hee Jeon and Yo In'nami
[Bilingual Processing and Acquisition 13] 2022
► pp. 109–158

Chapter 5
L2 writing and its internal correlates

A meta-analysis

| Gifu City Women’s University
| Teikyo University of Science

This study examined the overall average correlation between second/foreign language (L2) writing performance and frequently investigated features of writing (i.e., writing-internal correlates). The correlates of L2 writing performance included objective measures of text features (syntactic complexity, lexical complexity, accuracy, fluency, and cohesion) as well as subjective measures (content, argument, organization, cohesion, coherence, grammar, vocabulary, language use, mechanics, and accuracy). A series of moderator analyses were also carried out for each type of objective measure to examine the effects of participants’ age, L2 proficiency, learning context, first language (L1) and L2 distance, task type, writing scoring method, and some characteristics of objective measures. By doing so, the present study aimed to identify key correlates of L2 writing performance and compare their relative importance. To this end, a total of 103 retrieved studies contributed 1,045 effect sizes based on 15,537 independent participants. The results for objective measures demonstrated that fluency had the strongest mean correlation with L2 writing performance (r = .570), followed by accuracy (r = .477), lexical complexity (r = .295), syntactic complexity (r = .271), and cohesion (r = .198). All subjective measure components had strong mean correlations with L2 writing performance (r = .668 to .927), but content and language use features had the strongest effects and cohesion and coherence features showed the least effects. Participants’ age, learning context, L1–L2 distance, writing scoring method, and some measurement characteristics were found to be significant moderators for certain components. The findings of this study have implications for L2 instruction suggesting that fluency and language accuracy of L2 writing should be promoted across the various developmental stages of L2 learners in various conditions, whereas lexical and syntactic competence should be more focused upon when instructing child/adolescent or low/intermediate L2 writers.

Article outline
  • 1.Introduction
  • 2.Background to the meta-analysis
    • 2.1Review of objective measures
      • 2.1.1Syntactic complexity
      • 2.1.2Lexical complexity
      • 2.1.3Accuracy
      • 2.1.4Fluency
      • 2.1.5Cohesion
    • 2.2Review of subjective measures
    • 2.3Review of moderator variables
      • 2.3.1Age
      • 2.3.2L2 proficiency
      • 2.3.3Learning context
      • 2.3.4L1–L2 distance
      • 2.3.5Task type
      • 2.3.6Writing scoring method
      • 2.3.7Measurement types of complexity and fluency
  • 3.Research questions for the meta-analysis
  • 4.Method
    • 4.1Literature search and inclusion criteria
    • 4.2Acceptable measures of L2 writing performance and internal correlates
    • 4.3Coding the primary studies
    • 4.4Research synthesis
  • 5.Results
  • 6.Discussion
    • 6.1Objective measures
      • 6.1.1Syntactic complexity
      • 6.1.2Lexical complexity
      • 6.1.3Accuracy
      • 6.1.4Fluency
      • 6.1.5Cohesion
    • 6.2Subjective measures
  • 7.Conclusion
  • References
Available under the Creative Commons Attribution-NonCommercial-NoDerivatives (CC BY-NC-ND) 4.0 license.

For any use beyond this license, please contact the publisher at rights@benjamins.nl.

Published online: 04 August 2022
https://doi.org/10.1075/bpa.13.05koj
References
Bacha, N.
(2001) Writing evaluation: What can analytic versus holistic essay scoring tell us? System, 29 (3), 371–383. DOI logoGoogle Scholar
Barkaoui, K.
(2010a) Do ESL essay raters’ evaluation criteria change with experience? A mixed-methods, cross-sectional study. TESOL Quarterly, 44 (1), 31–57. DOI logoGoogle Scholar
(2010b) Explaining ESL essay holistic scores: A multilevel modeling approach. Language Testing, 27 (4), 515–535. DOI logoGoogle Scholar
Berman, R., & Verhoeven, L.
(2002) Cross-linguistic perspectives on the development of text-production abilities: Speech and writing. Written Language & Literacy, 5 (1), 1–43. DOI logoGoogle Scholar
Biber, D., Gray, B., & Poonpon, K.
(2011) Should we use characteristics of conversation to measure grammatical complexity in L2 writing development? TESOL Quarterly, 45 (1), 5–35. DOI logoGoogle Scholar
Biber, D., Gray, B., & Staples, S.
(2016) Predicting patterns of grammatical complexity across language exam task types and proficiency levels. Applied Linguistics, 37 (5), 639–668. DOI logoGoogle Scholar
Bouwer, R., Béguin, A., Sanders, T., & Van den Bergh, H.
(2015) Effect of genre on the generalizability of writing scores. Language Testing, 32 (1), 83–100. DOI logoGoogle Scholar
Brodkey, D., & Young, R.
(1981) Composition correctness scores. TESOL Quarterly, 15 (2), 159–167. DOI logoGoogle Scholar
Bulté, B., & Housen, A.
(2014) Conceptualizing and measuring short-term changes in L2 writing complexity. Journal of Second Language Writing, 26 , 42–65. DOI logoGoogle Scholar
Cooper, H.
(2010) Research synthesis and meta-analysis: A step-by-step approach (Vol. 2). Sage.Google Scholar
Crossley, S. A., Kyle, K., Allen, L. K., Guo, L., & McNamara, D. S.
(2014) Linguistic microfeatures to predict L2 writing proficiency: A case study in automated writing evaluation. The Journal of Writing Assessment, 7 (1), 1–34.Google Scholar
Crossley, S. A., Kyle, K., & McNamara, D. S.
(2016a) The development and use of cohesive devices in L2 writing and their relations to judgments of essay quality. Journal of Second Language Writing, 32 , 1–16. DOI logoGoogle Scholar
(2016b) The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion. Behavior Research Methods, 48 (4), 1227–1237. DOI logoGoogle Scholar
Crossley, S. A., & McNamara, D. S.
(2012) Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading, 35 (2), 115–135. DOI logoGoogle Scholar
Crossley, S. A., Salsbury, T., & McNamara, D. S.
(2012) Predicting the proficiency level of language learners using lexical indices. Language Testing, 29 (2), 243–263. DOI logoGoogle Scholar
(2015) Assessing lexical proficiency using analytic ratings: A case for collocation accuracy. Applied Linguistics, 36 (5), 570–590. DOI logoGoogle Scholar
De Jong, N. H., Steinel, M. P., Florijn, A. F., Schoonen, R., & Hulstijn, J. H.
(2012) Facets of speaking proficiency. Studies in Second Language Acquisition, 34 (1), 5–34. DOI logoGoogle Scholar
Engber, C. A.
(1992) A study of lexis and the relationship to quality in written texts of second language learners of English (Doctoral dissertation). Retrieved from ProQuest Dissertations & Theses Global. (303986307)
(1995) The relationship of lexical proficiency to the quality of ESL compositions. Journal of Second Language Writing, 4 (2), 139–155. DOI logoGoogle Scholar
Evola, J., Mamer, E., & Lentz, B.
(1980) Discrete-point versus global scoring for cohesive devices. In J. W. Oiler & K. Perkins (Eds.), Research and language testing (pp. 177–181). Newbury House.Google Scholar
Flahive, D. E., & Snow, B. G.
(1980) Measures of syntactic complexity in evaluating ESL compositions. In J. W. Oiler & K. Perkins (Eds.), Research and language testing (pp. 171–176). Newbury House.Google Scholar
Grabe, W., & Kaplan, R. B.
(1996) Theory and practice of writing: An applied linguistic perspective. Longman.Google Scholar
Guo, L., Crossley, S. A., & McNamara, D. S.
(2013) Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 18 (3), 218–238. DOI logoGoogle Scholar
Halliday, M. A. K., & Hasan, R.
(1976) Cohesion in English. Longman.Google Scholar
Hinkel, E.
(2002) Second language writers’ text: Linguistic and rhetorical features. Routledge. DOI logoGoogle Scholar
Housen, A., Kuiken, F., & Vedder, I.
(2012) Complexity, accuracy and fluency: Definitions, measurement and research. In A. Housen, F. Kuiken, & F. Vedder (Eds.), Dimensions of L2 performance and proficiency: Complexity, accuracy and fluency in SLA (pp. 1–19). John Benjamins. DOI logoGoogle Scholar
Hunt, K. W.
(1965) Grammatical structures written at three grade levels (Vol. 3). National Council of Teachers of English.Google Scholar
Jacobs, H. L., Zinkgraf, S. A., Wormuth, D. R., Hartfiel, V. F., & Hughey, J. B.
(1981) Testing ESL composition: A practical approach. Newbury House.Google Scholar
Jak, S.
(2015) Meta-analytic structural equation modelling. Springer. DOI logoGoogle Scholar
Jarvis, S.
(2002) Short texts, best-fitting curves and new measures of lexical diversity. Language Testing, 19 (1), 57–84. DOI logoGoogle Scholar
(2013) Capturing the diversity in lexical diversity. Language Learning, 63 (S1), 87–106. DOI logoGoogle Scholar
Jeon, E.-H., & Yamashita, Y.
(2014) L2 reading comprehension and its correlates: A meta-analysis. Language Learning, 64 (1), 160–212. DOI logoGoogle Scholar
Kaczmarek, C. M.
(1980) Scoring and rating essay tasks. In J. W. Oiler & K. Perkins (Eds.), Research and language testing (pp. 151–159). Newbury House.Google Scholar
Kim, M., & Crossley, S. A.
(2018) Modeling second language writing quality: A structural equation investigation of lexical, syntactic, and cohesive features in source-based and independent writing. Assessing Writing, 37 , 39–56. DOI logoGoogle Scholar
Kim, M., Crossley, S. A. & Kyle, K.
(2018) Lexical sophistication as a multidimensional phenomenon: Relations to second language lexical proficiency, development, and writing quality. The Modern Language Journal, 102 (1), 120–141. DOI logoGoogle Scholar
Kim, Y. A., & Na, Y. H.
(2009) Cohesive devices and quality of argumentative writing produced by Korean EFL learners. Studies in English Education, 14 (2), 1–29.Google Scholar
Koda, K.
(2005) Insights into second language reading: A cross-linguistic approach. Cambridge University Press. DOI logoGoogle Scholar
Koda, K., & Yamashita, J.
(2019) Reading to learn in a foreign language: An integrated approach to foreign language instruction and assessment. Routledge.Google Scholar
Kojima, M.
(2020) A study synthesis on the relationship between second language writing performance and text features: Focusing on text-based measures and study features. Learner Corpus Studies in Asia and the World, 5 , 1–24. DOI logoGoogle Scholar
Kojima, M., & Kaneta, T.
(2020) Raitingu hyoka to gengoteki shihyo no kankei: Meta bunseki ni yoru kenkyu seika no togo [The relationship between writing performance and linguistic indices: A meta-analysis]. In Y. Ishii & Y. Kondo (Eds.), Eigo Kyoiku ni okeru jido saiten: Genjo to kadai [Automated scoring in English language education: Its current situation and issues] (pp. 33–72). Hituzi Shobo.Google Scholar
Kroll, B.
(1982) Levels of error in ESL composition (Doctoral dissertation). Retrieved from ProQuest Dissertations & Theses Global. (303230742)
(1990) What does time buy? ESL student performance on home versus class compositions. In B. Kroll (Ed.), Second language writing: Research insights for the classroom (pp. 140–154). Cambridge University Press. DOI logoGoogle Scholar
Kyle, K., & Crossley, S.
(2016) The relationship between lexical sophistication and independent and source-based writing. Journal of Second Language Writing, 34 , 12–24. DOI logoGoogle Scholar
(2017) Assessing syntactic sophistication in L2 writing: A usage-based approach. Language Testing, 34 (4), 513–535. DOI logoGoogle Scholar
Kyle, K., & Crossley, S. A.
(2018) Measuring syntactic complexity in L2 writing using fine-grained clausal and phrasal indices. The Modern Language Journal, 102 (2), 333–349. DOI logoGoogle Scholar
Larsen-Freeman, D.
(2009) Adjusting expectations: The study of complexity, accuracy, and fluency in second language acquisition. Applied Linguistics, 30 (4), 579–589. DOI logoGoogle Scholar
Laufer, B., & Nation, P.
(1995) Vocabulary size and use: Lexical richness in L2 written production. Applied Linguistics, 16 (3), 307–322. DOI logoGoogle Scholar
Lee, Y.-W., Gentile, C., & Kantor, R.
(2008) Analytical scoring of TOEFL CBT essays: Scores by humans and e-rater, TOEFL Research Report, 81 , ETS RR-08-01. Educational Testing Service. DOI logoGoogle Scholar
(2010) Toward automated multi-trait scoring of essays: Investigating links among holistic, analytic, and text feature scores. Applied Linguistics, 31 (3), 391–417. DOI logoGoogle Scholar
Leki, I., Cumming, A., & Silva, T.
(2008) A synthesis of research on second language writing in English. Routledge.Google Scholar
Linnarud, M.
(1986) Lexis in composition: A performance analysis of Swedish learners’ written English. CWK Gleerup.Google Scholar
Liu, M., & Braine, G.
(2005) Cohesive features in argumentative writing produced by Chinese undergraduates. System, 33 (4), 623–636. DOI logoGoogle Scholar
Lu, X., & Ai, H.
(2015) Syntactic complexity in college-level English writing: Differences among writers with diverse L1 backgrounds. Journal of Second Language Writing, 29 , 16–27. DOI logoGoogle Scholar
Luk, Z. P. S., & Shirai, Y.
(2009) Is the acquisition order of grammatical morphemes impervious to L1 knowledge? Evidence from the acquisition of plural -s, articles, and possessive’s. Language Learning, 59 (4), 721–754. DOI logoGoogle Scholar
Malvern, D., & Richards, B.
(2002) Investigating accommodation in language proficiency interviews using a new measure of lexical diversity. Language Testing, 19 (1), 85–104. DOI logoGoogle Scholar
Martínez, A. C. L.
(2018) Analysis of syntactic complexity in secondary education EFL writers at different proficiency levels. Assessing Writing, 35 , 1–11. DOI logoGoogle Scholar
McCutchen, D., & Perfetti, C. A.
(1982) Coherence and connectedness in the development of discourse production. Text-Interdisciplinary Journal for the Study of Discourse, 2 (1–3), 113–140. DOI logoGoogle Scholar
Meara, P., & Bell, H.
(2001) P_Lex: A simple and effective way of describing the lexical characteristics of short L2 texts. Prospect, 16 (3), 5–19.Google Scholar
Murakami, A., & Alexopoulou, T.
(2016) L1 influence on the acquisition order of English grammatical morphemes: A learner corpus study. Studies in Second Language Acquisition, 38 (3), 365–401. DOI logoGoogle Scholar
Norris, J. M., & Ortega, L.
(2000) Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning, 50 (3), 417–528. DOI logoGoogle Scholar
(2009) Towards an organic approach to investigating CAF in instructed SLA: The case of complexity. Applied Linguistics, 30 (4), 555–578. DOI logoGoogle Scholar
Oh, E., Lee, C. M., & Moon, Y. I.
(2015) The contributions of planning, L2 linguistic knowledge and individual differences to L2 writing. The Journal of Asia TEFL, 12 (2), 45–85.Google Scholar
Oh, S.
(2006) Investigating the relationship between fluency measures and second language writing placement test decisions (Unpublished master’s thesis). University of Hawai’i. Retrieved from http://​hdl​.handle​.net​/10125​/20203
Ortega, L.
(2003) Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college-level L2 writing. Applied Linguistics, 24 (4), 492–518. DOI logoGoogle Scholar
(2015) Syntactic complexity in L2 writing: Progress and expansion. Journal of Second Language Writing, 29 , 82–94. DOI logoGoogle Scholar
Pallotti, G.
(2009) CAF: Defining, refining and differentiating constructs. Applied Linguistics, 30 (4), 590–601. DOI logoGoogle Scholar
Park, S. K.
(2013) Lexical analysis of Korean university students’ narrative and argumentative essays. English Teaching (영어교육), 68 (3), 131–157.Google Scholar
Perkins, K.
(1980) Using objective methods of attained writing proficiency to discriminate among holistic evaluations. TESOL Quarterly, 14 (1), 61–69. DOI logoGoogle Scholar
(1983) On the use of composition scoring techniques, objective measures, and objective tests to evaluate ESL writing ability. TESOL Quarterly, 17 (4), 651–671. DOI logoGoogle Scholar
Plonsky, L., & Oswald, F. L.
(2014) How big is “big”? Interpreting effect sizes in L2 research. Language Learning, 64 (4), 878–912. DOI logoGoogle Scholar
Read, J.
(2000) Assessing vocabulary. Cambridge University Press. DOI logoGoogle Scholar
Roca de Larios, J., Manchón, R., Murphy, L., & Marín, J.
(2008) The foreign language writer’s strategic behaviour in the allocation of time to writing processes. Journal of Second Language Writing, 17 (1), 30–47. DOI logoGoogle Scholar
Rosenthal, R.
(1979) The “file drawer problem” and tolerance for null results. Psychological Bulletin, 86 , 638–641. DOI logoGoogle Scholar
Sasaki, M.
(2009) Changes in English as a foreign language students’ writing over 3.5 years: A sociocognitive account. In R. M. Manchón (Ed.), Writing in foreign language contexts: Learning, teaching, and research (pp. 49–76). Multilingual Matters. DOI logoGoogle Scholar
Schoonen, R., Van Gelderen, A., De Glopper, K., Hulstijn, J., Simis, A., Snellings, P., & Stevenson, M.
(2003) First language and second language writing: The role of linguistic knowledge, speed of processing, and metacognitive knowledge. Language Learning, 53 (1), 165–202. DOI logoGoogle Scholar
Schoonen, R., Van Gelderen, A., Stoel, R. D., Hulstijn, J., & De Glopper, K.
(2011) Modeling the development of L1 and EFL writing proficiency of secondary school students. Language Learning, 61 (1), 31–79. DOI logoGoogle Scholar
Skehan, P.
(1998) A cognitive approach to language learning. Oxford University Press.Google Scholar
(2009) Modelling second language performance: Integrating complexity, accuracy, fluency, and lexis. Applied Linguistics, 30 (4), 510–532. DOI logoGoogle Scholar
Stoller, F. L.
(2004) Content-based instruction: Perspectives on curriculum planning. Annual Review of Applied Linguistics, 24 , 261–283. DOI logoGoogle Scholar
Thewissen, J.
(2013) Capturing L2 accuracy developmental patterns: Insights from an error-tagged EFL learner corpus. The Modern Language Journal, 97 (S1), 77–101. DOI logoGoogle Scholar
Verspoor, M., Schmid, M. S., & Xu, X.
(2012) A dynamic usage based perspective on L2 writing. Journal of Second Language Writing, 21 (3), 239–263. DOI logoGoogle Scholar
Viechtbauer, W.
(2010) Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36 (3), 1–48. DOI logoGoogle Scholar
Weigle, S. C.
(2002) Assessing Writing. Cambridge University Press. DOI logoGoogle Scholar
White, E. M.
(1984) Holisticism. College Composition and Communication, 35 (4), 400–409. DOI logoGoogle Scholar
Wolfe-Quintero, Y., Inagaki, S., & Kim, H.-Y.
(1998) Second language development in writing: Measures of fluency, accuracy, & complexity. University of Hawai’i Press.Google Scholar
Yang, W., Lu, X., & Weigle, S. C.
(2015) Different topics, different discourse: Relationships among writing topic, measures of syntactic complexity, and judgments of writing quality. Journal of Second Language Writing, 28 , 53–67. DOI logoGoogle Scholar
Yde, P., & Spoelders, M.
(1985) Text cohesion: An exploratory study with beginning writers. Applied Psycholinguistics, 6 (4), 407–415. DOI logoGoogle Scholar
Yu, G.
(2010) Lexical diversity in writing and speaking task performances. Applied Linguistics, 31 (2), 236–259. DOI logoGoogle Scholar
Yun, Y.
(2005) Factors explaining EFL learners’ performance in a timed essay writing test: A structural equation modeling approach (Doctoral dissertation). Retrieved from ProQuest Dissertations & Theses Global. (3199191)