Article In:
International Journal of Learner Corpus Research: Online-First Articles

The effect of lexical complexity on grading of Swedish EFL learners’ texts during high-stakes exams

ORCID logo | Luleå University of Technology
The present study concerns the effect of lexical complexity on grading of Swedish EFL learners’ texts during high-stakes exams. A learner corpus consisting of 142 texts graded by expert raters and 175 texts graded by teachers was analysed to establish if the latter graded in agreement with the former as intended by the Swedish National Agency for Education (SNAE). Four indices of lexical complexity available in TAALED and TAALES were chosen to explore if this is the case. The method includes conducting ordinal regression with interactions to determine the effect of the independent variables on grade and if these variables have the same effect in texts graded by teachers and expert raters. The findings reveal a discrepancy between expert raters and teachers as they appear to consider lexical complexity to a different extent. It was also found that expert raters and teachers graded more in agreement during source-based writing tasks compared to independent writing tasks.
Keywords: writing assessment, lexical complexity, high-stakes exams, inter-rater reliability
Article outline
  • 1.Introduction and aim
  • 2.Background
    • 2.1Lexical complexity and assessment of writing proficiency
    • 2.2Previous research on the assessment of Swedish EFL students
    • 2.3English as a subject in Swedish upper secondary school and the national tests
  • 3.Material and method
    • 3.1Material
    • 3.2Method
      • 3.2.1Lexical sophistication indices
      • 3.2.2Lexical diversity index
    • 3.3Statistical analysis
  • 4.Results
    • 4.1English 5
      • 4.1.1The effect of lexical complexity on grading
    • 4.2English 6
      • 4.2.1The effect of lexical complexity on grading
  • 5.Discussion
  • 6.Conclusion
  • Data availability statement
  • Notes
  • References
This content is being prepared for publication; it may be subject to changes.
References (50)
References
Bergström, D., Norberg, C., & Nordlund, M. (2021). “Words are picked up along the way”– Swedish EFL teachers’ conceptualizations of vocabulary knowledge and learning. Language Awareness, 31 1, 393–409. DOI logoGoogle Scholar
Bottini, R., & Le Foll, E. (in press). The more proficient the learners, the less sophisticated their L2 vocabulary? The curious effect of the reference corpus on mean-frequency measures of lexical sophistication in written and spoken production. International Journal of Learner Corpus Research. Preprint available on: [URL]
Brysbaert, M., New, B., & Keuleers, E. (2012). Adding part-of-speech information to the SUBTLEX-US word frequencies. Behavior Research Methods, 44 1, 991–997. DOI logoGoogle Scholar
Christensen, R. (2023). ordinal — Regression Models for Ordinal Data. R package version, 2023.12-4.1. [URL]
Covington, M. A., & McFall, J. D. (2010). Cutting the gordian knot: The moving-average type–token ratio (MATTR). Journal of Quantitative Linguistics, 17 (2), 94–100. DOI logoGoogle Scholar
Crossley, S. A., Roscoe, R., & McNamara, D. S. (2014). What is successful writing? An investigation into the multiple ways writers can write successful essays. Written Communication, 31 (2), 184–214. DOI logoGoogle Scholar
Council of Europe. (2001). Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge University Press. [URL]
Cumming, A., Kantor, R., Baba, K., Erdosy, U., Eouanzoui, K., & James, M. (2005). Differences in written discourse in independent and integrated prototype tasks for next generation TOEFL. Assessing Writing, 10 (1), 5–43. DOI logoGoogle Scholar
Educational Testing Service. (2023–05–30). Writing Scoring Guide. TOEFL iBT® Writing Scoring Guide Flyer (ets.org)Google Scholar
Egbert, J., Larsson, T., & Biber, D. (2020). Doing linguistics with a corpus: Methodological considerations for the everyday user. Cambridge University Press. DOI logoGoogle Scholar
Egbert, J., Burch, B., & Biber, D. (2020). Lexical dispersion and corpus design. International Journal of Corpus Linguistics, 25 (1), 89–115. DOI logoGoogle Scholar
Egbert, J., & Burch, B. (2023). Which words matter most? Operationalizing lexical prevalence for rank-ordered word lists. Applied Linguistics, 44 (1), 103–126. DOI logoGoogle Scholar
Eguchi, M., & Kyle, K. (2020). Continuing to explore the multidimensional nature of lexical sophistication: The case of oral proficiency interviews. The Modern Language Journal, 104 (2), 381–400. DOI logoGoogle Scholar
Engber, C. A. (1995). The relationship of lexical proficiency to the quality of ESL compositions. Journal of Second Language Writing, 4 (2), 139–155. DOI logoGoogle Scholar
Erickson, G., Kao, T., & Lin, Y. (2010). Good practice in language testing and assessment: A matter of responsibility and respect. A new look at teaching and testing: English as subject and vehicle. Selected papers from the 2009 LTTC International Conference on English Language Teaching and Testing (pp. 237–258). University of Gothenburg.Google Scholar
Erickson, G. (2020). National assessment of foreign languages in Sweden. University of Gothenburg. [URL]
Erickson, G., & Tholin, J. (2022). Overall, a good test, but… — Swedish lower secondary teachers’ perceptions and use of national test results of English. Languages, 7 (1), 64. DOI logoGoogle Scholar
Fox, J., Weisberg, S., Price, B., Adler, D., Bates, D., Baud-Bovy, G., Bolker, B., Ellison, S., Firth, D., Friendly, M., Gorjanc, G., Graves, S., Heidelberger, R., Krivitsky, P., Laboissiere, R., Maechler, M., Monette, G., Murdoch, D., Nilsson, H., … & R-Core. (2012). Package ‘car’ version, 2024 3.1–3. Vienna: R Foundation for Statistical Computing, 16 (332), 333. [URL]
Guo, L., Crossley, S. A., & McNamara, D. S. (2013). Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 18 (3), 218–238. DOI logoGoogle Scholar
Hashimoto, B. J., & Egbert, J. (2019). More than frequency? Exploring predictors of word difficulty for second language learners. Language Learning, 69 (4), 839–872. DOI logoGoogle Scholar
IELTS. (2023–05–30). Writing Band Descriptors. Writing Band Descriptors (ielts.org)Google Scholar
Jarvis, S. (2002). Short texts, best-fitting curves and new measures of lexical diversity. Language Testing, 19 (1), 57–84. DOI logoGoogle Scholar
Kaatari, H., Wang, Y., & Larsson, T. (2024). Introducing the Swedish Learner English Corpus: A corpus that enables investigations of the impact of extramural activities on L2 writing. Corpora, 19 (1), 17–30. DOI logoGoogle Scholar
Kim, M., Crossley, S. A., & Kyle, K. (2018). Lexical sophistication as a multidimensional phenomenon: Relations to second language lexical proficiency, development, and writing quality. The Modern Language Journal, 102 (1), 120–141. DOI logoGoogle Scholar
Kleiber, C., Zeileis, A., & Zeileis, M. A. (2020). R package ‘aer’ version, 1.2–14. AER: Applied Econometrics with R (r-project.org)Google Scholar
Kuperman, V., Stadthagen-Gonzalez, H., & Brysbaert, M. (2012). Age-of-acquisition ratings for 30,000 English words. Behavior Research Methods, 44 1, 978–990. DOI logoGoogle Scholar
Kyle, K. (2022). Measuring lexical richness. In S. Webb (Ed.), The Routledge Handbook of Vocabulary Studies (pp. 454–476). Routledge.Google Scholar
Kyle, K., & Crossley, S. A. (2015). Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Quarterly, 49 (4), 757–786. DOI logoGoogle Scholar
Kyle, K., & Crossley, S. (2016). The relationship between lexical sophistication and independent and source-based writing. Journal of Second Language Writing, 34 1, 12–24. DOI logoGoogle Scholar
Kyle, K., Crossley, S., & Berger, C. (2018). The tool for the automatic analysis of lexical sophistication (TAALES): version 2.0. Behavior Research Methods, 50 1, 1030–1046. DOI logoGoogle Scholar
Kyle, K., Crossley, S. A., & Jarvis, S. (2021). Assessing the validity of lexical diversity indices using direct judgements. Language Assessment Quarterly, 18 (2), 154–170. DOI logoGoogle Scholar
Laufer, B., & Nation, P. (1995). Vocabulary size and use: Lexical richness in L2 written production. Applied Linguistics, 16 (3), 307–322. DOI logoGoogle Scholar
Mangiafico, S. (2024). rcompanion: Functions to support extension education program evaluation. Rutgers cooperative xxtension, New Brunswick, New Jersey. version 2.4.36, [URL]
Plakans, L., & Gebril, A. (2013). Using multiple texts in an integrated writing assessment: Source text use as a predictor of score. Journal of Second Language Writing, 22 (3), 217–230. DOI logoGoogle Scholar
R Development Core Team. (2023). R: A language and environment for statistical computing. [Computer software]. Vienna, Austria: R Foundation for Statistical Computing. [URL]
Read, J. (2000). Assessing Vocabulary. Cambridge University Press. DOI logoGoogle Scholar
Schlegel, B. E., Steenbergen, M., Schlegel, M. B., & Imports, M. A. S. S. (2018). R Package brant, version 0.3–0. brant: Test for Parallel Regression Assumption (r-project.org)Google Scholar
Schoonen, R., Van Gelderen, A., Stoel, R. D., Hulstijn, J., & De Glopper, K. (2011). Modeling the development of L1 and EFL writing proficiency of secondary school students. Language Learning, 61 (1), 31–79. DOI logoGoogle Scholar
Skolverket. (2022). Ämnesplan i Engelska. Skolverket. [URL]
University of Gothenburg. (2023–05–30a). Översikt — Skriftlig produktion och interaktion, Engelska 5. Exempel på uppgiftstyper för Engelska 5 | Projektet Nationella prov i främmande språk — Nafs, Göteborgs universitet (gu.se)Google Scholar
. (2023–05–30b). Exempel på texter bedömda som E, Engelska 5. Exempel på uppgiftstyper för Engelska 5 | Projektet Nationella prov i främmande språk — Nafs, Göteborgs universitet (gu.se)Google Scholar
Venables, W. N., & Ripley, B. D. (2002). Modern Applied Statistics with S. (4th ed.) Springer. DOI logoGoogle Scholar
Vögelin, C., Jansen, T., Keller, S. D., Machts, N., & Möller, J. (2019). The influence of lexical features on teacher judgements of ESL argumentative essays. Assessing Writing, 39 1, 50–63. DOI logoGoogle Scholar
Warnby, M. (2022). Receptive academic vocabulary knowledge and extramural English involvement — is there a correlation?. ITL-International Journal of Applied Linguistics, 173 (1), 120–152. DOI logoGoogle Scholar
Yang, W. (2014). Mapping the relationships among the cognitive complexity of independent writing tasks, L2 writing quality, and complexity, accuracy and fluency of L2 writing [Doctoral dissertation, Georgia State University]. ScholarWorks@Georgia State University. DOI logo
Yang, W., Lu, X., & Weigle, S. C. (2015). Different topics, different discourse: Relationships among writing topic, measures of syntactic complexity, and judgments of writing quality. Journal of Second Language Writing, 28 1, 53–67. DOI logoGoogle Scholar
Yang, W., & Kim, Y. (2020). The effect of topic familiarity on the complexity, accuracy, and fluency of second language writing. Applied Linguistics Review, 11 (1), 79–108. DOI logoGoogle Scholar
Yoon, H. J. (2017). Linguistic complexity in L2 writing revisited: Issues of topic, proficiency, and construct multidimensionality. System, 66 1, 130–141. DOI logoGoogle Scholar
Yu, G. (2010). Lexical diversity in writing and speaking task performances. Applied Linguistics, 31 (2), 236–259. DOI logoGoogle Scholar
Zenker, F., & Kyle, K. (2021). Investigating minimum text lengths for lexical diversity indices. Assessing Writing, 47 1, 100505. DOI logoGoogle Scholar