Cover not available

Article In: Journal of Second Language Studies: Online-First Articles

Evaluating an LLM as an automated rater

CEFR-based automated essay scoring for German L2 writing in the MERLIN corpus

ORCiD logo with link | Shandong University, China
ORCiD logo with link | Zhejiang University, China
This content is being prepared for publication; it may be subject to changes.
Abstract
Automated essay scoring (AES) has advanced rapidly with the emergence of GPT-based models, yet empirical research has predominantly focused on English, leaving German L2 writing underexplored. To address this gap, this study evaluates GPT-5 as an AES rater for German L2 writing using 200 essays sampled from the MERLIN Corpus, ranging from CEFR levels A1 to C1, with 40 essays per level. GPT-5 ratings were compared with MERLIN human ratings for the overall CEFR rating and six analytic dimensions. Agreement between GPT-5 and human ratings was examined using Quadratic Weighted Kappa (QWK) and Kendall’s tau-b (τb). Results revealed substantial agreement between GPT-5 and human ratings for the overall CEFR rating and most analytic dimensions, with weaker alignment observed for sociolinguistic appropriateness and orthography. Based on these findings, we propose a human-in-the-loop approach in which GPT-5 serves as an automated rater and highlight the need for dimension-specific refinement of automated assessment.
Article outline
  • 1.Introduction
  • 2.Literature review
    • 2.1Automated essay scoring (AES)
    • 2.2GPT-based LLMs for AES
    • 2.3Research question
  • 3.Methods
    • 3.1Data source
    • 3.2Procedure
      • 3.2.1Human rater evaluation
      • 3.2.2GPT-5 as an AES rater
      • 3.2.3Prompt design
    • 3.3Data analysis
  • 4.Results
    • 4.1Agreement with human ratings: QWK
    • 4.2Rank-order consistency: Kendall’s τb
  • 5.Discussion
  • 6.Limitations and recommendations
  • 7.Conclusion
  • Declaration of interest statement
  • References
References (54)
References
Allen, M. (2017). Kendall’s tau. In M. Allen (Ed.), The SAGE encyclopedia of communication research methods (Vol. 4, pp. 829–831). Thousand Oaks: SAGE Publications. Google Scholar logo with link to Google Scholar
Alvero, A. J., Lee, J., Regla-Vargas, A., Kizilcec, R. F., Joachims, T., & Antonio, A. L. (2024). Large language models, social demography, and hegemony: Comparing authorship in human and synthetic text. Journal of Big Data, 11(1), 138. Google Scholar logo with link to Google Scholar
Attali, Y. (2007). Construct validity of e-rater® in scoring TOEFL® essays. ETS Research Report Series, 2007(1), i–22. Google Scholar logo with link to Google Scholar
Bärenfänger, O. (2013). Assessing the reliability and scale functionality of the MERLIN written speech sample ratings. University of Leipzig: Language Learning Centre. [URL]
Barkaoui, K. (2010). Variability in ESL essay rating processes: The role of the rating scale and rater experience. Language Assessment Quarterly, 7(1), 54–74. Google Scholar logo with link to Google Scholar
Bayley, R., & Regan, V. (2004). Introduction: The acquisition of sociolinguistic competence. Journal of Sociolinguistics, 8(3), 323–338. Google Scholar logo with link to Google Scholar
Bui, N. M., & Barrot, J. S. (2025). ChatGPT as an automated essay scoring tool in the writing classrooms: How it compares with human scoring. Education and Information Technologies, 30(2), 2041–2058. Google Scholar logo with link to Google Scholar
Burstein, J., Tetreault, J., Chodorow, M., & Blanchard, D. (2013). Automated evaluation of discourse coherence quality in essay writing. In M. D. Shermis & J. Burstein (Eds.), Handbook of automated essay evaluation (pp. 267–280). New York: Routledge. Google Scholar logo with link to Google Scholar
Choi, I., & Cho, Y. (2018). The impact of spelling errors on trained raters’ scoring decisions. Language Education & Assessment, 1(2), 45–58. Google Scholar logo with link to Google Scholar
Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220. Google Scholar logo with link to Google Scholar
Correnti, R., Matsumura, L. C., Wang, E., Litman, D., Rahimi, Z., & Kisa, Z. (2020). Automated scoring of students’ use of text evidence in writing. Reading Research Quarterly, 55(3), 493–520. Google Scholar logo with link to Google Scholar
Council of Europe. (2020). Common European Framework of Reference for Languages: Learning, teaching, assessment — Companion volume. Strasbourg: Council of Europe Publishing. Retrieved from [URL]
Dikli, S., & Bleyle, S. (2014). Automated Essay Scoring feedback for second language writers: How does it compare to instructor feedback? Assessing Writing, 22, 1–17. Google Scholar logo with link to Google Scholar
Duncan, D. (2024). Does ChatGPT have sociolinguistic competence? Journal of Computer-Assisted Linguistic Research, 8, 51–75. Google Scholar logo with link to Google Scholar
Eckes, T. (2008). Rater types in writing performance assessments: A classification approach to rater variability. Language Testing, 25(2), 155–185. Google Scholar logo with link to Google Scholar
Funayama, H., Sato, T., Matsubayashi, Y., Mizumoto, T., Suzuki, J., & Inui, K. (2022). Balancing cost and quality: An exploration of human-in-the-loop frameworks for automated short answer scoring. In M. M. Rodrigo, N. Matsuda, A. I. Cristea, & V. Dimitrova (Eds.), Artificial intelligence in education (pp. 465–476). Cham: Springer International Publishing. Google Scholar logo with link to Google Scholar
García-Varela, F., Nussbaum, M., Mendoza, M., Martínez-Troncoso, C., & Bekerman, Z. (2025). ChatGPT as a stable and fair tool for automated essay scoring. Education Sciences, 15(8), 946. Google Scholar logo with link to Google Scholar
Görgen, R., De Simone, E., Schulte-Körne, G., & Moll, K. (2021). Predictors of reading and spelling skills in German: The role of morphological awareness. Journal of Research in Reading, 44(1), 210–227. Google Scholar logo with link to Google Scholar
Hancke, J., & Meurers, D. (2013). Exploring CEFR classification for German based on rich linguistic modeling. In K. Tenfjord, A. Golden, F. Meunier, & K. De Smedt (Eds.), Learner corpus research 2013: Book of abstracts (pp. 54–56). Bergen, Norway. [URL]
Hannah, L., Jang, E. E., Shah, M., & Gupta, V. (2023). Validity arguments for automated essay scoring of young students’ writing traits. Language Assessment Quarterly, 20(4–5), 399–420. Google Scholar logo with link to Google Scholar
Huang, S. J. (2014). Automated versus human scoring: A case study in an EFL context. Electronic Journal of Foreign Language Teaching, 11(1), 149–164. [URL]
Ifenthaler, D. (2023). Automated essay scoring systems. In O. Zawacki-Richter, & I. Jung (Eds.), Handbook of open, distance and digital education (pp. 1057–1071). Cham: Springer. Google Scholar logo with link to Google Scholar
Kendall, M. G. (1938). A new measure of rank correlation. Biometrika, 30, 81–93. Google Scholar logo with link to Google Scholar
Kim, Y. (2025). Automated essay scoring with GPT-4 for a local placement test: Investigating prompting strategies, intra-rater reliability, and alignment with human scores. TESOL Quarterly, 59(S1), S318–S329. Google Scholar logo with link to Google Scholar
Kumar, V. S., & Boulanger, D. (2021). Automated essay scoring and the deep learning black box: How are rubric scores determined? International Journal of Artificial Intelligence in Education, 31(3), 538–584. Google Scholar logo with link to Google Scholar
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. Google Scholar logo with link to Google Scholar
Lee, A. V. Y., Luco, A. C., & Tan, S. C. (2023). A human-centric automated essay scoring and feedback system for the development of ethical reasoning. Educational Technology & Society, 26(1), 147–159. [URL]
Li, J., Huang, J., & Sheeran, T. (2025). ChatGPT4o as an AI peer assessor in EFL speaking classrooms: Examining scoring reliability and feedback effectiveness. SAGE Open, 15(3), 21582440251369938. Google Scholar logo with link to Google Scholar
Li, W., & Liu, H. (2024). Applying large language models for automated essay scoring for non-native Japanese. Humanities and Social Sciences Communications, 11(1), 723. Google Scholar logo with link to Google Scholar
Liew, P. Y., & Tan, I. K. T. (2025). On automated essay grading using large language models. In Proceedings of the 2024 8th International Conference on Computer Science and Artificial Intelligence, 204–211. Google Scholar logo with link to Google Scholar
Liu, Y., Qi, H., & Lu, X. (2025). Enhancing GPT-based automated essay scoring: The impact of fine-tuning and linguistic complexity measures. Computer Assisted Language Learning, 1–20. Google Scholar logo with link to Google Scholar
Ludwig, S., Mayer, C., Hansen, C., Eilers, K., & Brandt, S. (2021). Automated essay scoring using transformer models. Psych, 3(4), 897–915. Google Scholar logo with link to Google Scholar
Lumley, T. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19(3), 246–276. Google Scholar logo with link to Google Scholar
Mao, L., Liu, O. L., Roohr, K., Belur, V., Mulholland, M., Lee, H. S., & Pallant, A. (2018). Validation of automated scoring for a formative assessment that employs scientific argumentation. Educational Assessment, 23(2), 121–138. Google Scholar logo with link to Google Scholar
MERLIN Project. (2014). Bewertungsraster. Retrieved on 9 October 2025, from [URL]
Memarian, B., & Doleck, T. (2024). Human-in-the-loop in artificial intelligence in education: A review and entity-relationship (ER) analysis. Computers in Human Behavior: Artificial Humans, 2(1), 100053. Google Scholar logo with link to Google Scholar
Mizumoto, A., & Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2), 100050. Google Scholar logo with link to Google Scholar
Monaghan, W., & Bridgeman, B. (2005). E-rater as a quality control on human scores (R&D Connections No. 2). Princeton, NJ: Educational Testing Service. [URL]
Naismith, B., Mulcaire, P., & Burstein, J. (2023). Automated evaluation of written discourse coherence using GPT-4. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), 394–403. Google Scholar logo with link to Google Scholar
Nunes, A., Cordeiro, C., Limpo, T., & Castro, S. L. (2022). Effectiveness of automated writing evaluation systems in school settings: A systematic review of studies from 2000 to 2020. Journal of Computer Assisted Learning, 38(2), 599–620. Google Scholar logo with link to Google Scholar
OpenAI. (2022). Introducing ChatGPT. Retrieved on 28 October 2025, from [URL]
Pack, A., Barrett, A., & Escalante, J. (2024). Large language models and automated essay scoring of English language learner writing: Insights into validity and reliability. Computers and Education: Artificial Intelligence, 6, 100234. Google Scholar logo with link to Google Scholar
Ramesh, D., & Sanampudi, S. K. (2022). An automated essay scoring systems: A systematic literature review. Artificial Intelligence Review, 55, 2495–2527. Google Scholar logo with link to Google Scholar
Ramineni, C., & Williamson, D. M. (2013). Automated essay scoring: Psychometric guidelines and practices. Assessing Writing, 18(1), 25–39. Google Scholar logo with link to Google Scholar
Reilly, E. D., Williams, K. M., Stafford, R. E., Corliss, S. B., Walkow, J. C., & Kidwell, D. K. (2016). Global times call for global measures: Investigating automated essay scoring in linguistically-diverse MOOCs. Online Learning, 20(2), 217–229. Google Scholar logo with link to Google Scholar
Seifart, F. (2006). Orthography development. In J. Gippert, N. P. Himmelmann, & U. Mosel (Eds.), Essentials of language documentation (pp. 275–300). Berlin, New York: De Gruyter Mouton. Google Scholar logo with link to Google Scholar
Seßler, K., Fürstenberg, M., Bühler, B., & Kasneci, E. (2025). Can AI grade your essays? A comparative analysis of large language models and teacher ratings in multidimensional essay scoring. In Proceedings of the 15th International Learning Analytics and Knowledge Conference (LAK’25). New York: Association for Computing Machinery, 462–472. Google Scholar logo with link to Google Scholar
Shermis, M. D., & Burstein, J. (Eds.). (2013). Handbook of automated essay evaluation: Current applications and new directions (1st ed.). New York: Routledge. Google Scholar logo with link to Google Scholar
Suhan, M., & Wolf, M. K. (2026). A comparative study of the human, automated scoring model, and GPT-4 ratings of young EFL students’ writing. Language Testing, 43(1), 66–78. Google Scholar logo with link to Google Scholar
Uyar, A. C., & Büyükahıska, D. (2025). Artificial intelligence as an automated essay scoring tool: A focus on ChatGPT. International Journal of Assessment Tools in Education, 12(1), 20–32. Google Scholar logo with link to Google Scholar
Vögelin, C., Jansen, T., Keller, S. D., & Möller, J. (2021). The impact of vocabulary and spelling on judgments of ESL essays: An analysis of teacher comments. The Language Learning Journal, 49(6), 631–647. Google Scholar logo with link to Google Scholar
Wang, J., & Brown, M. S. (2008). Automated essay scoring versus human scoring: A correlational study. Contemporary Issues in Technology and Teacher Education, 8(4). [URL]
Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2–13. Google Scholar logo with link to Google Scholar
Zupanc, K., & Bosnic, Z. (2015). Advances in the field of automated essay evaluation. Informatica, 39(4), 383–395. [URL]
Mobile Menu Logo with link to supplementary files background Layer 1 jbe-platform.quadraat-sans-bold jbe-platform.quadraat-sans-bold-dark jbe-platform.quadraat-sans-bold-light jbe-platform.quadraat-sans-bold prag Twitter_Logo_Blue