Cover not available

Article In: Journal of Second Language Studies: Online-First Articles

Diagnostic grammar assessment in second language writing using prompt-engineered large language models

ORCiD logo with link | University of Alberta, Canada
 | University of Alberta, Canada
 | University of Alberta, Canada
This content is being prepared for publication; it may be subject to changes.
Abstract
Grammar remains one of the most challenging areas for English as a Second Language (ESL) writers, and instructors often struggle to provide timely, consistent, and detailed feedback.
Recent advances in large language models (LLMs) offer new opportunities to automate grammatical error detection, but their use for deeper diagnostic assessment is still underexplored. This study examines how prompt engineering can guide GPT-4o to generate structured grammatical error labels and how these labels can be transformed into Q-matrix-based diagnostic profiles through Q-matrix coding. Using a dataset of 132 student essays, GPT-4o was prompted with a standardized EAP codebook to identify and label grammatical errors across categories such as verb tense, word form, sentence structure, and coherence. Model outputs were compared with instructor annotations using Precision, Recall, F1-score, and Cohen’s Kappa. The model showed moderate-to-substantial alignment with the instructor annotations (Precision = 0.74, Recall = 0.78, F1 = 0.76, Kappa = 0.75). These labelled errors were then mapped onto four grammatical skill domains to construct Q-matrix-based learner-by-skill diagnostic profiles.
Analysis revealed nine diagnostic patterns and four learner clusters with distinct grammar challenges. Findings show that LLM-generated error tags, when guided by clear prompts, can support Q-matrix-based diagnostic profiling of grammatical difficulties. The integration of prompt engineering and Q-matrix coding provides a scalable and interpretable framework for summarizing learners’ patterns of grammatical difficulty, supporting more targeted instructional feedback. This study demonstrates a novel pathway for connecting automated error detection with Q-matrix-based diagnostic profiling in ESL writing.
Article outline
  • 1.Introduction
  • 2.Literature review
    • 2.1Grammatical error detection in ESL writing
    • 2.2Large models in grammar correction
    • 2.3Prompt engineering in writing tasks
    • 2.4Prompted LLMs and diagnostic profiling
    • 2.5Q-Matrix-based diagnostic profiling
  • 3.Methods
    • 3.1Dataset for automated grammatical error detection
    • 3.2Prompt engineering and model setup
    • 3.3Model evaluation
      • Phase 1
      • Phase 2
      • Phase 3
    • 3.4Data analysis
  • 4.Results
    • 4.1Model performance evaluation
    • 4.2Phase 2: Q-matrix construction
    • 4.3Phase 3: Q-matrix-based diagnostic profiling
  • 5.Discussion and conclusions
  • 6.Limitations and directions for future research
  • Author queries
  • References
References (81)
References
Aryadoust, V. (2018). A cognitive diagnostic assessment study of the listening test of the Singapore–Cambridge general certificate of education O-Level: Application of DINA, DINO, G-DINA, HO-DINA, and RRUM. International Journal of Listening, 35(1), 29–52. Google Scholar logo with link to Google Scholar
Aryadoust, V., Zakaria, A., & Jia, Y. (2024). Investigating the affordances of OpenAI’s large language model in developing listening assessments. Computers and Education: Artificial Intelligence, 6, 100204. Google Scholar logo with link to Google Scholar
Bitchener, J. (2012). Written Corrective Feedback in Second Language Acquisition and Writing. Routledge. Google Scholar logo with link to Google Scholar
Bora, P. (2023). Importance of writing skill to develop students’ communication skill. Journal for Research Scholars and Professionals of English Language Teaching, 7(35). Google Scholar logo with link to Google Scholar
Brown, T., Mann, B. F., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J. C. S., Winter, C., & Hesse, C. (2020). Language models are few-shot learners. ArXiv (Cornell University), 4. Google Scholar logo with link to Google Scholar
Bryant, C., Felice, M., & Briscoe, T. (2017). Automatic annotation and evaluation of error types for grammatical error correction. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Google Scholar logo with link to Google Scholar
Bryant, C., Zheng, Y., Muhammad Reza Qorib, Cao, H., Hwee Tou Ng, & Briscoe, T. (2023). Grammatical error correction: A survey of the state of the art. Computational Linguistics, 1–59. Google Scholar logo with link to Google Scholar
Catabay, M. Q. (2023). Analysis of second language learners’ errors in composition writing: Basis for the proposed English remedial program in higher institution. Journal of Law and Sustainable Development, 11(4), e895–e895. Google Scholar logo with link to Google Scholar
Chollampatt, S., & Ng, H. T. (2018). Neural Quality Estimation of Grammatical Error Correction (E. Riloff, D. Chiang, J. Hockenmaier, & J. Tsujii, Eds.). ACLWeb; Association for computational linguistics. Google Scholar logo with link to Google Scholar
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. Google Scholar logo with link to Google Scholar
Dahlmeier, D., Ng, H., & Wu, S. (2013). Building a large annotated corpus of learner English: The NUS Corpus of Learner English (pp. 22–31). Association for computational linguistics. [URL]
de la Torre, J. (2009). DINA model and parameter estimation: A Didactic. Journal of Educational and Behavioral Statistics, 34(1), 115–130. Google Scholar logo with link to Google Scholar
de la Torre, J., & Chiu, C.-Y. (2016). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253–273. Google Scholar logo with link to Google Scholar
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019, May 24). BERT: Pre-training of deep bidirectional transformers for language understanding. ArXiv.org. Google Scholar logo with link to Google Scholar
Fareed, M., Ashraf, A., & Bilal, M. (2016). ESL learners’ writing skills: Problems, factors and suggestions. Journal of Education & Social Sciences, 4(2), 83–94. Google Scholar logo with link to Google Scholar
Ferris, D. (2011). Treatment of error in second language student writing. University Of Michigan Press. Google Scholar logo with link to Google Scholar
Ferris, D., & Roberts, B. (2001). Error feedback in L2 writing classes: How explicit does it need to be? Journal of Second Language Writing, 10(3), 161–184. Google Scholar logo with link to Google Scholar
Fleiss, J. L., & Cohen, J. (1973). The equivalence of weighted Kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33(3), 613–619. Google Scholar logo with link to Google Scholar
Gierl, M. J., & Lai, H. (2013). Evaluating the quality of medical multiple-choice items created with automated processes. Medical Education, 47(7), 726–733. Google Scholar logo with link to Google Scholar
Gierl, M. J., Latifi, S., Lai, H., Boulais, A.-P., & De Champlain, A. (2014). Automated essay scoring and the future of educational assessment in medical education. Medical Education, 48(10), 950–962. Google Scholar logo with link to Google Scholar
Guo, L., & Zhou, W. (2021). Nonparametric methods for cognitive diagnosis to multiple-choice test items. Acta Psychologica Sinica, 53(9), 1032. Google Scholar logo with link to Google Scholar
Hendrickson, J. M. (1978). Error correction in foreign language teaching: Recent theory, Research, and Practice. The Modern Language Journal, 62(8), 387. Google Scholar logo with link to Google Scholar
Hinkel, E., & Hinkel, E. (2003). Teaching Academic ESL Writing. Routledge. Google Scholar logo with link to Google Scholar
Hyland, K., & Hyland, F. (2006). Feedback on second language students’ writing. Language Teaching, 39(2), 83–101. Google Scholar logo with link to Google Scholar
Jang, E. E. (2009). Cognitive diagnostic assessment of L2 reading comprehension ability: Validity arguments for Fusion Model application to LanguEdge assessment. Language Testing, 26(1), 031–73. Google Scholar logo with link to Google Scholar
Jordan, R. R. (1997). English for academic purposes. Cambridge University Press. Google Scholar logo with link to Google Scholar
Klie, J.-C., Webber, B., & Gurevych, I. (2023). Annotation error detection: Analyzing the past and present for a more coherent future. Computational Linguistics, 1–42. Google Scholar logo with link to Google Scholar
Knoth, N., Tolzin, A., Janson, A., & Jan Marco Leimeister. (2024). AI literacy and its implications for prompt engineering strategies. Computers and Education. Artificial Intelligence, 6, 100225–100225. Google Scholar logo with link to Google Scholar
Kwarikunda, D., Schiefele, U., Muwonge, C. M., & Ssenyonga, J. (2022). Profiles of learners based on their cognitive and metacognitive learning strategy use: occurrence and relations with gender, intrinsic motivation, and perceived autonomy support. Humanities and Social Sciences Communications, 9(1). Google Scholar logo with link to Google Scholar
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. Google Scholar logo with link to Google Scholar
Larsen-Freeman, D. (2003). Teaching language: From grammar to grammaring. Boston, MA: Thomson/Heinle.Google Scholar logo with link to Google Scholar
Larsen-Walker, M. (2017). Can data driven learning address L2 writers’ habitual errors with English linking adverbials? System, 69, 26–37. Google Scholar logo with link to Google Scholar
Le, V., Nissen, J. M., Tang, X., Zhang, Y., Mehrabi, A., Morphew, J. W., Chang, H. H., & Dusen, B. V. (2025). Applying cognitive diagnostic models to mechanics concept inventories. Physical Review Physics Education Research, 21(1). Google Scholar logo with link to Google Scholar
Leacock, C., Chodorow, M., Gamon, M., & Tetreault, J. (2014). Automated grammatical error detection for language learners, second edition. In synthesis lectures on human language technologies. Springer. Google Scholar logo with link to Google Scholar
Lee, G.-G., Latif, E., Wu, X., Liu, N., & Zhai, X. (2024). Applying large language models and chain-of-thought for automatic scoring. Computers and Education. Artificial Intelligence, 100213–100213. Google Scholar logo with link to Google Scholar
Lee, Y.-W., & Sawaki, Y. (2009). Application of three cognitive diagnosis models to ESL reading and listening assessments. Language Assessment Quarterly, 6(3), 239–263. Google Scholar logo with link to Google Scholar
Leighton, J., & Gierl, M. (Eds.). (2007). Cognitive diagnostic assessment for education. Cambridge University Press. Google Scholar logo with link to Google Scholar
Leki, I. (1991). The preferences of ESL students for error correction in college-level writing Classes. Foreign Language Annals, 24(3), 203–218. Google Scholar logo with link to Google Scholar
Li, H., Hunter, C. V., & Lei, P.-W. (2016). The selection of cognitive diagnostic models for a reading comprehension test. Language Testing, 33(3), 391–409. Google Scholar logo with link to Google Scholar
Li, Y., Huang, C., & Liu, J. (2023). Diagnosing Primary Students’ Reading Progression: Is Cognitive Diagnostic Computerized Adaptive Testing the Way Forward? Journal of Educational and Behavioral Statistics, 48(6), 842–865. Google Scholar logo with link to Google Scholar
Liu, J., Xu, G., & Ying, Z. (2012). Data-driven learning of Q-Matrix. Applied Psychological Measurement, 36(7), 548–564. Google Scholar logo with link to Google Scholar
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., Wu, Z., Zhu, D., Li, X., Niu, Q., Shen, D., Liu, T., & Ge, B. (2023). Summary of ChatGPT-related research and perspective towards the future of large language models. Meta-Radiology, 1(2). Google Scholar logo with link to Google Scholar
Matelsky, J. K., Parodi, F., Liu, T., Lange, R. D., & Kording, K. P. (2023, July 25). A large language model-assisted education tool to provide feedback on open-ended responses. ArXiv.org. Google Scholar logo with link to Google Scholar
Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., & Fernández-Leal, Á. (2023). Human-in-the-loop machine learning: a state of the art. Artificial Intelligence Review, 56. Google Scholar logo with link to Google Scholar
Myles, J. (2002, September 1). Second Language Writing and Research: The Writing Process and Error Analysis in Student Texts. Tesl-Ej.org. [URL]
Ng, H., Wu, S., Briscoe, T., Hadiwinoto, C., Susanto, R., & Bryant, C. (2014). The CoNLL-2014 Shared Task on Grammatical Error Correction (pp. 26–27). Association for Computational Linguistics. [URL].
Omelianchuk, K., Atrasevych, V., Chernodub, A., & Skurzhanskyi, O. (2020, July 1). GECToR –Grammatical Error Correction: Tag, Not Rewrite. ACLWeb; Association for Computational Linguistics. Google Scholar logo with link to Google Scholar
OpenAI. (2024, May 13). Hello GPT-4o. OpenAI. [URL]
Pradhan, S., Hacioglu, K., Krugler, V., Ward, W., Martin, J. H., & Jurafsky, D. (2005). Support vector learning for semantic argument classification. Machine Learning, 60(1–3), 11–39. Google Scholar logo with link to Google Scholar
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019). Exploring the limits of transfer learning with a unified Text-to-Text Transformer. ArXiv (Cornell University). Google Scholar logo with link to Google Scholar
Ravand, H. (2016). Application of a cognitive diagnostic model to a high-stakes reading comprehension test. Journal of Psychoeducational Assessment, 34(8), 782–799. Google Scholar logo with link to Google Scholar
Ravand, H., & Robitzsch, A. (2018). Cognitive diagnostic model of best choice: a study of reading comprehension. Educational Psychology, 38(10), 1255–1277. Google Scholar logo with link to Google Scholar
Rei, M., & Yannakoudakis, H. (2016). Compositional sequence labelling models for error detection in learner writing. ArXiv (Cornell University). Google Scholar logo with link to Google Scholar
Rousseeuw, P. J. (1987). Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20(0377–0427), 53–65. Google Scholar logo with link to Google Scholar
Rupp, A. A., Templin, J., & Henson, R. A. (2010). Diagnostic Measurement. Guilford Press.Google Scholar logo with link to Google Scholar
Sawaki, Y., Kim, H.-J., & Gentile, C. (2009). Q-Matrix construction: Defining the link between constructs and test items in large-scale reading and listening comprehension assessments. Language Assessment Quarterly, 6(3), 190–209. Google Scholar logo with link to Google Scholar
Shermis, M. D. (2014). State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration. Assessing Writing, 20, 53–76. Google Scholar logo with link to Google Scholar
Shermis, M. D., Garvan, C. W., & Diao, Y. (2008). The impact of automated essay scoring on writing outcomes. [URL]
Shi, X., Ma, X., Du, W., & Gao, X. (2023). Diagnosing Chinese EFL learners’ writing ability using polytomous cognitive diagnostic models. Language Testing, 026553222311628–026553222311628. Google Scholar logo with link to Google Scholar
Shin, J., & Gierl, M. J. (2024). Automated short-response scoring for automated item generation in science assessments. The Routledge International Handbook of Automated Essay Evaluation, 504–534. Google Scholar logo with link to Google Scholar
Snehal Prabhudesai, Yang L., Asthana, S., Huan, X., Q. Vera Liao, & Banovic, N. (2023). Understanding uncertainty: How lay decision-makers perceive and interpret uncertainty in human-AI decision making. ACM Digital Library. Google Scholar logo with link to Google Scholar
Song, L., Wang, W., & Ding, S. (2024). Q-matrix theory and its applications in cognitive diagnostic assessment. Advances in Psychological Science, 32(6), 1010–1010. Google Scholar logo with link to Google Scholar
Steiss, J., Tate, T., Graham, S., Cruz, J., Hebert, M., Wang, J., Moon, Y., Tseng, W., Warschauer, M., & Carol Booth Olson. (2024). Comparing the quality of human and ChatGPT feedback of students’ writing. Learning and Instruction, 91, 101894–101894. Google Scholar logo with link to Google Scholar
Sutton, C. (2012). An introduction to conditional random fields. Foundations and Trends® in Machine Learning, 4(4), 267–373. Google Scholar logo with link to Google Scholar
Firoozi, T., Mohammadi, H., & Gierl, M. J. (2024). Using automated procedures to score educational essays written in three languages. Journal of Educational Measurement. Google Scholar logo with link to Google Scholar
Firoozi, T., Daniels, L., Daniels, V., & Gierl, M. (2025). An Augmented Intelligence System for Automated Quality Control and Feedback Generation of Multiple Choice Test Items. Communications in Computer and Information Science, 86–93. Google Scholar logo with link to Google Scholar
Taylor, L., & Geranpayeh, A. (2011). Assessing listening for academic purposes: Defining and operationalising the test construct. Journal of English for Academic Purposes, 10(2), 89–101. Google Scholar logo with link to Google Scholar
van Dis, E. A. M., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023). ChatGPT: Five priorities for research. Nature, 614(7947), 224–226. Google Scholar logo with link to Google Scholar
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. [URL]
Wang, J., Qiu, M., Gu, Y., & Park, J. (2025, January 21). Refined Evaluation for End-to-End Grammatical Error Correction Using an Alignment-Based Approach. ResearchGate; unknown. [URL]
Wen, H., Liu, Y., & Zhao, N. (2020). Longitudinal cognitive diagnostic assessment based on the HMM/ANN Model. Frontiers in Psychology, 11. Google Scholar logo with link to Google Scholar
Wu, H., Liang, X., Yürekli, H., Becker, B. J., Paek, I., & Binici, S. (2019). Exploring the impact of Q-Matrix specifications through a DINA model in a large-scale Mathematics assessment. Journal of Psychoeducational Assessment, 38(5), 581–598. Google Scholar logo with link to Google Scholar
Wu, X., Li, N., Wu, R., & Liu, H. (2025). Cognitive analysis and path construction of Chinese students’ mathematics cognitive process based on CDA. Scientific Reports, 15(1). Google Scholar logo with link to Google Scholar
Xu, T., Wu, X., Sun, S., & Kong, Q. (2023). Cognitive diagnostic analysis of students’ mathematical competency based on the DINA model. Psychology in the Schools, 60(9), 3135–3150. Google Scholar logo with link to Google Scholar
Yang, C. T.-Y., & Chen, H. H.-J. (2025). ChatGPT and L2 Chinese writing: evaluating the impact of model version and prompt language on automated corrective feedback. Computer Assisted Language Learning, 1–29. Google Scholar logo with link to Google Scholar
Yannakoudakis, H., Rei, M., Andersen, Ø. E., & Yuan, Z. (2017). Neural sequence-labelling models for grammatical error correction. Research Portal (King’s College London). Google Scholar logo with link to Google Scholar
Zhang, L. J. (2013). Second language writing as and for second language learning. Journal of Second Language Writing, 22(4), 446–447. Google Scholar logo with link to Google Scholar
Zhang, Z., Zhang, A., Li, M., & Smola, A. (2022). Automatic chain of thought prompting in large language models. ArXiv (Cornell University). Google Scholar logo with link to Google Scholar
Zhang, Y., Zhao, B., Jian, M., & Wu, X. (2025). Cognitive diagnostic analysis of mathematics key competencies based on PISA data. PLoS ONE, 20(2), e0315539–e0315539. Google Scholar logo with link to Google Scholar
Zhou, Y., Andrei Ioan Muresanu, Han, Z., Paster, K., Silviu Pitis, Chan, H., & Ba, J. (2023). Large Language Models are Human-Level Prompt Engineers. OpenReview. [URL]
Zhu, J., Shi, X., & Zhang, S. (2021). Machine learning-based grammar error detection method in English composition. Scientific Programming, 2021, 1–10. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 jbe-platform.quadraat-sans-bold jbe-platform.quadraat-sans-bold-dark jbe-platform.quadraat-sans-bold-light jbe-platform.quadraat-sans-bold prag Twitter_Logo_Blue