Article published in:
Natural language processing for learner corpus research
Edited by Kristopher Kyle
[International Journal of Learner Corpus Research 7:1] 2021
► pp. 17–52

Automated annotation of learner English

An evaluation of software tools

| University of Arizona
| University of Arizona
| Northern Arizona University

This paper explores the use of natural language processing (NLP) tools and their utility for learner language analyses through a comparison of automatic linguistic annotation against a gold standard produced by humans. While there are a number of automated annotation tools for English currently available, little research is available on the accuracy of these tools when annotating learner data. We compare the performance of three linguistic annotation tools (a tagger and two parsers) on academic writing in English produced by learners (both L1 and L2 English speakers). We focus on lexico-grammatical patterns, including both phrasal and clausal features, since these are frequently investigated in applied linguistics studies. Our results report both precision and recall of annotation output for argumentative texts in English across four L1s: Arabic, Chinese, English, and Korean. We close with a discussion of the benefits and drawbacks of using automatic tools to annotate learner language.

Keywords: learner NLP, automated annotation, learner English, writing research
Article outline
  • 1.Introduction
  • 2.Lexico-grammatical patterns in L2 English academic writing
    • 2.1Performance of automated annotation
  • 3.Tool performance evaluation
    • 3.1Methods
      • 3.1.1English academic writing corpus
      • 3.1.2Choice of automatic annotation tools
      • 3.1.3Gold standard labels
      • 3.1.4Feature extraction
        • 3.1.4.1Attributive adjectives
        • 3.1.4.2Noun-noun sequences
        • 3.1.4.3Relative clause
        • 3.1.4.4Complement clause
      • 3.1.5Output alignment
      • 3.1.6Analysis
    • 3.2Results
      • 3.2.1Phrasal features
      • 3.2.2Clausal features
  • 4.Discussion and conclusion
  • Acknowledgements
  • References
Published online: 01 March 2021
https://doi.org/10.1075/ijlcr.20003.pic
References
Ansarifar, A., Shahriari, H., & Pishghadam, R.
(2018) Phrasal complexity in academic writing: A comparison of abstracts written by graduate students and expert writers in applied linguistics. Journal of English for Academic Purposes, 311, 58–71. DOI logoGoogle Scholar
Biber, D.
(1988) Variation across speech and writing. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
(2006) University language: A corpus-based study of spoken and written registers. Amsterdam: John Benjamins Publishing. DOI logoGoogle Scholar
Biber, D., & Gray, B.
(2013) Discourse characteristics of writing and speaking task types on the TOEFL ibt® test: a lexico-grammatical analysis. ETS Research Report Series 2013(1), i–128. DOI logoGoogle Scholar
(2016) Grammatical complexity in academic English: Linguistic change in writing. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Biber, D., Gray, B., & Poonpon, K.
(2011) Should we use characteristics of conversation to measure grammatical complexity in L2 writing development? Tesol Quarterly, 45(1), 5–35. DOI logoGoogle Scholar
Biber, D., Johansson, S., Leech, G., Conrad, S., & Finegan, E.
(1999) Longman grammar of written and spoken English. Harlow: Longman.Google Scholar
Buchholz, S., & Marsi, E.
(2006) CoNLL-X shared task on multilingual dependency parsing. In L. Màrquez & D. Klein (Eds.), Proceedings of the tenth conference on computational natural language learning (pp. 149–164). Stroudsburg: Association for Computational Linguistics. DOI logoGoogle Scholar
Canty, A., & Ripley, B.
(2019) Boot: Bootstrap R (S-Plus) Functions. R package version 11.3–22.Google Scholar
Casal, J. E., & Lee, J. J.
(2019) Syntactic complexity and writing quality in assessed first-year L2 writing. Journal of Second Language Writing, 441, 51–62. DOI logoGoogle Scholar
Cer, D. M., de Marneffe, M., Jurafsky, D., & Manning, C.
(2010) Parsing to Stanford Dependencies: Trade-offs between Speed and Accuracy. In N. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, M. Rosner, & D. Tapias (Eds.), Proceedings of the 2010 International Conference on Language Resources and Evaluation (pp. 1–5). European Language Resources Association (ELRA).Google Scholar
Charles, M.
(2007) Argument or evidence? Disciplinary variation in the use of the noun that pattern in stance construction. English for Specific Purposes, 26(2), 203–218. DOI logoGoogle Scholar
Charniak, E.
(2000) A maximum-entropy-inspired parser. In J. Wiebe (Ed.), Proceedings of the 1st North American chapter of the Association for Computational Linguistics conference (pp. 132–139). Stroudsburg: Association for Computational Linguistics.Google Scholar
Chen, D., & Manning, C.
(2014) A fast and accurate dependency parser using neural networks. In A. Moschitti, B. Pang, W. Daelemans (Eds.), Proceedings of the 2014 conference on empirical methods in natural language processing (pp. 740–750). Stroudsburg: Association for Computational Linguistics.Google Scholar
Crossley, S. A., & McNamara, D. S.
(2009) Computational assessment of lexical differences in L1 and L2 writing. Journal of Second Language Writing, 18(2), 119–135. DOI logoGoogle Scholar
(2014) Does writing development equal writing quality? A computational investigation of syntactic complexity in L2 learners. Journal of Second Language Writing, 261, 66–79. DOI logoGoogle Scholar
de Marneffe, M., & Manning, C.
(2008) The Stanford typed dependencies representation. In Coling 2008: proceedings of the workshop on cross-framework and cross-domain parser evaluation (pp. 1–8). Stroudsburg: Association for Computational Linguistics. DOI logoGoogle Scholar
Eisenstein, J.
(2019) Introduction to natural language processing. Cambridge, MA: The MIT Press.Google Scholar
ETS
(2014) A guide to understanding TOEFL iBT® scores. Educational Testing Service.Google Scholar
Francis, W., & Kučera, H.
(1964) Brown corpus. Providence, Rhode Island: Department of Linguistics, Brown University.Google Scholar
Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z.
(2004) Coh-Metrix: Analysis of text on cohesion and language. Behavior research methods, instruments, & computers, 36(2), 193–202. DOI logoGoogle Scholar
Granger, S.
(2008) Learner corpora in foreign language education. In S. Thorne & S. May (Eds.), Language, Education and Technology. Encyclopedia of Language and Education (pp. 1427–1441). Berlin: Springer. DOI logoGoogle Scholar
Halacsy, P., Kornai, A., & Oravecz, C.
(2007) Hunpos: an open source trigram tagger. In S. Ananiadou (Ed.), Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions (pp. 209–212). Stroudsburg: Association for Computational Linguistics. DOI logoGoogle Scholar
Hempelmann, C. F., Rus, V., Graesser, A. C., & McNamara, D. S.
(2006) Evaluating state-of-the-art treebank-style parsers for Coh-metrix and other learning technology environments. Natural Language Engineering, 12(2), 131–144. DOI logoGoogle Scholar
Honnibal, M., & Montani, I.
(2017) spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing [Python Library version 2.3.2].Google Scholar
Jiang, J., Bi, P., & Liu, H.
(2019) Syntactic complexity development in the writings of EFL learners: Insights from a dependency syntactically-annotated corpus. Journal of Second Language Writing, 461, 100666–100679. DOI logoGoogle Scholar
Johansson, S., Leech, G., & Goodluck, H.
(1978) Manual of information to accompany the Lancaster-Olso/Bergen corpus of British English, for use with digital computers. Oslo. Department of English, University of Oslo. Retrieved from http://​korpus​.uib​.no​/icame​/manuals​/LOB​/INDEX​.HTM
Jurafsky, D., & Martin, J. H.
(2008) Speech and language processing: An introduction to natural Language processing, computational linguistics, and speech recognition. Upper Saddle River, NJ: Pearson Prentice Hall.Google Scholar
Klein, D., & Manning, C. D.
(2003) Fast exact inference with a factored model for natural language parsing. In S. Becker, S. Thrun, & K. Obermayer (Eds.), Advances in neural information processing systems 15 (pp. 3–10). Cambridge, MA: The MIT Press.Google Scholar
Koehn, P.
(2004) Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 conference on empirical methods in natural language processing (pp. 388–395). Stroudsburg: Association for Computational Linguistics.Google Scholar
Kyle, K.
(2016) Measuring syntactic development in L2 writing: Fine Grained Indices of Syntactic Complexity and Usage-based Indices of Syntactic Sophistication (Unpublished doctoral dissertation). Georgia State University, Atlanta, GA.Google Scholar
Kyle, K., & Crossley, S. A.
(2018) Measuring syntactic complexity in L2 writing using fine-grained clausal and phrasal indices. The Modern Language Journal, 102(2), 333–349. DOI logoGoogle Scholar
Levy, R., & Andrew, G.
(2006) Tregex and Tsurgeon: tools for querying and manipulating tree data structures. In N. Calzolari, K. Choukri, A. Gangemi, B. Maegaard, J. Mariani, J. Odijk, & D. Tapias (Eds.), Proceedings of the Fifth International Conference on Language Resources and Evaluation (pp. 2231–2234). European Language Resources Association (ELRA).Google Scholar
Liu, L., & Li, L.
(2016) Noun Phrase Complexity in EFL Academic Writing: A Corpus-Based Study of Postgraduate Academic Writing. Journal of Asia TEFL, 13(1), 48–66.Google Scholar
Lu, X.
(2010) Automatic analysis of syntactic complexity in second language writing. International journal of corpus linguistics, 15(4), 474–496. DOI logoGoogle Scholar
Lu, X., & Ai, H.
(2015) Syntactic complexity in college-level English writing: Differences among writers with diverse L1 backgrounds. Journal of Second Language Writing, 291, 16–27. DOI logoGoogle Scholar
Marcus, M., Marcinkiewicz, M., & Santorini, B.
(1993) Building a large annotated corpus of English: The Penn Treebank. Computational linguistics, 19(2), 313–330.Google Scholar
Marcus, M., Kim, G., Marcinkiewicz, M. A., MacIntyre, R., Bies, A., Ferguson, M., Katz, K., & Schasberger, B.
(1994) The Penn Treebank: annotating predicate argument structure. In Proceedings of Human Language Technology Workshop (pp. 114–119). Stroudsburg: Association for Computational Linguistics. DOI logoGoogle Scholar
McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z.
(2014) Automated evaluation of text and discourse with Coh-Metrix. New York, NY: Cambridge University Press. DOI logoGoogle Scholar
Nivre, J., Hall, J., Nilsson, J., Chanev, A., Eryigit, G., Kubler, S., Marinov, S., & Marsi, E.
(2007) MaltParser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(2), 95–135. DOI logoGoogle Scholar
Norris, J. M., & Ortega, L.
(2009) Towards an organic approach to investigating CAF in instructed SLA: The case of complexity. Applied Linguistics, 30(4), 555–578. DOI logoGoogle Scholar
Ott, N., & Ziai, R.
(2010) Evaluating dependency parsing performance on German learner language. In M. Dickinson, K. Müürisep, & M. Passarotti (Eds.), Proceedings of the Ninth International Workshop on Treebanks and Linguistic Theories (pp. 175–186). Tartu: NEALT.Google Scholar
Paquot, M.
(2019) The phraseological dimension in interlanguage complexity research. Second Language Research, 35(1), 121–145. DOI logoGoogle Scholar
Pérez-Paredes, P., & Díez-Bedmar, M. B.
(2019) Researching learner language through POS keyword and syntactic complexity analyses. In S. Götz & J. Mukherjee (Eds.), Learner Corpora and Language Teaching (pp. 101–127). Amsterdam: John Benjamins Publishing. DOI logoGoogle Scholar
Parkinson, J., & Musgrave, J.
(2014) Development of noun phrase complexity in the writing of English for academic purposes students. Journal of English for Academic Purposes, 141, 48–59. DOI logoGoogle Scholar
Paul, D., & Baker, J.
(1992) The design for the Wall Street Journal-based CSR corpus. In Proceedings of the workshop on Speech and Natural Language (pp. 357–362). Stroudsburg: Association for Computational Linguistics. DOI logoGoogle Scholar
Peters, T.
(2018) Difflib: Helpers for computing differences between objects. [Python library]. Retrieved from https://​docs​.python​.org​/3​/library​/difflib​.html
Polio, C., & Yoon, H. J.
(2018) The reliability and validity of automated tools for examining variation in syntactic complexity across genres. International Journal of Applied Linguistics, 28(1), 165–188. DOI logoGoogle Scholar
Rayson, P.
(2008) From key words to key semantic domains. International Journal of Corpus Linguistics, 13(4), 519–549. DOI logoGoogle Scholar
(2009) Wmatrix: A Web-based Corpus-processing Environment. Lancaster: Computing Department, Lancaster University.Google Scholar
Riezler, S., & Maxwell, J. T.
(2005) On some pitfalls in automatic evaluation and significance testing for MT. In Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization (pp. 57–64). New Brunswick: Association for Computational Linguistics.Google Scholar
Santorini, B.
(1990) Part-of-speech tagging guidelines for the Penn Treebank (3rd Revision, 2nd Edition). Philadelphia: Department of Computer Science, University of Pennsylvania. Retrieved from https://​catalog​.ldc​.upenn​.edu​/docs​/LDC99T42​/tagguid1​.pdf
Schmid, H.
(2019) Deep learning-based morphological taggers and lemmatizers for annotating historical texts. In Proceedings of the Digital Access to Textual Cultural Heritage conference (DATeCH) (pp. 133–137). New York: Association for Computing Machinery.Google Scholar
Shenoy, G. G., Dsouza, E. H., & Kübler, S.
(2017) Performing stance detection on Twitter data using computational linguistics techniques. arXiv, arXiv:1703.02019.Google Scholar
Simar, L., & Wilson, P. W.
(1998) Sensitivity analysis of efficiency scores: How to bootstrap in nonparametric frontier models. Management Science, 44(1), 49–61. DOI logoGoogle Scholar
Staples, S., & Reppen, R.
(2016) Understanding first-year L2 writing: A lexico-grammatical analysis across L1s, genres, and language ratings. Journal of Second Language Writing, 321, 17–35. DOI logoGoogle Scholar
Staples, S., Biber, D., & Reppen, R.
(2018) Using Corpus-Based Register Analysis to Explore the Authenticity of High-Stakes Language Exams: A Register Comparison of TOEFL iBT and Disciplinary Writing Tasks. The Modern Language Journal, 102(2), 310–332. DOI logoGoogle Scholar
Sokolova, M., & Lapalme, G.
(2009) A systematic analysis of performance measures for classification tasks. Information processing & management, 45(4), 427–437. DOI logoGoogle Scholar
van Rooy, B.
(2015) Annotating learner corpora. In S. Granger, G. Gilquin, & F. Meunier (Eds.), The Cambridge Handbook of Learner Corpus Research (pp. 79–106). Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Yoon, H., & Polio, C.
(2017) ESL students’ linguistic development in two written genres. TESOL Quarterly, 51(2), 275–301. DOI logoGoogle Scholar