Article published In:
International Journal of Corpus Linguistics: Online-First Articles

A comparison of automated and manual analyses of syntactic complexity in L2 English writing

ORCID logo | Vietnam National University, Hanoi | University of Brussels (VUB)
ORCID logo | University of Brussels (VUB)

Automated tools for syntactic complexity measurement are increasingly used for analyzing various kinds of second language corpora, even though these tools were originally developed and tested for texts produced by advanced learners. This study investigates the reliability of automated complexity measurement for beginner and lower-intermediate L2 English data by comparing manual and automated analyses of a corpus of 80 texts written by Dutch-speaking learners. Our quantitative and qualitative analyses reveal that the reliability of automated complexity measurement is substantially affected by learner errors, parser errors, and Tregex pattern undergeneration. We also demonstrate the importance of aligning the definitions of analytical units between the computational tool and human annotators. In order to enhance the reliability of automated analyses, it is recommended that certain modifications are made to the system, and non-advanced L2 English data are preprocessed prior to automated analyses.

Keywords: L2 Syntactic Complexity Analyzer, Tool for the Automatic Analysis of Syntactic Sophistication and Complexity, syntactic complexity, automated analysis
Article outline
  • 1.Introduction
  • 2.L2SCA in L2 English research
    • 2.1 L2SCA system information
    • 2.2 L2SCA system reliability
    • 2.3Studies that used L2SCA
  • 3.Methodology
    • 3.1Corpus
    • 3.2Syntactic complexity measures
    • 3.3Corpus analysis
    • 3.4Comparison of manual and automated analyses
      • 3.4.1System-annotator agreement in unit and structure identification (RQ 1a)
      • 3.4.2Sources of system-annotator disagreement (RQ 1b)
      • 3.4.3System reliability in syntactic complexity scoring (RQ 2)
  • 4.Results
    • 4.1RQ1a: Unit and structure identification
    • 4.2RQ 1b: Sources of system-annotator disagreement
      • 4.2.1Words
      • 4.2.2Clausal and supra-clausal structures
        • Tregex pattern undergeneration
        • Parsing errors
        • Learner errors
    • 4.3RQ 2a: System reliability in syntactic complexity scoring
    • 4.4RQ 2b: Impact of learner errors and the use of broader definitions of clauses and dependent clauses
      • 4.4.1Impact of learner errors
      • 4.4.2Impact of broader clause and dependent clause definitions
  • 5.Discussion
  • 6.Conclusion
  • References
Published online: 17 October 2022
DOI logo
https://doi.org/10.1075/ijcl.20181.cha
References
Bulté, B.
(2013) The Development of Complexity in Second Language Acquisition: A Dynamic Systems Approach [Unpublished doctoral dissertation]. Vrije Universiteit Brussel.
Bulté, B., & Housen, A.
(2012) Defining and operationalising L2 complexity. In A. Housen, F. Kuiken, & I. Vedder (Eds.), Dimensions of L2 Performance and Proficiency: Complexity, Accuracy and Fluency in SLA (pp. 21–46). John Benjamins. DOI logoGoogle Scholar
(2014) Conceptualizing and measuring short-term changes in L2 writing complexity. Journal of Second Language Writing, 26 1, 42–65. DOI logoGoogle Scholar
(2018) Syntactic complexity in L2 writing: Individual pathways and emerging group trends. International Journal of Applied Linguistics, 28 (1), 147–164. DOI logoGoogle Scholar
Bulté, B., & Roothooft, H.
(2020) Investigating the interrelationship between rated L2 proficiency and linguistic complexity in L2 speech. System, 91 1, 102246. DOI logoGoogle Scholar
Abney, S. P.
(1987) The English Noun Phrase in its Sentential Aspect [Doctoral dissertation, Massachusetts Institute of Technology]. DSpace@MIT. [URL]
Ai, H., & Lu, X.
(2013) A corpus-based comparison of syntactic complexity in NNS and NS university students’ writing. In A. Díaz-Negrillo, N. Ballier, & P. Thompson (Eds.), Automatic Treatment and Analysis of Learner Corpus Data (pp. 249–264). John Benjamins. DOI logoGoogle Scholar
Bi, P., & Jiang, J.
(2020) Syntactic complexity in assessing young adolescent EFL learners’ writings: Syntactic elaboration and diversity. System, 91 1, 102248. DOI logoGoogle Scholar
Biber, D.
(1988) Variation Across Speech and Writing. Cambridge University Press. DOI logoGoogle Scholar
Cambridge University Press & Assessment
n.d.). Cambridge learner corpus – error codes. [URL]
Casal, J. E., & Lee, J. J.
(2019) Syntactic complexity and writing quality in assessed first-year L2 writing. Journal of Second Language Writing, 44 1, 51–62. DOI logoGoogle Scholar
Chen, X. B., & Meurers, D.
(2016) CTAP: A Web-Based Tool Supporting Automatic Complexity Analysis. Proceedings of The Workshop on Computational Linguistics for Linguistic Complexity (pp. 113–119). Association for Computational Linguistics. [URL]
Choi, J. D., Tetreault, J., & Stent, A.
(2015) It depends: Dependency parser comparison using a web-based evaluation tool. In C. Zong & M. Strube (Eds.), Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (pp. 387–396). Association for Computational Linguistics. [URL]. DOI logo
Cooper, T. C.
(1976) Measuring written syntactic patterns of second language learners of German. The Journal of Educational Research, 69 (5), 176–183. DOI logoGoogle Scholar
De Clercq, B., & Housen, A.
(2017) A cross-linguistic perspective on syntactic complexity in L2 Development: Syntactic elaboration and diversity. The Modern Language Journal, 101 (2), 315–334. DOI logoGoogle Scholar
Gaillat, T., & Ballier, N.
(2019) Prototype de feedback visuel des productions écrites d’apprenants francophones de l’anglais sous Moodle [Prototype of visual feedback for written productions of French-speaking learners of English on Moodle]. In Actes de la conférence EIAH2019. Association des Technologies de l’Information pour l’Education et la Formation. [URL]
Granger, S., Dagneaux, E., Meunier, F., & Paquot, M.
(2009) International Corpus of Learner English (Version 2.0). Presses universitaires de Louvain.Google Scholar
Huibregtse, I., Admiraal, W., & Meara, P.
(2002) Scores on a yes-no vocabulary test: Correction for guessing and response style. Language Testing, 19 (3), 227–245. DOI logoGoogle Scholar
Hunt, K. W.
(1965) Grammatical Structures Written at Three Grade Levels. National Council of Teachers of English.Google Scholar
Hwang, H., Jung, H., & Kim, H.
(2020) Effects of written versus spoken production modalities on syntactic complexity measures in beginning-level child EFL learners. The Modern Language Journal, 104 (1), 267–283. DOI logoGoogle Scholar
Jiang, J., Bi, P., & Liu, H.
(2019) Syntactic complexity development in the writings of EFL learners: Insights from a dependency syntactically-annotated corpus. Journal of Second Language Writing, 46 1, 100666. DOI logoGoogle Scholar
Kameen, P. T.
(1979) Syntactic skill and ESL writing quality. In C. Yorio, K. Perkins, & J. Schachter (Eds.), On TESOL ’79: The Learner in Focus (pp. 343–364). TESOL.Google Scholar
Khushik, G. A., & Huhta, A.
(2019) Investigating syntactic complexity in EFL learners’ writing across Common European Framework of Reference levels A1, A2, and B1. Applied Linguistics, 41 (4), 506–532. DOI logoGoogle Scholar
Klein, D., & Manning, C. D.
(2003) Fast exact inference with a factored model for natural language parsing. In S. Beker, S. Thrun, & K. Obermayer (Eds.), Advances in Neural Information Processing Systems 15 1 (pp. 3–10). MIT Press.Google Scholar
Kummerfeld, J. K., Hall, D., Curran, J., & Klein, D.
(2012) Parser showdown at the wall street corral: An empirical investigation of error types in parser output. In J. Tsujii, J. Henderson, & M. Paşca (Eds.), The 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (pp. 1048–1059). [URL]
Kyle, K.
(2016) Measuring Syntactic Development in L2 Writing: Fine Grained Indices of Syntactic Complexity and Usage-based Indices of Syntactic Sophistication [Doctoral dissertation, Georgia State University]. ScholarWorks @ Georgia State University. [URL]
(2021) Natural language processing for learner corpus research. International Journal of Learner Corpus Research, 7 (1), 1–16. DOI logoGoogle Scholar
Larsson, T., & Kaatari, H.
(2020) Syntactic complexity across registers: Investigating (in)formality in second-language writing. Journal of English for Academic Purposes, 45 1, 100850. DOI logoGoogle Scholar
Lu, X.
(2010) Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics, 15 (4), 474–496. DOI logoGoogle Scholar
(2011) A corpus-based evaluation of syntactic complexity measures as indices of college-level ESL writers’ language development. TESOL Quarterly, 45 (1), 36–62. DOI logoGoogle Scholar
(2014) Computational Methods for Corpus Annotation and Analysis. Springer. DOI logoGoogle Scholar
(2017) Automated measurement of syntactic complexity in corpus-based L2 writing research and implications for writing assessment. Language Testing, 34 (4), 493–511. DOI logoGoogle Scholar
Lu, X., & Ai, H.
(2015) Syntactic complexity in college-level English writing: Differences among writers with diverse L1 backgrounds. Journal of Second Language Writing, 29 1, 16–27. DOI logoGoogle Scholar
Lu, X., Casal, J. E., & Liu, Y.
(2020) The rhetorical functions of syntactically complex sentences in social science research article introductions. Journal of English for Academic Purposes, 44 1, 100832. DOI logoGoogle Scholar
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., & McClosky, D.
(2014) The Stanford CoreNLP Natural Language Processing Toolkit. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations (pp. 55–60). Association for Computational Linguistics. [URL]. DOI logo
McNamara, D. S., Crossley, S. A., & McCarthy, P. M.
(2010) Linguistic features of writing quality. Written Communication, 27 (1), 57–86. DOI logoGoogle Scholar
McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z.
(2014) Automated Evaluation Of Text And Discourse With Coh-metrix. Cambridge University Press. DOI logoGoogle Scholar
Meurers, D., & Dickinson, M.
(2017) Evidence and interpretation in language learning research: Opportunities for collaboration with computational linguistics. Language Learning, 67 (S1), 66–95. DOI logoGoogle Scholar
Nicholls, D.
(2003) The Cambridge Learner Corpus: Error coding and analysis for lexicography and ELT. Proceedings of the Corpus Linguistics 2003 conference, 572–581.Google Scholar
Ortega, L.
(2003) Syntactic complexity measures and their relationship to L2 Proficiency: A research synthesis of college-level L2 Writing. Applied Linguistics, 24 (4), 492–518. DOI logoGoogle Scholar
Pallotti, G.
(2015) A simple view of linguistic complexity. Second Language Research, 31 (1), 117–134. DOI logoGoogle Scholar
Polio, C. G.
(1997) Measures of linguistic accuracy in second language writing research. Language Learning, 47 (1), 101–143. DOI logoGoogle Scholar
Polio, C., & Yoon, H.
(2018) The reliability and validity of automated tools for examining variation in syntactic complexity across genres. International Journal of Applied Linguistics, 28 (1), 165–188. DOI logoGoogle Scholar
R Core Team
(2020) R: A language and environment for statistical computing (Version 4.0.0) [Computer software]. R Foundation for Statistical Computing. [URL]
Spoelman, M., & Verspoor, M.
(2010) Dynamic patterns in development of accuracy and complexity: A longitudinal case study in the acquisition of Finnish. Applied Linguistics, 31 (4), 532–553. DOI logoGoogle Scholar
Stanford Natural Language Processing Group
n.d.). Stanford Parser FAQ. Retrieved November 10, 2020, from [URL]
Verspoor, M., de Bot, C., & Xu, X.
(2015) The effects of English bilingual education in the Netherlands. Journal of Immersion and Content-Based Language Education, 3 (1), 4–27. DOI logoGoogle Scholar
Verspoor, M., Schmid, M. S., & Xu, X.
(2012) A dynamic usage based perspective on L2 writing. Journal of Second Language Writing, 21 (3), 239–263. DOI logoGoogle Scholar
Vyatkina, N.
(2013) Specific syntactic complexity: Developmental profiling of individuals based on an annotated learner corpus. The Modern Language Journal, 97 (S1), 11–30. DOI logoGoogle Scholar
Walter, T.
(2017) Measuring Syntactic Complexity in the Academic Writing of English Students at the University of Vienna. [Doctoral dissertation, Universität Wien]. u:theses Universität Wien. DOI logoGoogle Scholar
Wu, X., Mauranen, A., & Lei, L.
(2020) Syntactic complexity in English as a Lingua Franca academic writing. Journal of English for Academic Purposes, 43 1, 100798. DOI logoGoogle Scholar