Cover not available
Article published In:
Natural language processing for learner corpus research
Edited by Kristopher Kyle
[International Journal of Learner Corpus Research 7:1] 2021
► pp. 83–130

Analyzing the linguistic complexity of German learner language in a reading comprehension task

Using proficiency classification to investigate short answer data, cross-data generalizability, and the impact of linguistic analysis quality

| University of Tübingen
| University of Tübingen
While traditionally linguistic complexity analysis of learner language is mostly based on essays, there is increasing interest in other task types. This is crucial for obtaining a broader empirical basis for characterizing language proficiency and highlights the need to advance our understanding of how task and learner properties interact in shaping the linguistic complexity of learner productions. It also makes it important to determine which complexity measures generalize well across which tasks.In this paper, we investigate the linguistic complexity of answers to reading comprehension questions written by foreign language learners of German at the college level. Analyzing the corpus with computational linguistic methods identifying a wide range of complexity features, we explore which linguistic complexity analyses can successfully be performed for such short answers, how learner proficiency impacts the results, how generalizable they are across different contexts, and how the quality of the underlying analysis impacts the results.
Keywords: complexity analysis, L2 German, proficiency assessment, reading comprehension, natural language processing
Article outline
  • 1.Introduction
  • 2.Related work
  • 3.This study
  • 4.Data
    • 4.1CREG-29K
    • 4.2CREG-KU, CREG-OSU, and CREG-7K
    • 4.3CREG-104
      • 4.3.1Manual annotation of learner language and target hypotheses
  • 5.Automatic complexity analysis
    • 5.1Feature description
      • Lexical complexity
      • Morphological complexity
      • Phrasal complexity
      • Clausal complexity
      • Discourse complexity
      • Language use
      • Human processing
      • Surface measures
    • 5.2System description
  • 6.Determining German L2 proficiency using linguistic complexity analysis
    • 6.1Course-level classification
      • 6.1.1Set-up of study 1
      • 6.1.2Results of study 1
    • 6.2Generalizability of complexity modeling
      • 6.2.1Set-up of study 2
      • 6.2.2Results of study 2
  • 7.Performance of complexity models on learner language
    • 7.1Accuracy of NLP analysis
      • 7.1.1Set-up of study 3.1
      • 7.1.2Results of study 3.1
    • 7.2Effect on linguistic complexity analysis
      • 7.2.1Set-up of study 3.2
      • 7.2.2Results of study 3.2
    • 7.3Effect on proficiency classification
      • 7.3.1Set-up of study 3.3
      • 7.3.2Results of study 3.3
  • 8.Discussion
  • 9.Conclusion
  • Acknowledgements
  • Notes
  • References
Published online: 1 March 2021
DOI logo
https://doi.org/10.1075/ijlcr.20006.wei
References
Alexopoulou, T., Michel, M., Murakami, A., & Meurers, D.
(2017) Task effects on linguistic complexity and accuracy: A large-scale learner corpus analysis employing natural language processing techniques. Language Learning, 671, 181–209. DOI logoGoogle Scholar
Biber, D., Gray, B., & Staples, S.
(2016) Predicting patterns of grammatical complexity across language exam task types and proficiency levels. Applied Linguistics, 37(5), 639–668. DOI logoGoogle Scholar
Björkelund, A., Bohnet, B., Hafdell, L., & Nugues, P.
(2010) A high-performance syntactic and semantic dependency parser. In Demonstration volume of the 23rd COLING (pp. 23–27). Beijing.Google Scholar
Bohnet, B., & Nivre, J.
(2012) A transition-based system for joint part-of-speech tagging and labeled non-projective dependency parsing. In Proceedings of the 2012 joint conference on EMNLP and computational natural language learning (pp. 1455–1465). Jeju Island, Korea: Association for Computational Linguistics.Google Scholar
Brants, S., Dipper, S., Hansen, S., Lezius, W., & Smith, G.
(2002) The TIGER treebank. In Proceedings of the workshop on treebanks and linguistic theories. Sozopol.Google Scholar
Brants, T., Skut, W., & Uszkoreit, H.
(1999) Syntactic annotation of a German newspaper corpus. In Proceedings of the ATALA treebank workshop. Paris.Google Scholar
Brezina, V., & Pallotti, G.
(2019) Morphological complexity in written l2 texts. Second Language Research, 35(1), 99–119. DOI logoGoogle Scholar
Brown, C., Snodgrass, T., Kemper, S. J., Herman, R., & Covington, M. A.
(2008) Automatic measurement of propositional idea density from part-of-speech tagging. Behavior Research Methods, 40(2), 540–545. DOI logoGoogle Scholar
Brysbaert, M., Buchmeier, M., Conrad, M., Jacobs, A. M., Bölte, J., & Böhl, A.
(2011) The word frequency effect: A review of recent developments and implications for the choice of frequency estimates in German. Experimental Psychology, 581, 412–424. DOI logoGoogle Scholar
Caines, A., & Buttery, P.
(2017) The effect of task and topic on opportunity of use in learner corpora. In Learner corpus research: New perspectives and applications. London: Bloomsbury.Google Scholar
Chen, D., & Manning, C.
(2014) A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 conference on EMNLP (pp. 740–750). Doha, Qatar.Google Scholar
Crossley, S. A.
(2020) Linguistic features in writing quality and development: An overview. Journal of Writing Research, 11(3), 415–443. DOI logoGoogle Scholar
Crossley, S. A., Skalicky, S., & Dascalu, M.
(2019) Moving beyond classic readability formulas: new methods and new models. Journal of Research in Reading, 42(3–4), 541–561. DOI logoGoogle Scholar
Crossley, S. A., Weston, J. L., Sullivan, S. T. M., & McNamara, D. S.
(2011) The development of writing proficiency as a function of grade level: A linguistic analysis. Written Communication, 28(3), 282–311. DOI logoGoogle Scholar
De Clercq, B., & Housen, A.
(2019) The development of morphological complexity: A cross-linguistic study of L2 French and English. Second Language Research Special Issue on Linguistic Complexity, 35(1), 71–97.Google Scholar
Dell’Orletta, F., Montemagni, S., & Venturi, G.
(2014) Assessing document and sentence readability in less resourced languages and across textual genres. Recent Advances in Automatic Readability Assessment and Text Simplification. Special issue of the International Journal of Applied Linguistics, 165(2), 163–193.Google Scholar
Díaz-Negrillo, A., Meurers, D., Valera, S., & Wunsch, H.
(2010) Towards interlanguage POS annotation for effective learner corpora in SLA and FLT. Language Forum, 36(1–2), 139–154.Google Scholar
Duden
(2009) Deutsche Grammatik (4th ed., Vol. 41). Dudenverlag.Google Scholar
Ellis, N. C.
(2002) Frequency effecs in language processing. A review with implications for theories of implicit and explicit language acquisition. Studies in Second Language Acquisition, 24(2), 143–188. DOI logoGoogle Scholar
Ellis, R.
(2003) Task-based language learning and teaching. Oxford, UK: Oxford University Press.Google Scholar
François, T., & Fairon, C.
(2012) An “AI readability” formula for French as a foreign language. In Proceedings of the 2012 joint conference on EMNLP and computational natural language learning.Google Scholar
Galasso, S.
(2014) Exploring textual cohesion characteristics for German readability classification (Bachelor Thesis in Computational Linguistics). Department of Linguistics, University of Tübingen. ([URL])
Geertzen, J., Alexopoulou, T., & Korhonen, A.
(2013) Automatic linguistic annotation of large scale L2 databases: The EF-Cambridge open language database (EFCAMDAT). In Proceedings of the 31st SLRF. Cascadilla Press.Google Scholar
Gibson, E.
(2000) The dependency locality theory: A distance-based theory of linguistic complexity. In A. Marantz, Y. Miyashita, & W. O’Neil (Eds.), Image, language, brain: papers from the first mind articulation project symposium (pp. 95–126). MIT.Google Scholar
Goldhahn, D., Eckart, T., & Quasthoff, U.
(2012) Building large monolingual dictionaries at the leipzig corpora collection: From 100 to 200 languages. Proceedings of the 8th International Language Ressources and Evaluation, 759–765.Google Scholar
Hamp, B., & Feldweg, H.
(1997) GermaNet – a lexical-semantic net for German. In Proceedings of ACL workshop automatic information extraction and building of lexical semantic resources for NLP applications. Madrid.Google Scholar
Hancke, J.
(2013) Automatic prediction of CEFR proficiency levels based on linguistic features of learner language (Unpublished master’s thesis). Department of Linguistics, University of Tübingen.Google Scholar
Hancke, J., Vajjala, S., & Meurers, D.
(2012) Readability classification for German using lexical, syntactic, and morphological features. In Proceedings of the 24th COLING (pp. 1063–1080). Mumbay, India.Google Scholar
Heister, J., Würzner, K.-M., Bubenzer, J., Pohl, E., Hanneforth, T., Geyken, A., & Kliegl, R.
(2011) dlexDB – eine lexikalische Datenbank für die psychologische und linguistische Forschung. Psychologische Rundschau, 621, 10–20. DOI logoGoogle Scholar
Höhle, T. N.
(1986) Der Begriff ‘Mittelfeld’. Anmerkungen über die Theorie der topologischen Felder. In A. Schöne (Ed.), Kontroversen alte und neue. Akten des VII. Internationalen Germanistenkongresses Göttingen 1985 (pp. 329–340). Tübingen: Niemeyer. (Bd. 3)Google Scholar
Housen, A., De Clercq, B., Kuiken, F., & Vedder, I.
(2019) Multiple approaches to complexity in second language research. Second Language Research. Special Issue on Linguistic Complexity, 35(1), 2–31.Google Scholar
Housen, A., & Kuiken, F.
(2009) Complexity, accuracy and fluency in second language acquisition. Applied Linguistics, 30(4), 461–473. DOI logoGoogle Scholar
Housen, A., Kuiken, F., & Vedder, I.
(2012) Complexity, accuracy and fluency: Definitions, measurement and research. In A. Housen, F. Kuiken, & I. Vedder (Eds.), Dimensions of L2 performance and proficiency (pp. 1–20). John Benjamins. DOI logoGoogle Scholar
Hunt, K. W.
(1965) A synopsis of clause-to-sentence length factors. The English Journal, 54(4), 300+305-309. DOI logoGoogle Scholar
Lavalley, R., Berkling, K., & Stüker, S.
(2015) Preparing children’s writing database for automated processing. In Proceedings of the workshop on language teaching, learning and technology at speech and language technologies in education (pp. 9–15).Google Scholar
Lüdeling, A.
(2008) Mehrdeutigkeiten und Kategorisierung: Probleme bei der Annotation von Lernerkorpora. In M. Walter & P. Grommes (Eds.), Fortgeschrittene Lernervarietäten: Korpuslinguistik und Zweispracherwerbsforschung (pp. 119–140). Tübingen: Max Niemeyer Verlag.Google Scholar
Lüdeling, A., Walter, M., Kroymann, E., & Adolphs, P.
(2005) Multi-level error annotation in learner corpora. In Proceedings of corpus linguistics. Birmingham.Google Scholar
McCarthy, P. M.
(2005) An assessment of the range and usefulness of lexical diversity measures and the potential of the measure of textual, lexical diversity (MTLD) (Unpublished doctoral dissertation). University of Memphis.Google Scholar
McCarthy, P. M., & Jarvis, S.
(2010) MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42(2), 381–392. DOI logoGoogle Scholar
Meurers, D.
(2005) On the use of electronic corpora for theoretical linguistics. case studies from the syntax of German. Lingua, 115(11), 1619–1639. DOI logoGoogle Scholar
(2015) Learner corpora and natural language processing. In S. Granger, G. Gilquin, & F. Meunier (Eds.), The cambridge handbook of learner corpus research (pp. 537–566). Cambridge University Press. DOI logoGoogle Scholar
(2020) Natural language processing and language learning. In C. A. Chapelle (Ed.), The concise encyclopedia of applied linguistics (pp. 817–831). Oxford: Wiley.Google Scholar
Meurers, D., & Dickinson, M.
(2017) Evidence and interpretation in language learning research: Opportunities for collaboration with computational linguistics. Language Learning, 67(2). DOI logoGoogle Scholar
Michel, M., Murakami, A., Alexopoulou, T., & Meurers, D.
(2019) Effects of task type on morphosyntactic complexity across proficiency: Evidence from a large learner corpus of A1 to C2 writings. Instructed Second Language Acquisition, 31, 124–152. DOI logoGoogle Scholar
Ott, N., & Ziai, R.
(2010) Evaluating dependency parsing performance on German learner language. In M. Dickinson, K. Müürisep, & M. Passarotti (Eds.), Proceedings of the ninth international workshop on treebanks and linguistic theories (Vol. 91, pp. 175–186). Tartu, Estonia: Tartu University Press. [URL]Google Scholar
Ott, N., Ziai, R., & Meurers, D.
(2012) Creation and analysis of a reading comprehension exercise corpus: Towards evaluating meaning in context. In T. Schmidt & K. Wörner (Eds.), Multilingual corpora and multilingual corpus analysis (pp. 47–69). Amsterdam: Benjamins. DOI logoGoogle Scholar
Petrov, S., & Klein, D.
(2007) Improved inference for unlexicalized parsing. In Proceedings of the NAACL main conference (pp. 404–411). Rochester, New York.Google Scholar
Pilán, I., Vajjala, S., & Volodina, E.
(2015) A readable read: Automatic assessment of language learning materials based on linguistic complexity. In Proceedings of CICLING 2015.Google Scholar
Reis, M.
(2001) Bilden Modalverben im Deutschen eine syntaktische Klasse? In R. Müller & M. Reis (Eds.), Modalität und Modalverben im Deutschen. Hamburg: Helmut Buske. (Linguistische Berichte – Sonderhefte)Google Scholar
Seeker, W., & Kuhn, J.
(2012) Making ellipses explicit in dependency conversion for a German treebank. In Proceedings of the 8th international conference on language resources and evaluation (pp. 3132–3139). Istanbul, Turkey.Google Scholar
Shain, C., van Schijndel, M., Futrell, R., Gibson, E., & Schuler, W.
(2016) Memory access during incremental sentence processing causes reading time latency. In Proceedings of the workshop on computational linguistics for linguistic complexity (p. 49–58). Osaka.Google Scholar
Staples, S., Egbert, J., Biber, D., & Gray, B.
(2016) Academic writing development at the university level: Phrasal and clausal complexity across level of study, discipline, and genre. Written Communication, 33(2), 149–183. DOI logoGoogle Scholar
Tagliamonte, S. A.
(2011) Variationist sociolinguistics: Change, observation, interpretation. John Wiley & Sons.Google Scholar
Telljohann, H., Hinrichs, E., & Kübler, S.
(2004) The TüBa-D/Z treebank: Annotating German with a context-free backbone. In Proceedings of the fourth LREC. Lissabon.Google Scholar
Tharwat, A.
(2018) Classification assessment methods. Applied Computing and Informatics.Google Scholar
Thielen, C., Schiller, A., Teufel, S., & Stöckert, C.
(1999) Guidelines für das Tagging deutscher Textkorpora mit STTS (Tech. Rep.). Stuttgart/Tübingen: Institut für Maschinelle Sprachverarbeitung Stuttgart and Seminar für Sprachwissenschaft Tübingen.Google Scholar
Tracy-Ventura, N., & Myles, F.
(2015) The importance of task variability in the design of learner corpora for SLA research. International Journal of Learner Corpus Research, 1(1), 58–95. DOI logoGoogle Scholar
Vajjala, S., & Meurers, D.
(2012) On improving the accuracy of readability classification using insights from second language acquisition. In Proceedings of the seventh BEA workshop (pp. 163–173).Google Scholar
Weiss, Z.
(2015) More linguistically motivated features of language complexity in readability classification of German textbooks: Implementation and evaluation (Bachelor’s Thesis). Department of Linguistics, University of Tübingen. ([URL])
(2017) Using measures of linguistic complexity to assess German L2 proficiency in learner corpora under consideration of task-effects (Unpublished master’s thesis). University of Tübingen, Germany. ([URL])
Weiss, Z., & Meurers, D.
(2018) Modeling the readability of German targeting adults and children: An empirically broad analysis and its cross-corpus validation. In Proceedings of the 27th COLING. Santa Fe, New Mexico, USA. [URL]Google Scholar
(2019a) Analyzing linguistic complexity and accuracy in academic language development of German across elementary and secondary school. In Proceedings of the 14th BEA workshop. Florence, Italy. DOI logoGoogle Scholar
(2019b) Broad linguistic modeling is beneficial for German L2 proficiency assessment. In A. Abel, A. Glaznieks, V. Lyding, & L. Nicolas (Eds.), Widening the scope of learner corpus research. Selected papers from the fourth learner corpus research conference. Louvain-La-Neuve: Presses Universitaires de Louvain.Google Scholar
Wöllstein, A.
(2014) Topologisches Satzmodell (2nd ed.). Heidelberg: Winter.Google Scholar
Yoon, H.-J.
(2017) Linguistic complexity in L2 writing revisited: Issues of topic, proficiency, and construct multidimensionality. System, 661, 130–141. DOI logoGoogle Scholar
Yoon, H.-J., & Polio, C.
(2016) The linguistic development of students of English as a second language in two written genres. TESOL Quarterly, 275–301.Google Scholar
Ziai, R.
(2018) Short answer assessment in context: The role of information structure (Unpublished doctoral dissertation). Eberhard-Karls Universität Tübingen.Google Scholar
Ziegler, N.
(2018) Pre-task planning in L2 text-chat: Examining learners’ process and performance. Language Learning & Technology, 22(3), 193–213.Google Scholar