Analyzing the linguistic complexity of German learner language in a reading comprehension task
Using proficiency classification to investigate short answer data, cross-data generalizability, and the impact of linguistic
analysis quality
While traditionally linguistic complexity analysis of learner language is mostly based on essays, there is increasing
interest in other task types. This is crucial for obtaining a broader empirical basis for characterizing language proficiency and highlights
the need to advance our understanding of how task and learner properties interact in shaping the linguistic complexity of learner
productions. It also makes it important to determine which complexity measures generalize well across which tasks.
In this paper, we investigate the linguistic complexity of answers to reading comprehension questions written by foreign
language learners of German at the college level. Analyzing the corpus with computational linguistic methods identifying a wide range of
complexity features, we explore which linguistic complexity analyses can successfully be performed for such short answers, how learner
proficiency impacts the results, how generalizable they are across different contexts, and how the quality of the underlying analysis
impacts the results.
Article outline
- 1.Introduction
- 2.Related work
- 3.This study
- 4.Data
- 4.1CREG-29K
- 4.2CREG-KU, CREG-OSU, and CREG-7K
- 4.3CREG-104
- 4.3.1Manual annotation of learner language and target hypotheses
- 5.Automatic complexity analysis
- 5.1Feature description
- Lexical complexity
- Morphological complexity
- Phrasal complexity
- Clausal complexity
- Discourse complexity
- Language use
- Human processing
- Surface measures
- 5.2System description
- 6.Determining German L2 proficiency using linguistic complexity analysis
- 6.1Course-level classification
- 6.1.1Set-up of study 1
- 6.1.2Results of study 1
- 6.2Generalizability of complexity modeling
- 6.2.1Set-up of study 2
- 6.2.2Results of study 2
- 7.Performance of complexity models on learner language
- 7.1Accuracy of NLP analysis
- 7.1.1Set-up of study 3.1
- 7.1.2Results of study 3.1
- 7.2Effect on linguistic complexity analysis
- 7.2.1Set-up of study 3.2
- 7.2.2Results of study 3.2
- 7.3Effect on proficiency classification
- 7.3.1Set-up of study 3.3
- 7.3.2Results of study 3.3
- 8.Discussion
- 9.Conclusion
- Acknowledgements
- Notes
-
References
References (68)
Alexopoulou, T., Michel, M., Murakami, A., & Meurers, D.
(
2017)
Task effects on linguistic complexity and accuracy: A large-scale learner corpus analysis employing natural language processing techniques.
Language Learning, 671, 181–209.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Biber, D., Gray, B., & Staples, S.
(
2016)
Predicting patterns of grammatical complexity across language exam task types and proficiency levels.
Applied Linguistics, 37(5), 639–668.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Björkelund, A., Bohnet, B., Hafdell, L., & Nugues, P.
(
2010)
A high-performance syntactic and semantic dependency parser. In
Demonstration volume of the 23rd COLING (pp. 23–27). Beijing.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Bohnet, B., & Nivre, J.
(
2012)
A transition-based system for joint part-of-speech tagging and labeled non-projective dependency parsing. In
Proceedings of the 2012 joint conference on EMNLP and computational natural language learning (pp. 1455–1465). Jeju Island, Korea: Association for Computational Linguistics.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Brants, S., Dipper, S., Hansen, S., Lezius, W., & Smith, G.
(
2002)
The TIGER treebank. In
Proceedings of the workshop on treebanks and linguistic theories. Sozopol.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Brants, T., Skut, W., & Uszkoreit, H.
(
1999)
Syntactic annotation of a German newspaper corpus. In
Proceedings of the ATALA treebank workshop. Paris.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Brezina, V., & Pallotti, G.
(
2019)
Morphological complexity in written l2 texts.
Second Language Research, 35(1), 99–119.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Brown, C., Snodgrass, T., Kemper, S. J., Herman, R., & Covington, M. A.
(
2008)
Automatic measurement of propositional idea density from part-of-speech tagging.
Behavior Research Methods, 40(2), 540–545.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Brysbaert, M., Buchmeier, M., Conrad, M., Jacobs, A. M., Bölte, J., & Böhl, A.
(
2011)
The word frequency effect: A review of recent developments and implications for the choice of frequency estimates in German.
Experimental Psychology, 581, 412–424.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Caines, A., & Buttery, P.
(
2017)
The effect of task and topic on opportunity of use in learner corpora. In
Learner corpus research: New perspectives and applications. London: Bloomsbury.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Chen, D., & Manning, C.
(
2014)
A fast and accurate dependency parser using neural networks. In
Proceedings of the 2014 conference on EMNLP (pp. 740–750). Doha, Qatar.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Crossley, S. A.
(
2020)
Linguistic features in writing quality and development: An overview.
Journal of Writing Research, 11(3), 415–443.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Crossley, S. A., Skalicky, S., & Dascalu, M.
(
2019)
Moving beyond classic readability formulas: new methods and new models.
Journal of Research in Reading, 42(3–4), 541–561.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Crossley, S. A., Weston, J. L., Sullivan, S. T. M., & McNamara, D. S.
(
2011)
The development of writing proficiency as a function of grade level: A linguistic analysis.
Written Communication, 28(3), 282–311.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
De Clercq, B., & Housen, A.
(
2019)
The development of morphological complexity: A cross-linguistic study of L2 French and English.
Second Language Research Special Issue on Linguistic Complexity, 35(1), 71–97.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Dell’Orletta, F., Montemagni, S., & Venturi, G.
(
2014)
Assessing document and sentence readability in less resourced languages and across textual genres.
Recent Advances in Automatic Readability Assessment and Text Simplification. Special issue of the International Journal of Applied Linguistics, 165(2), 163–193.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Díaz-Negrillo, A., Meurers, D., Valera, S., & Wunsch, H.
(
2010)
Towards interlanguage POS annotation for effective learner corpora in SLA and FLT.
Language Forum, 36(1–2), 139–154.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Duden
(
2009)
Deutsche Grammatik (4th ed., Vol. 41). Dudenverlag.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Ellis, N. C.
(
2002)
Frequency effecs in language processing. A review with implications for theories of implicit and explicit language acquisition.
Studies in Second Language Acquisition, 24(2), 143–188.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Ellis, R.
(
2003)
Task-based language learning and teaching. Oxford, UK: Oxford University Press.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
François, T., & Fairon, C.
(
2012)
An “AI readability” formula for French as a foreign language. In
Proceedings of the 2012 joint conference on EMNLP and computational natural language learning.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Galasso, S.
(
2014)
Exploring textual cohesion characteristics for German readability classification (Bachelor Thesis in Computational Linguistics). Department of Linguistics, University of Tübingen. (
[URL])
Geertzen, J., Alexopoulou, T., & Korhonen, A.
(
2013)
Automatic linguistic annotation of large scale L2 databases: The EF-Cambridge open language database (EFCAMDAT). In
Proceedings of the 31st SLRF. Cascadilla Press.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Gibson, E.
(
2000)
The dependency locality theory: A distance-based theory of linguistic complexity. In
A. Marantz,
Y. Miyashita, &
W. O’Neil (Eds.),
Image, language, brain: papers from the first mind articulation project symposium (pp. 95–126). MIT.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Goldhahn, D., Eckart, T., & Quasthoff, U.
(
2012)
Building large monolingual dictionaries at the leipzig corpora collection: From 100 to 200 languages.
Proceedings of the 8th International Language Ressources and Evaluation, 759–765.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hamp, B., & Feldweg, H.
(
1997)
GermaNet – a lexical-semantic net for German. In
Proceedings of ACL workshop automatic information extraction and building of lexical semantic resources for NLP applications. Madrid.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hancke, J.
(
2013)
Automatic prediction of CEFR proficiency levels based on linguistic features of learner language (Unpublished master’s thesis). Department of Linguistics, University of Tübingen.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hancke, J., Vajjala, S., & Meurers, D.
(
2012)
Readability classification for German using lexical, syntactic, and morphological features. In
Proceedings of the 24th COLING (pp. 1063–1080). Mumbay, India.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Heister, J., Würzner, K.-M., Bubenzer, J., Pohl, E., Hanneforth, T., Geyken, A., & Kliegl, R.
(
2011)
dlexDB – eine lexikalische Datenbank für die psychologische und linguistische Forschung.
Psychologische Rundschau, 621, 10–20.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Höhle, T. N.
(
1986)
Der Begriff ‘Mittelfeld’. Anmerkungen über die Theorie der topologischen Felder. In
A. Schöne (Ed.),
Kontroversen alte und neue. Akten des VII. Internationalen Germanistenkongresses Göttingen 1985 (pp. 329–340). Tübingen: Niemeyer. (
Bd. 3)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Housen, A., De Clercq, B., Kuiken, F., & Vedder, I.
(
2019)
Multiple approaches to complexity in second language research.
Second Language Research. Special Issue on Linguistic Complexity, 35(1), 2–31.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Housen, A., & Kuiken, F.
(
2009)
Complexity, accuracy and fluency in second language acquisition.
Applied Linguistics, 30(4), 461–473.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Housen, A., Kuiken, F., & Vedder, I.
Hunt, K. W.
(
1965)
A synopsis of clause-to-sentence length factors.
The English Journal, 54(4), 300+305-309.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Lavalley, R., Berkling, K., & Stüker, S.
(
2015)
Preparing children’s writing database for automated processing. In
Proceedings of the workshop on language teaching, learning and technology at speech and language technologies in education (pp. 9–15).
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Lüdeling, A.
(
2008)
Mehrdeutigkeiten und Kategorisierung: Probleme bei der Annotation von Lernerkorpora. In
M. Walter &
P. Grommes (Eds.),
Fortgeschrittene Lernervarietäten: Korpuslinguistik und Zweispracherwerbsforschung (pp. 119–140). Tübingen: Max Niemeyer Verlag.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Lüdeling, A., Walter, M., Kroymann, E., & Adolphs, P.
(
2005)
Multi-level error annotation in learner corpora. In
Proceedings of corpus linguistics. Birmingham.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
McCarthy, P. M.
(
2005)
An assessment of the range and usefulness of lexical diversity measures and the potential of the measure of textual, lexical diversity (MTLD) (Unpublished doctoral dissertation). University of Memphis.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
McCarthy, P. M., & Jarvis, S.
(
2010)
MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment.
Behavior Research Methods, 42(2), 381–392.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Meurers, D.
(
2005)
On the use of electronic corpora for theoretical linguistics. case studies from the syntax of German.
Lingua, 115(11), 1619–1639.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Meurers, D.
(
2015)
Learner corpora and natural language processing. In
S. Granger,
G. Gilquin, &
F. Meunier (Eds.),
The cambridge handbook of learner corpus research (pp. 537–566). Cambridge University Press.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Meurers, D.
(
2020)
Natural language processing and language learning. In
C. A. Chapelle (Ed.),
The concise encyclopedia of applied linguistics (pp. 817–831). Oxford: Wiley.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Meurers, D., & Dickinson, M.
(
2017)
Evidence and interpretation in language learning research: Opportunities for collaboration with computational linguistics.
Language Learning, 67(2).
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Michel, M., Murakami, A., Alexopoulou, T., & Meurers, D.
(
2019)
Effects of task type on morphosyntactic complexity across proficiency: Evidence from a large learner corpus of A1 to C2 writings.
Instructed Second Language Acquisition, 31, 124–152.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Ott, N., & Ziai, R.
(
2010)
Evaluating dependency parsing performance on German learner language. In
M. Dickinson,
K. Müürisep, &
M. Passarotti (Eds.),
Proceedings of the ninth international workshop on treebanks and linguistic theories (Vol. 91, pp. 175–186). Tartu, Estonia: Tartu University Press.
[URL]![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Ott, N., Ziai, R., & Meurers, D.
Petrov, S., & Klein, D.
(
2007)
Improved inference for unlexicalized parsing. In
Proceedings of the NAACL main conference (pp. 404–411). Rochester, New York.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pilán, I., Vajjala, S., & Volodina, E.
(
2015)
A readable read: Automatic assessment of language learning materials based on linguistic complexity. In
Proceedings of CICLING 2015.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Reis, M.
(
2001)
Bilden Modalverben im Deutschen eine syntaktische Klasse? In
R. Müller &
M. Reis (Eds.),
Modalität und Modalverben im Deutschen. Hamburg: Helmut Buske. (Linguistische Berichte – Sonderhefte)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Seeker, W., & Kuhn, J.
(
2012)
Making ellipses explicit in dependency conversion for a German treebank. In
Proceedings of the 8th international conference on language resources and evaluation (pp. 3132–3139). Istanbul, Turkey.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Shain, C., van Schijndel, M., Futrell, R., Gibson, E., & Schuler, W.
(
2016)
Memory access during incremental sentence processing causes reading time latency. In
Proceedings of the workshop on computational linguistics for linguistic complexity (p. 49–58). Osaka.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Staples, S., Egbert, J., Biber, D., & Gray, B.
(
2016)
Academic writing development at the university level: Phrasal and clausal complexity across level of study, discipline, and genre.
Written Communication, 33(2), 149–183.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Tagliamonte, S. A.
(
2011)
Variationist sociolinguistics: Change, observation, interpretation. John Wiley & Sons.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Telljohann, H., Hinrichs, E., & Kübler, S.
(
2004)
The TüBa-D/Z treebank: Annotating German with a context-free backbone. In
Proceedings of the fourth LREC. Lissabon.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Tharwat, A.
(
2018)
Classification assessment methods.
Applied Computing and Informatics.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Thielen, C., Schiller, A., Teufel, S., & Stöckert, C.
(
1999)
Guidelines für das Tagging deutscher Textkorpora mit STTS (Tech. Rep.). Stuttgart/Tübingen: Institut für Maschinelle Sprachverarbeitung Stuttgart and Seminar für Sprachwissenschaft Tübingen.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Tracy-Ventura, N., & Myles, F.
Vajjala, S., & Meurers, D.
(
2012)
On improving the accuracy of readability classification using insights from second language acquisition. In
Proceedings of the seventh BEA workshop (pp. 163–173).
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Weiss, Z.
(
2015)
More linguistically motivated features of language complexity in readability classification of German textbooks: Implementation and evaluation (Bachelor’s Thesis). Department of Linguistics, University of Tübingen. (
[URL])
Weiss, Z.
(
2017)
Using measures of linguistic complexity to assess German L2 proficiency in learner corpora under consideration of task-effects (Unpublished master’s thesis). University of Tübingen, Germany. (
[URL])
Weiss, Z., & Meurers, D.
(
2018)
Modeling the readability of German targeting adults and children: An empirically broad analysis and its cross-corpus validation. In
Proceedings of the 27th COLING. Santa Fe, New Mexico, USA.
[URL]![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Weiss, Z., & Meurers, D.
(
2019a)
Analyzing linguistic complexity and accuracy in academic language development of German across elementary and secondary school. In
Proceedings of the 14th BEA workshop. Florence, Italy.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Weiss, Z., & Meurers, D.
(
2019b)
Broad linguistic modeling is beneficial for German L2 proficiency assessment. In
A. Abel,
A. Glaznieks,
V. Lyding, &
L. Nicolas (Eds.),
Widening the scope of learner corpus research. Selected papers from the fourth learner corpus research conference. Louvain-La-Neuve: Presses Universitaires de Louvain.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Wöllstein, A.
(
2014)
Topologisches Satzmodell (2nd ed.). Heidelberg: Winter.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Yoon, H.-J.
(
2017)
Linguistic complexity in L2 writing revisited: Issues of topic, proficiency, and construct multidimensionality.
System, 661, 130–141.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Yoon, H.-J., & Polio, C.
(
2016)
The linguistic development of students of English as a second language in two written genres.
TESOL Quarterly, 275–301.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Ziai, R.
(
2018)
Short answer assessment in context: The role of information structure (Unpublished doctoral dissertation). Eberhard-Karls Universität Tübingen.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Ziegler, N.
(
2018)
Pre-task planning in L2 text-chat: Examining learners’ process and performance.
Language Learning & Technology, 22(3), 193–213.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Cited by (3)
Cited by 3 other publications
Shin, Gyu-Ho, Boo Kyung Jung & Seongmin Mun
2024.
Transformer-based text similarity and second language proficiency: A case of written production by learners of Korean.
Natural Language Processing Journal 6
► pp. 100060 ff.
![DOI logo](//benjamins.com/logos/doi-logo.svg)
Möhring, Jupp & Monika Ruszó
2023.
Schreibkompetenz als Variable des Studienerfolgs von internationalen Studienanfänger*innen an deutschen Hochschulen. In
IDT 2022: *mit.sprache.teil.haben Band 1: Mit Sprache handeln,
► pp. 315 ff.
![DOI logo](//benjamins.com/logos/doi-logo.svg)
Weiss, Zarah, Kim Lange-Schubert, Barbara Geist & Detmar Meurers
2022.
Sprachliche Komplexität im Unterricht.
Zeitschrift für germanistische Linguistik 50:1
► pp. 159 ff.
![DOI logo](//benjamins.com/logos/doi-logo.svg)
This list is based on CrossRef data as of 4 july 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.