Syntactic complexity measures as linguistic correlates of proficiency level in learner Russian
The study reports on the results of a corpus-based evaluation of automatically extracted syntactic complexity measures as indices of Russian as a foreign language (FL) and Russian as a heritage language (HL) writing development. A list of 12 syntactic complexity measures was tested on a set of longitudinal, classroom-based data. The analyses demonstrated that the syntactic complexity measures help delineate four proficiency levels (Intermediate Mid, Intermediate High, Advanced Low and Advanced Mid as established through the ACTFL proficiency guidelines), and that the changes in syntactic complexity indices across levels pattern slightly differently in the FL vs. HL learner groups. Our research confirms that the overall trends in interlanguage development in Russian align with the “complexification” trends found for other second languages.
Article outline
- 1.Introduction
- 2.Syntactic complexity
- 2.1Syntactic complexity: Key concepts and considerations
- 2.2Syntactic complexity indices as correlates of language development
- 2.3Language proficiency as a proficiency scale rating
- 2.4Syntactic complexity measures in learner Russian
- 3.The method
- 3.1Participants and data
- 3.2Data annotation
- 4.Analyses
- 5.Results
- 5.1Descriptive results
- 5.2Correlations between syntactic complexity measures
- 5.3Analysis of variance between proficiency levels and language backgrounds
- 5.4Ranking syntactic measures using a machine learning classifier
- 6.Discussion and conclusion
-
Notes
-
References
References
Adams, Rebecca, Alwi, Nik M., Aloesnita, Nik & Newton, Jonathan
2015 Task complexity effects on the complexity and accuracy of writing via text chat.
Journal of Second Language Writing 29: 64–81.
Alexopoulou, Theodora, Michel, Marije, Murakami, Akira & Meurers, Detmar
2017 Task effects on linguistic complexity and accuracy: A large-scale corpus analysis employing natural language processing techniques.
Language Learning 67(S1): 180–208.
Alsufieva, Anna A., Kisselev, Olesya V. & Freels, Sandra G.
2012 Results 2012: Using flagship data to develop a Russian learner corpus of academic writing.
Russian Language Journal/
Русский зяык 62: 79–105.
Bachman, Lyle F.
1988 Problems in examining the validity of the ACTFL oral proficiency interview.
Studies in Second Language Acquisition 10(2): 149–164.
Bailyn, John F.
2012 The Syntax of Russian. Cambridge: CUP.
Ballier, Nicolas, Gaillat, Thomas, Simpkin, Andrew, Stearns, Bernardo, Bouyé, Manon & Zrrouk, Manel
2019 A supervised model for the automatic assessment of language levels based on learner errors. In
Transforming Learning with Meaningful Technologies,
Maren Scheffel,
Julien Broisin,
Viktoria Pammer-Schindler,
Andri Ioannou &
Jan Schneider (eds), 308–320. Cham: Springer.
Barkaoui, Khaled & Hadidi, Ali
2020 Assessing Change in English Second Language Writing Performance. New York NY: Routledge.
Barykina, Alevtina N., Burmistrova, Valentina P. & Dobrovol’skaia, Valeria V.
1978 Posobie po razvitiiu navykov pis’mennoi rechi. Moscow: Russkii iazyk.
Biber, Douglas, Gray, Bethany & Poonpon, Kornwipa
2011 Should we use characteristics of conversation to measure grammatical complexity in L2 writing development? TESOL Quarterly 45(1): 5–35.
Bulté, Bram & Housen, Alex
2012 Defining and operationalising L2 complexity. In
Dimensions of L2 Performance and Proficiency: Complexity, Accuracy and Fluency in SLA [
Language Learning & Language Teaching 32],
Alex Housen,
Folkert Kuiken &
Ineke Vedder (eds), 21–46. Amsterdam: John Benjamins.
Bulté, Bram & Housen, Alex
2014 Conceptualizing and measuring short-term changes in L2 writing complexity.
Journal of Second Language Writing 26: 42–65.
Callies, Marcus & Götz, Sandra
Chen, Yu-Hua & Baker, Paul
2014 Investigating criterial discourse features across second language development: Lexical bundles in rated learner essays, CEFR B1, B2 and C1.
Applied Linguistics 37(6): 849–880.
Crossley, Scott A. & McNamara, Danielle S.
2014 Does writing development equal writing quality? A computational investigation of syntactic complexity in L2 learners.
Journal of Second Language Writing 26: 66–79.
Dengub, Evgeny
2012 Investigating Syntactic and Lexical Complexity, Accuracy, and Fluency in the Writing of Heritage Speakers of Russian. PhD dissertation, Bryn Mawr College.
de Marneffe, Marie-Catherine, Dozat, Timothy, Silveira, Natalia, Haverinen, Katri, Ginter, Filip, Nivre, Joakim & Manning, Christopher D.
2014 Universal Stanford dependencies: A cross-linguistic typology. In
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), 4585–4592. Reykjavik: European Language Resources Association (ELRA).
[URL] (15 December 2021).
Diessel, Holger
2004 The Acquisition of Complex Sentences [
Cambridge Studies in Linguistics 105]. Cambridge: CUP.
Friginal, Eric & Weigle, Sandra
2014 Exploring multiple profiles of L2 writing using multi-dimensional analysis.
Journal of Second Language Writing 26: 80–95.
Freels, Sandra, Kisselev, Olesya & Alsufieva, Anna
2016 Adding breadth to the undergraduate curriculum. Flagship approaches to interdisciplinary language learning. In
Exploring the US Language Flagship Program: Professional Competence in a Second Language by Graduation [
New Perspectives on Language and Education 50],
Dianna Murphy &
Karen Evans-Romaine (eds), 51–69. Bristol: Multilingual Matters.
Guo, Liang, Crossley, Scott A. & McNamara, Danielle S.
2013 Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study.
Assessing Writing 18(3): 218–238.
Haspelmath, Martin
2007 Coordination. In
Language Typology and Syntactic Description,
Timothy Shopen (ed.), 1–51. Cambridge: CUP.
Hawkins, John A. & Buttery, Paula
2010 Criterial features in learner corpora: Theory and illustrations.
English Profile Journal 1: e5.
Housen, Alex, De Clercq, Bastien, Kuiken, Folkert & Vedder, Ineke
2019 Multiple approaches to complexity in second language research.
Second Language Research 35(1): 3–21.
Huang, Ting, Steinkrauss, Rasmus & Verspoor, Marjolijn
2021 Variability as predictor in L2 writing proficiency.
Journal of Second Language Writing 52: 100787.
Karlsson, Frank
2010 Syntactic recursion and iteration. In
Recursion and Human Language,
Harry van der Hulst (ed.), 43–67. Berlin: Mouton de Gruyter.
Kisselev, Olesya & Alsufieva, Anna
2017 The development of syntactic complexity in the writing of Russian language learners: A longitudinal corpus study.
Russian Language Journal/
Русский язык 67: 27–54.
Kisselev, Olesya & Comer, William
2019 Interdepartmental collaboration and curriculum design: Creating a Russian environmental sustainability course for advanced study. In
Foreign Language Teaching and the Environment: Theory, Curricula, Institutional Structures,
Charlotte Melin (ed.), 180–196. New York NY: Modern Language Association.
Kisselev, Olesya, Dubinina, Irina & Polinsky, Maria
2020 Form-focused instruction in the heritage language classroom: Toward research-informed heritage language pedagogy.
Frontiers in Education 5: 53.
Kisselev, Olesya, Klimov, Alexandr, & Kopotev, Mikhail
2021 Syntactic complexity measures as indices of language proficiency in writing: Focus on heritage learners of Russian.
Heritage Language Journal 18(3): 1–30.
Khimik, Vasilij. V.
2003 Osnovy naučnoi reči: Učebnoe posobie dlia studentov nefilologičeskih vysših zavedenij. Moscow: Akademiia.
Kuiken, Folkert & Vedder, Ineke
2019 Syntactic complexity across proficiency and languages: L2 and L1 writing in Dutch, Italian and Spanish.
International Journal of Applied Linguistics 29(2): 192–210.
Kyle, Kristopher & Crossley, Scott A.
2018 Measuring syntactic complexity in L2 writing using fine-grained clausal and phrasal indices.
The Modern Language Journal 102(2): 333–349.
Lan, Ge, Liu, Qiandi & Staples, Shelley
2019 Grammatical complexity: ‘What Does It Mean’ and ‘So What’ for L2 writing classrooms? Journal of Second Language Writing 46: 100673.
Lissón, Paula & Ballier, Nicolas
2018 Investigating lexical progression through lexical diversity metrics in a corpus of French L3.
Discours. Revue de linguistique, psycholinguistique et informatique. A Journal of Linguistics, Psycholinguistics and Computational Linguistics 23: 1–26.
Lobanova, Natalja A. & Slesareva, Irma P.
1980 Uchebnik russkogo iazyka dlia inostrannykh studentov-filologov. Sistematiziruiushchii kurs [
Russian Language Handbook for Foreign Students of Philology: A Systematizing Course]. Moscow: Russkii iazyk.
Long, Michael H., Gor, Kira & Jackson, Scott
2012 Linguistic correlates of second language proficiency: Proof of concept with ILR 2–3 in Russian.
Studies in Second Language Acquisition 34(1): 99–126.
Lu, Xiaofei
2011 A corpus-based evaluation of syntactic complexity measures as indices of college-level ESL writers’ language development.
TESOL Quarterly 45(1): 35–62.
Lu, Xiaofei & Ai, Haiyang
2015 Syntactic complexity in college-level English writing: Differences among writers with diverse L1 backgrounds.
Journal of Second Language Writing 29: 16–27.
Mazgutova, Diane & Kormos, Judit
2015 Syntactic and lexical development in an intensive English for academic purposes programme.
Journal of Second Language Writing 29: 3–15.
Menke, Mandy R. & Strawbridge, Tripp
2019 The writing of Spanish majors: A longitudinal analysis of syntactic complexity.
Journal of Second Language Writing 46: 100665.
Michel, Marije, Murakami, Akira, Alexopoulou, Theodora & Meurers, Detmar
2019 Effects of task type on morphosyntactic complexity across proficiency: Evidence from a large learner corpus of A1 to C2 writings.
Instructed Second Language Acquisition 3(2): 124–152.
Montrul, Silvina
2016 The Acquisition of Heritage Languages. Cambridge: CUP.
Mostafa, Tamanna & Crossley, Scott A.
2020 Verb argument construction complexity indices and L2 writing quality: Effects of writing tasks and prompts.
Journal of Second Language Writing 49: 100730.
Norris, John M. & Ortega, Lourdes
2009 Towards an organic approach to investigating CAF in instructed SLA: The case of complexity.
Applied Linguistics 30(4): 555–578.
Ortega, Lourdes
2003 Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college-level L2 writing.
Applied Linguistics 24(4): 492–518.
Osborne, Timothy
2006 Syntax gapping vs. non-gapping coordination.
Linguistische Berichte 207: 307–337.
Pallotti, Gabriele
2009 CAF: Defining, refining and differentiating constructs.
Applied Linguistics 30(4): 590–601.
Petrov, Slav, Das, Dipanjan & McDonald, Ryan
2011 A universal part-of-speech tagset.
arXiv preprint arXiv:1104.2086.
Polat, Nihat, Mahalingappa, Laura & Mancilla, Rae L.
2020 Longitudinal growth trajectories of written syntactic complexity: The case of Turkish learners in an intensive English program.
Applied Linguistics 41(5): 688–711.
Polio, Charlene & Park, Ji-Hyun
2016 Language development in second language writing. In
Handbook of Second and Foreign Language Writing,
Rosa M. Manchón &
Paul Kei Matsuda (eds), 287–306. Berlin: De Gruyter Mouton.
Prokhorova, Kira V.
1998 Naučnyj stil’: učebno-metodičeskoe posobie dlja studentov-žurnalistov. Sankt-Peterburg: Sankt-Peterburg State University.
Reich, Peter & Schütze, Carson
1991 Syntactic embedding: What can people really do? Toronto Working Papers in Linguistics 11: 91–97.
Sag, Ivan A., Gazdar, Gerald, Wasow, Thomas & Weisler, Steven
1985 Coordination and how to distinguish categories.
Natural Language & Linguistic Theory 3(2): 117–171.
Staples, Shelley, Adriana Picoral, Aleksey Novikov, and Bruna Sommer-Farias
2019 Directions for Future Use of Using Existing Corpora in the Study of L2 Writing. In
The Routledge Handbook of Second Language Acquisition and Writing, 356–369. Routledge.
Straka, Milan & Straková, Jana
2017 Tokenizing, POS tagging, lemmatizing and parsing UD 2.0 with UDPipe. In
Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, 88–99. Vancouver: Association for Computational Linguistics.
[URL] (15 December 2022).
Tack, Anaïs, François, Thomas, Roekhaut, Sophie & Fairon, Cédrick
2017 Human and automated CEFR-based grading of short answers. In
Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications.
Joel Tetreault,
Jill Burstein,
Claudia Leacock &
Helen Yannakoudakis (eds), 169–179. Copenhagen: Association of Computational Linguistics.
[URL] (15 December 2021).
Testelets, Yakov G.
2001 Vvedenie v Obščij Sintaksis [
Introduction to General Syntax]. Mocsow: Rossijskij Gosudarstviennyj Gumanitarnyj Universitet (RGGU).
Treffers-Daller, Jeanine, Parslow, Patrick & Williams, Shirley
2018 Back to basics: How measures of lexical diversity can help discriminate between CEFR levels.
Applied Linguistics 39(3): 302–327.
Tschirner, Erwin, Bärenfänger, Olaf & Wanner, Irmgard
2012 Assessing Evidence of Validity of Assigning CEFR Ratings to the ACTFL Oral Proficiency Interview (OPI) and the Oral Proficiency Interview by Computer (OPIc). Leipzig: Universität Leipzig, Herder-Institut.
UDPipe 1 Models
2021 <
[URL] (30 June 2021).
Verspoor, Marjolijn, Schmid, Monika S., Xu, Xiaoyan
2012 A dynamic usage based perspective on L2 writing.
Journal of Second Language Writing 21(3): 239–263.
de Vries, Mark
2005 Coordination and syntactic hierarchy.
Studia Linguistica 59(1): 83–105.
Vyatkina, Nina
2012 The development of second language writing complexity in groups and individuals: A longitudinal learner corpus study.
The Modern Language Journal 96(4): 576–598.
Vyatkina, Nina
2013 Specific syntactic complexity: Developmental profiling of individuals based on an annotated learner corpus.
The Modern Language Journal 97(1): 11–30.
Yang, Weiwei, Lu, Xiaofei & Weigle, Sara C.
2015 Different topics, different discourse: Relationships among writing topic, measures of syntactic complexity, and judgments of writing quality.
Journal of Second Language Writing 28: 53–67.
Yoon, Hyung-Jo & Polio, Charlene
2017 The linguistic development of students of English as a second language in two written genres.
TESOL Quarterly 51(2): 275–301.
Wolfe-Quintero, Kate, Inagaki, Shunji & Kim, Hae-Young
1998 Second Language Development in Writing: Measures of Fluency, Accuracy, & Complexity. Honolulu HI: Second Language Teaching & Curriculum Center, University of Hawai’i at Manoa.
Zeman, Daniel & Resnik, Philip
2008 Cross-language parser adaptation between related languages. In
Proceedings of the IJCNLP-08 Workshop on NLP for Less Privileged Languages, 35–42. Hyderabad: Asian Federation of Natural Language Processing.
[URL] (15 December 2021).
Cited by
Cited by 2 other publications
Hwang, Haerim & Hyunwoo Kim
2024.
Korean Syntactic Complexity Analyzer (KOSCA): An NLP application for the analysis of syntactic complexity in second language production.
Language Testing
Kopotev, Mikhail, Aleksandr Klimov & Olesya Kisselev
2023.
Exploring collocational complexity in L2 Russian: A corpus-driven contrastive analysis.
International Journal of Bilingualism
This list is based on CrossRef data as of 28 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.