Cover not available
Part of
Complexity, Accuracy and Fluency in Learner Corpus Research
Edited by Agnieszka Leńko-Szymańska and Sandra Götz
[Studies in Corpus Linguistics 104] 2022
► pp. 51–80

Syntactic complexity measures as linguistic correlates of proficiency level in learner Russian

| University of Texas
| University of Helsinki
| University of Helsinki
The study reports on the results of a corpus-based evaluation of automatically extracted syntactic complexity measures as indices of Russian as a foreign language (FL) and Russian as a heritage language (HL) writing development. A list of 12 syntactic complexity measures was tested on a set of longitudinal, classroom-based data. The analyses demonstrated that the syntactic complexity measures help delineate four proficiency levels (Intermediate Mid, Intermediate High, Advanced Low and Advanced Mid as established through the ACTFL proficiency guidelines), and that the changes in syntactic complexity indices across levels pattern slightly differently in the FL vs. HL learner groups. Our research confirms that the overall trends in interlanguage development in Russian align with the “complexification” trends found for other second languages.
Keywords: syntactic complexity, L2 writing, linguistic proficiency, learner Russian, learner corpus
Article outline
  • 1.Introduction
  • 2.Syntactic complexity
    • 2.1Syntactic complexity: Key concepts and considerations
    • 2.2Syntactic complexity indices as correlates of language development
    • 2.3Language proficiency as a proficiency scale rating
    • 2.4Syntactic complexity measures in learner Russian
  • 3.The method
    • 3.1Participants and data
    • 3.2Data annotation
  • 4.Analyses
  • 5.Results
    • 5.1Descriptive results
    • 5.2Correlations between syntactic complexity measures
    • 5.3Analysis of variance between proficiency levels and language backgrounds
    • 5.4Ranking syntactic measures using a machine learning classifier
  • 6.Discussion and conclusion
  • Notes
  • References
Published online: 1 December 2022
DOI logo
https://doi.org/10.1075/scl.104.03kis
References (68)
References
Adams, Rebecca, Alwi, Nik M., Aloesnita, Nik & Newton, Jonathan. 2015. Task complexity effects on the complexity and accuracy of writing via text chat. Journal of Second Language Writing 29: 64–81. DOI logoGoogle Scholar
Alexopoulou, Theodora, Michel, Marije, Murakami, Akira & Meurers, Detmar. 2017. Task effects on linguistic complexity and accuracy: A large-scale corpus analysis employing natural language processing techniques. Language Learning 67(S1): 180–208. DOI logoGoogle Scholar
Alsufieva, Anna A., Kisselev, Olesya V. & Freels, Sandra G. 2012. Results 2012: Using flagship data to develop a Russian learner corpus of academic writing. Russian Language Journal/Русский зяык 62: 79–105.Google Scholar
Bachman, Lyle F. 1988. Problems in examining the validity of the ACTFL oral proficiency interview. Studies in Second Language Acquisition 10(2): 149–164. DOI logoGoogle Scholar
Bailyn, John F. 2012. The Syntax of Russian. Cambridge: CUP. DOI logoGoogle Scholar
Ballier, Nicolas, Gaillat, Thomas, Simpkin, Andrew, Stearns, Bernardo, Bouyé, Manon & Zrrouk, Manel. 2019. A supervised model for the automatic assessment of language levels based on learner errors. In Transforming Learning with Meaningful Technologies, Maren Scheffel, Julien Broisin, Viktoria Pammer-Schindler, Andri Ioannou & Jan Schneider (eds), 308–320. Cham: Springer. DOI logoGoogle Scholar
Barkaoui, Khaled & Hadidi, Ali. 2020. Assessing Change in English Second Language Writing Performance. New York NY: Routledge. DOI logoGoogle Scholar
Barykina, Alevtina N., Burmistrova, Valentina P. & Dobrovol’skaia, Valeria V. 1978. Posobie po razvitiiu navykov pis’mennoi rechi. Moscow: Russkii iazyk.Google Scholar
Biber, Douglas, Gray, Bethany & Poonpon, Kornwipa. 2011. Should we use characteristics of conversation to measure grammatical complexity in L2 writing development? TESOL Quarterly 45(1): 5–35. DOI logoGoogle Scholar
Bulté, Bram & Housen, Alex. 2012. Defining and operationalising L2 complexity. In Dimensions of L2 Performance and Proficiency: Complexity, Accuracy and Fluency in SLA [Language Learning & Language Teaching 32], Alex Housen, Folkert Kuiken & Ineke Vedder (eds), 21–46. Amsterdam: John Benjamins. DOI logoGoogle Scholar
. 2014. Conceptualizing and measuring short-term changes in L2 writing complexity. Journal of Second Language Writing 26: 42–65. DOI logoGoogle Scholar
Callies, Marcus & Götz, Sandra. 2015. Learner corpora in language testing and assessment: Prospects and challenges. In Learner Corpora in Language Testing and Assessment [Studies in Corpus Linguistics 70], Marcus Callies & Sandra Götz (eds), 1–10. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Chen, Yu-Hua & Baker, Paul. 2014. Investigating criterial discourse features across second language development: Lexical bundles in rated learner essays, CEFR B1, B2 and C1. Applied Linguistics 37(6): 849–880. DOI logoGoogle Scholar
Crossley, Scott A. & McNamara, Danielle S. 2014. Does writing development equal writing quality? A computational investigation of syntactic complexity in L2 learners. Journal of Second Language Writing 26: 66–79. DOI logoGoogle Scholar
Dengub, Evgeny. 2012. Investigating Syntactic and Lexical Complexity, Accuracy, and Fluency in the Writing of Heritage Speakers of Russian. PhD dissertation, Bryn Mawr College.
de Marneffe, Marie-Catherine, Dozat, Timothy, Silveira, Natalia, Haverinen, Katri, Ginter, Filip, Nivre, Joakim & Manning, Christopher D. 2014. Universal Stanford dependencies: A cross-linguistic typology. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), 4585–4592. Reykjavik: European Language Resources Association (ELRA). <[URL]> (15 December 2021).
Diessel, Holger. 2004. The Acquisition of Complex Sentences [Cambridge Studies in Linguistics 105]. Cambridge: CUP. DOI logoGoogle Scholar
Friginal, Eric & Weigle, Sandra. 2014. Exploring multiple profiles of L2 writing using multi-dimensional analysis. Journal of Second Language Writing 26: 80–95. DOI logoGoogle Scholar
Freels, Sandra, Kisselev, Olesya & Alsufieva, Anna. 2016. Adding breadth to the undergraduate curriculum. Flagship approaches to interdisciplinary language learning. In Exploring the US Language Flagship Program: Professional Competence in a Second Language by Graduation [New Perspectives on Language and Education 50], Dianna Murphy & Karen Evans-Romaine (eds), 51–69. Bristol: Multilingual Matters. DOI logoGoogle Scholar
Guo, Liang, Crossley, Scott A. & McNamara, Danielle S. 2013. Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing 18(3): 218–238. DOI logoGoogle Scholar
Haspelmath, Martin. 2007. Coordination. In Language Typology and Syntactic Description, Timothy Shopen (ed.), 1–51. Cambridge: CUP. DOI logoGoogle Scholar
Hawkins, John A. & Buttery, Paula. 2010. Criterial features in learner corpora: Theory and illustrations. English Profile Journal 1: e5. DOI logoGoogle Scholar
Housen, Alex, De Clercq, Bastien, Kuiken, Folkert & Vedder, Ineke. 2019. Multiple approaches to complexity in second language research. Second Language Research 35(1): 3–21. DOI logoGoogle Scholar
Huang, Ting, Steinkrauss, Rasmus & Verspoor, Marjolijn. 2021. Variability as predictor in L2 writing proficiency. Journal of Second Language Writing 52: 100787. DOI logoGoogle Scholar
Karlsson, Frank. 2010. Syntactic recursion and iteration. In Recursion and Human Language, Harry van der Hulst (ed.), 43–67. Berlin: Mouton de Gruyter. DOI logoGoogle Scholar
Kisselev, Olesya & Alsufieva, Anna. 2017. The development of syntactic complexity in the writing of Russian language learners: A longitudinal corpus study. Russian Language Journal/Русский язык 67: 27–54.Google Scholar
Kisselev, Olesya & Comer, William. 2019. Interdepartmental collaboration and curriculum design: Creating a Russian environmental sustainability course for advanced study. In Foreign Language Teaching and the Environment: Theory, Curricula, Institutional Structures, Charlotte Melin (ed.), 180–196. New York NY: Modern Language Association.Google Scholar
Kisselev, Olesya, Dubinina, Irina & Polinsky, Maria. 2020. Form-focused instruction in the heritage language classroom: Toward research-informed heritage language pedagogy. Frontiers in Education 5: 53. DOI logoGoogle Scholar
Kisselev, Olesya, Klimov, Alexandr, & Kopotev, Mikhail. 2021. Syntactic complexity measures as indices of language proficiency in writing: Focus on heritage learners of Russian. Heritage Language Journal 18(3): 1–30. DOI logoGoogle Scholar
Khimik, Vasilij. V. 2003. Osnovy naučnoi reči: Učebnoe posobie dlia studentov nefilologičeskih vysših zavedenij. Moscow: Akademiia.Google Scholar
Kuiken, Folkert & Vedder, Ineke. 2019. Syntactic complexity across proficiency and languages: L2 and L1 writing in Dutch, Italian and Spanish. International Journal of Applied Linguistics 29(2): 192–210. DOI logoGoogle Scholar
Kyle, Kristopher & Crossley, Scott A. 2018. Measuring syntactic complexity in L2 writing using fine-grained clausal and phrasal indices. The Modern Language Journal 102(2): 333–349. DOI logoGoogle Scholar
Lan, Ge, Liu, Qiandi & Staples, Shelley. 2019. Grammatical complexity: ‘What Does It Mean’ and ‘So What’ for L2 writing classrooms? Journal of Second Language Writing 46: 100673. DOI logoGoogle Scholar
Lissón, Paula & Ballier, Nicolas. 2018. Investigating lexical progression through lexical diversity metrics in a corpus of French L3. Discours. Revue de linguistique, psycholinguistique et informatique. A Journal of Linguistics, Psycholinguistics and Computational Linguistics 23: 1–26. DOI logoGoogle Scholar
Lobanova, Natalja A. & Slesareva, Irma P. 1980. Uchebnik russkogo iazyka dlia inostrannykh studentov-filologov. Sistematiziruiushchii kurs [Russian Language Handbook for Foreign Students of Philology: A Systematizing Course]. Moscow: Russkii iazyk.Google Scholar
Long, Michael H., Gor, Kira & Jackson, Scott. 2012. Linguistic correlates of second language proficiency: Proof of concept with ILR 2–3 in Russian. Studies in Second Language Acquisition 34(1): 99–126. DOI logoGoogle Scholar
Lu, Xiaofei. 2011. A corpus-based evaluation of syntactic complexity measures as indices of college-level ESL writers’ language development. TESOL Quarterly 45(1): 35–62. DOI logoGoogle Scholar
Lu, Xiaofei & Ai, Haiyang. 2015. Syntactic complexity in college-level English writing: Differences among writers with diverse L1 backgrounds. Journal of Second Language Writing 29: 16–27. DOI logoGoogle Scholar
Mazgutova, Diane & Kormos, Judit. 2015. Syntactic and lexical development in an intensive English for academic purposes programme. Journal of Second Language Writing 29: 3–15. DOI logoGoogle Scholar
Menke, Mandy R. & Strawbridge, Tripp. 2019. The writing of Spanish majors: A longitudinal analysis of syntactic complexity. Journal of Second Language Writing 46: 100665. DOI logoGoogle Scholar
Michel, Marije, Murakami, Akira, Alexopoulou, Theodora & Meurers, Detmar. 2019. Effects of task type on morphosyntactic complexity across proficiency: Evidence from a large learner corpus of A1 to C2 writings. Instructed Second Language Acquisition 3(2): 124–152. DOI logoGoogle Scholar
Montrul, Silvina. 2016. The Acquisition of Heritage Languages. Cambridge: CUP. DOI logoGoogle Scholar
Mostafa, Tamanna & Crossley, Scott A. 2020. Verb argument construction complexity indices and L2 writing quality: Effects of writing tasks and prompts. Journal of Second Language Writing 49: 100730. DOI logoGoogle Scholar
Norris, John M. & Ortega, Lourdes. 2009. Towards an organic approach to investigating CAF in instructed SLA: The case of complexity. Applied Linguistics 30(4): 555–578. DOI logoGoogle Scholar
Ortega, Lourdes. 2003. Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college-level L2 writing. Applied Linguistics 24(4): 492–518. DOI logoGoogle Scholar
Osborne, Timothy. 2006. Syntax gapping vs. non-gapping coordination. Linguistische Berichte 207: 307–337.Google Scholar
Pallotti, Gabriele. 2009. CAF: Defining, refining and differentiating constructs. Applied Linguistics 30(4): 590–601. DOI logoGoogle Scholar
Petrov, Slav, Das, Dipanjan & McDonald, Ryan. 2011. A universal part-of-speech tagset. arXiv preprint arXiv:1104.2086.Google Scholar
Polat, Nihat, Mahalingappa, Laura & Mancilla, Rae L. 2020. Longitudinal growth trajectories of written syntactic complexity: The case of Turkish learners in an intensive English program. Applied Linguistics 41(5): 688–711. DOI logoGoogle Scholar
Polio, Charlene & Park, Ji-Hyun. 2016. Language development in second language writing. In Handbook of Second and Foreign Language Writing, Rosa M. Manchón & Paul Kei Matsuda (eds), 287–306. Berlin: De Gruyter Mouton. DOI logoGoogle Scholar
Prokhorova, Kira V. 1998. Naučnyj stil’: učebno-metodičeskoe posobie dlja studentov-žurnalistov. Sankt-Peterburg: Sankt-Peterburg State University.Google Scholar
Reich, Peter & Schütze, Carson. 1991. Syntactic embedding: What can people really do? Toronto Working Papers in Linguistics 11: 91–97.Google Scholar
Sag, Ivan A., Gazdar, Gerald, Wasow, Thomas & Weisler, Steven. 1985. Coordination and how to distinguish categories. Natural Language & Linguistic Theory 3(2): 117–171. DOI logoGoogle Scholar
Staples, Shelley, Adriana Picoral, Aleksey Novikov, and Bruna Sommer-Farias. 2019. Directions for Future Use of Using Existing Corpora in the Study of L2 Writing. In The Routledge Handbook of Second Language Acquisition and Writing, 356–369. Routledge.Google Scholar
Straka, Milan & Straková, Jana. 2017. Tokenizing, POS tagging, lemmatizing and parsing UD 2.0 with UDPipe. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, 88–99. Vancouver: Association for Computational Linguistics. <[URL]> (15 December 2022). DOI logo
Tack, Anaïs, François, Thomas, Roekhaut, Sophie & Fairon, Cédrick. 2017. Human and automated CEFR-based grading of short answers. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications. Joel Tetreault, Jill Burstein, Claudia Leacock & Helen Yannakoudakis (eds), 169–179. Copenhagen: Association of Computational Linguistics. <[URL]> (15 December 2021). DOI logo
Testelets, Yakov G. 2001. Vvedenie v Obščij Sintaksis [Introduction to General Syntax]. Mocsow: Rossijskij Gosudarstviennyj Gumanitarnyj Universitet (RGGU).Google Scholar
Treffers-Daller, Jeanine, Parslow, Patrick & Williams, Shirley. 2018. Back to basics: How measures of lexical diversity can help discriminate between CEFR levels. Applied Linguistics 39(3): 302–327. DOI logoGoogle Scholar
Tschirner, Erwin, Bärenfänger, Olaf & Wanner, Irmgard. 2012. Assessing Evidence of Validity of Assigning CEFR Ratings to the ACTFL Oral Proficiency Interview (OPI) and the Oral Proficiency Interview by Computer (OPIc). Leipzig: Universität Leipzig, Herder-Institut.Google Scholar
UDPipe 1 Models. 2021. <[URL]> (30 June 2021).
Verspoor, Marjolijn, Schmid, Monika S., Xu, Xiaoyan. 2012. A dynamic usage based perspective on L2 writing. Journal of Second Language Writing 21(3): 239–263. DOI logoGoogle Scholar
de Vries, Mark. 2005. Coordination and syntactic hierarchy. Studia Linguistica 59(1): 83–105. DOI logoGoogle Scholar
Vyatkina, Nina. 2012. The development of second language writing complexity in groups and individuals: A longitudinal learner corpus study. The Modern Language Journal 96(4): 576–598. DOI logoGoogle Scholar
. 2013. Specific syntactic complexity: Developmental profiling of individuals based on an annotated learner corpus. The Modern Language Journal 97(1): 11–30. DOI logoGoogle Scholar
Yang, Weiwei, Lu, Xiaofei & Weigle, Sara C. 2015. Different topics, different discourse: Relationships among writing topic, measures of syntactic complexity, and judgments of writing quality. Journal of Second Language Writing 28: 53–67. DOI logoGoogle Scholar
Yoon, Hyung-Jo & Polio, Charlene. 2017. The linguistic development of students of English as a second language in two written genres. TESOL Quarterly 51(2): 275–301. DOI logoGoogle Scholar
Wolfe-Quintero, Kate, Inagaki, Shunji & Kim, Hae-Young. 1998. Second Language Development in Writing: Measures of Fluency, Accuracy, & Complexity. Honolulu HI: Second Language Teaching & Curriculum Center, University of Hawai’i at Manoa. DOI logoGoogle Scholar
Zeman, Daniel & Resnik, Philip. 2008. Cross-language parser adaptation between related languages. In Proceedings of the IJCNLP-08 Workshop on NLP for Less Privileged Languages, 35–42. Hyderabad: Asian Federation of Natural Language Processing. <[URL]> (15 December 2021).
Cited by (2)

Cited by two other publications

Hwang, Haerim & Hyunwoo Kim
2024. Korean Syntactic Complexity Analyzer (KOSCA): An NLP application for the analysis of syntactic complexity in second language production . Language Testing 41:3 ► pp. 506 ff. DOI logo
Kopotev, Mikhail, Aleksandr Klimov & Olesya Kisselev
2023. Exploring collocational complexity in L2 Russian: A corpus-driven contrastive analysis. International Journal of Bilingualism DOI logo

This list is based on CrossRef data as of 22 september 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.