Cover not available
Part of
Critical Reflections on Data in Second Language Acquisition
Edited by Aarnes Gudmestad and Amanda Edmonds
[Language Learning & Language Teaching 51] 2018
► pp. 63–88

Chapter 3
Data analysis and sampling

Methodological issues concerning proficiency in SLA research

| University of Nevada,

This chapter addresses the construct of second-language (L2) proficiency as it pertains to L2 data analysis. In L2 studies, a common practice is to group participants into proficiency categories (i.e., groups such as ‘intermediate’ or ‘advanced’; a practice known as dichotomization). Nevertheless, I argue that, for theoretical and empirical reasons, proficiency should be analyzed as a true continuous variable. Theoretically, we conceptualize adult L2 acquisition as a continuous process driven by basic learning mechanisms that may be constrained by underlying principles. Empirically, I present evidence illustrating how creating categorical groups by carving up a continuous dependent measure is statistically inappropriate. Finally, I address the importance of sampling practices and why it is preferable to include participants from a broad proficiency spectrum.

Keywords: L2 proficiency, dichotomization, data analysis, quantitative research methods
Article outline
  • Introduction
  • Background
    • The construct of L2 proficiency
    • The measurement of L2 proficiency in empirical studies
  • Issues in data analysis: Dichotomizing proficiency
  • Data sampling and proficiency
  • Conclusion
  • Acknowledgements
  • Notes
  • References
Published online: 10 September 2018
DOI logo
https://doi.org/10.1075/lllt.51.04lea
References
Abutalebi, J., Cappa, S. F., & Perani, D.
(2001) The bilingual brain as revealed by functional neuroimaging. Bilingualism: Language and Cognition, 4(2), 179–190.DOI logoGoogle Scholar
Alderson, J. C.
(2005) Diagnosing foreign language proficiency: The interface between learning and assessment. London: Continuum.Google Scholar
AlFallay, I.
(2004) The role of some selected psychological and personality traits of the rater in the accuracy of self-and peer-assessment. System, 32(3), 407–425.DOI logoGoogle Scholar
Altman, D. G., & Royston, P.
(2006) The cost of dichotomising continuous variables. BMJ, 332(7549), 1080.DOI logoGoogle Scholar
Alvesson, M., & Sandberg, J.
(2011) Generating research questions through problematization. Academy of Management Review, 36(2), 247–271.DOI logoGoogle Scholar
American Council on the Teaching of Foreign Languages
(2012) ACTFL proficiency guidelines. Retrieved from: [URL]
Baayen, R. H.
(2008) Analyzing linguistic data: A practical introduction to statistics using R. Cambridge: Cambridge University Press.DOI logoGoogle Scholar
Bachman, L. F.
(1990) Constructing measures and measuring constructs. In B. Harley, P. Allen, J. Cummins, & M. Swain (Eds.), The development of second language proficiency (pp.26–49). Cambridge: Cambridge University Press.DOI logoGoogle Scholar
Baddeley, A.
(2007) Working memory, thought, and action. Oxford: Oxford University Press.DOI logoGoogle Scholar
Benati, A.
(Ed.) (2011) Issues in second language proficiency. London: Continuum.Google Scholar
Bijvoet, E., & Fraurud, K.
(2012) Studying high-level (L1–L2) development and use among young people in multilingual Stockholm: The role of perceptions of ambient sociolinguistic variation. Studies in Second Language Acquisition 34, 291–319.DOI logoGoogle Scholar
Birdsong, D.
(2014) Dominance and age in bilingualism. Applied Linguistics, 35(4), 374–392.DOI logoGoogle Scholar
Blanche, P., & Merino, B. J.
(1989) Self-assessment of foreign-language skills: Implications for teachers and researchers. Language Learning, 39(3), 313–338.DOI logoGoogle Scholar
Bowden, H. W.
(2016) Assessing second-language oral proficiency for research: The Spanish elicited imitation task. Studies in Second Language Acquisition, 38(4), 647–675.DOI logoGoogle Scholar
Canale, M., & Swain, M.
(1980) Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics 1, 1–47.DOI logoGoogle Scholar
Cattell, R. B.
(1960) The multiple abstract variance analysis equations and solutions: for nature-nurture research on continuous variables. Psychological Review, 67(6), 353–372.DOI logoGoogle Scholar
Cohen, J.
(1983) The cost of dichotomization. Applied Psychological Measurement, 7, 247–253.DOI logoGoogle Scholar
Cook, V.
(1999) Going beyond the native speaker in language teaching. TESOL Quarterly, 33(2), 185–209.DOI logoGoogle Scholar
Council of Europe
(2001) A Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge: Cambridge University Press.Google Scholar
Cummins, J.
(1980) Psychological assessment of immigrant children: Logic or intuition? Journal of Multilingual and Multicultural Development 1, 97–111.DOI logoGoogle Scholar
(1984) Bilingualism and special education: Issues in assessment and pedagogy. Clevedon: Multilingual Matters.Google Scholar
(2008) BICS and CALP: Empirical and theoretical status of the distinction. In B. Street & N. H. Hornberger (Eds.), Encyclopedia of language and education, Volume 2: Literacy (2nd ed., pp.–71–83). New York, NY: Springer Science.DOI logoGoogle Scholar
Cunnings, I.
(2012) An overview of mixed-effects statistical models for second language researchers. Second Language Research, 28(3), 369–382.DOI logoGoogle Scholar
Davis, J. M.
(2015) Sampling and what it means. In J. D. Brown & C. Coombe (Eds.), The Cambridge guide to research in language teaching and learning (pp.198–205). Cambridge: Cambridge University Press.Google Scholar
de Bot, K., Lowie, W., & Verspoor, M.
(2007) A dynamic systems theory approach to second language acquisition. Bilingualism: Language and Cognition, 10(1), 7–21.DOI logoGoogle Scholar
DeCat, C., & Serratrice, L.
under review). Predicting language proficiency in bilingual children. Retrieved on June 20, 2017 from [URL]
DeCoster, J., Iselin, A. M. R., & Gallucci, M.
(2009) A conceptual and empirical examination of justifications for dichotomization. Psychological Methods, 14(4), 349.DOI logoGoogle Scholar
DeKeyser, R. M.
(2005) What makes learning second-language grammar difficult? A review of issues. Language Learning, 55(1), 1–25.DOI logoGoogle Scholar
DeVellis, R. F.
(2003) Scale development: Theory and applications. Thousand Oaks, CA: Sage.Google Scholar
Dussias, P. E., Kroff, J. R. V., Tamargo, R. E. G., & Gerfen, C.
(2013) When gender and looking go hand in hand. Studies in Second Language Acquisition, 35(2), 353–387.DOI logoGoogle Scholar
Ellis, N. C.
(2002) Frequency effects in language acquisition: A review with implications for theories of implicit and explicit language acquisition. Studies in Second Language Acquisition, 24, 143–188.Google Scholar
Feldt, L.
(1961) The use of extreme groups to test for the presence of relationship. Psychometrika, 26(3), 307–316.DOI logoGoogle Scholar
Flege, J. E., & Liu, S.
(2001) The effect of experience on adult’s acquisition of a second language. Studies in Second Language Acquisition, 23(4), 527–552.Google Scholar
Flege, J. E., Yeni-Komshian, G. H., & Liu, S.
(1999) Age constraints on second-language acquisition. Journal of Memory and Language, 41(1), 78–104.DOI logoGoogle Scholar
Gabriel, C., & Kireva, E.
(2014) Prosodic transfer in learner and contact varieties: Speech rhythm and intonation of Buenos Aires Spanish and L2 Castilian Spanish produced by Italian native speakers. Studies in Second Language Acquisition, 36(2), 257–281.DOI logoGoogle Scholar
Gries, S. T.
(2013) Statistics for linguistics with R: A practical introduction. Berlin: Walter de Gruyter.DOI logoGoogle Scholar
Grosjean, F.
(1998) Studying bilinguals: Methodological and conceptual issues. Bilingualism: Language and Cognition, 1(2), 131–149.DOI logoGoogle Scholar
Hakuta, K.
(1976) A case study of a Japanese child learning English. Language Learning, 26, 321–351.DOI logoGoogle Scholar
Hoover, M. L., & Dwivedi, V. D.
(1998) Syntactic processing by skilled bilinguals. Language Learning, 48(1), 1–29.DOI logoGoogle Scholar
Hopp, H.
(2006) Syntactic features and reanalysis in near-native processing. Second Language Research, 22(3), 369–397.DOI logoGoogle Scholar
(2013) Grammatical gender in adult L2 acquisition: Relations between lexical and syntactic variability. Second Language Research, 29(1), 33–56.DOI logoGoogle Scholar
Hughes, A.
(2003) Testing for language teachers (2nd ed.). Cambridge: Cambridge University Press.Google Scholar
Hulstijn, J. H.
(2011) Language proficiency in native and nonnative speakers: An agenda for research and suggestions for second-language assessment. Language Assessment Quarterly, 8(3), 229–249.DOI logoGoogle Scholar
(2012) The construct of language proficiency in the study of bilingualism from a cognitive perspective. Bilingualism: Language and Cognition, 15(2), 422–433.DOI logoGoogle Scholar
Jackson, C.
(2008) Proficiency level and the interaction of lexical and morphosyntactic information during L2 sentence processing. Language Learning, 58(4), 875–909.DOI logoGoogle Scholar
Jegerski, J.
(2016) Number attraction effects in near-native Spanish sentence comprehension. Studies in Second Language Acquisition, 38(1), 5–33.DOI logoGoogle Scholar
Jiang, N.
(2004) Morphological insensitivity in second language processing. Applied Psycholinguistics, 25(4), 603–634.DOI logoGoogle Scholar
Just, M. A., Carpenter, P. A., & Woolley, J. D.
(1982) Paradigms and processes in reading comprehension. Journal of experimental psychology: General, 111(2), 228–238.DOI logoGoogle Scholar
Kramsch, C., & Whiteside, A.
(2007) Three fundamental concepts in second language acquisition and their relevance in multilingual contexts. The Modern Language Journal, 91, 907–922.DOI logoGoogle Scholar
Kunce, J. T., Cook, D. W., & Miller, D. E.
(1975) Random variables and correlational overkill. Educational and Psychological Measurement, 35(3), 529–534.DOI logoGoogle Scholar
Larsen-Freeman, D.
(2009) Adjusting expectations: The study of complexity, accuracy, and fluency in second language acquisition. Applied Linguistics, 30(4), 579–589.DOI logoGoogle Scholar
Larson-Hall, J.
(2015) A guide to doing statistics in second language research using SPSS and R. New York, NY: Routledge.Google Scholar
Larson-Hall, J., & Plonsky, L.
(2015) Reporting and interpreting quantitative research findings: What gets reported and recommendations for the field. Language Learning, 65(1), 127–159.DOI logoGoogle Scholar
Leal Méndez, T., Farmer, T. A., & Slabakova, R.
(2014) A same-system view of L2 processing: Evidence from long-distance syntactic dependencies in L2 Spanish. In W. Orman & M. J. Valleau (Eds.), Proceedings from the annual 38th Boston University Conference of Language Development (Vol. 2, pp.266–278). Somerville, MA: Cascadilla Press.Google Scholar
Leal, T., Slabakova, R., & Farmer, T.
(2017) The fine-tuning of linguistic expectations over the course of L2 learning. Studies in Second Language Acquisition. DOI logo.Google Scholar
Leal, T., & Slabakova, R.
(in press). The relationship between L2 instruction, exposure, and the L2 acquisition of a syntax-discourse property in L2 Spanish. Submitted to a special issue of Language Teaching Research. DOI logo
Levshina, N.
(2015) How to do linguistics with R: Data exploration and statistical analysis. Amsterdam: John Benjamins.DOI logoGoogle Scholar
Lim, J. H., & Christianson, K.
(2013) Second language sentence processing in reading for comprehension and translation. Bilingualism: Language and Cognition, 16(3), 518–537.DOI logoGoogle Scholar
Lindeman, R. H., Merenda, P. F., & Gold, R. Z.
(1980) Introduction to bivariate and multivariate analysis. London: Foresman.Google Scholar
Linck, J. A., & Cunnings, I.
(2015) The utility and application of mixed-effects models in second language research. Language Learning, 65(1), 185–207.DOI logoGoogle Scholar
Luk, G., & Bialystok, E.
(2013) Bilingualism is not a categorical variable: Interaction between language proficiency and usage. Journal of Cognitive Psychology, 25(5), 605–621.DOI logoGoogle Scholar
MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D.
(2002) On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19–40.DOI logoGoogle Scholar
MacIntyre, P. D., Noels, K. A., & Clément, R.
(1997) Biases in self-ratings of second language proficiency: The role of language anxiety. Language Learning, 47(2), 265–287.DOI logoGoogle Scholar
MacWhinney, B.
(2004) A multiple process solution to the logical problem of language acquisition. Journal of Child Language, 31(4), 883–914.DOI logoGoogle Scholar
Marian, V., Blumenfeld, H. K., & Kaushanskaya, M.
(2007) The Language Experience and Proficiency Questionnaire (LEAP-Q): Assessing language profiles in bilinguals and multilinguals. Journal of Speech, Language, and Hearing Research, 50(4), 940–967.DOI logoGoogle Scholar
McNamara, T. F.
(1996) Measuring second language performance. London: Longman.Google Scholar
Montrul, S. A.
(2008) Incomplete acquisition in bilingualism: Re-examining the age factor. Amsterdam: John Benjamins.DOI logoGoogle Scholar
Mougeon, F., & Rehner, K.
(2009) From grade school to university: The variable use of on/nous by university FSL students. Canadian Modern Language Review, 66, 269–298.DOI logoGoogle Scholar
Moyer, A.
(1999) Ultimate attainment in L2 phonology. Studies in Second Language Acquisition, 21(1), 81–108.DOI logoGoogle Scholar
Norris, J. M.
(2015) Statistical significance testing in second language research: Basic problems and suggestions for reform. Language Learning, 65(S1), 97–126.DOI logoGoogle Scholar
Norris, J. M., & Ortega, L.
(2013) Assessing learner knowledge. In S. Gass & A. Mackey (Eds.), The Routledge handbook of second language acquisition (pp.573–589). New York: Routledge.Google Scholar
Oller, J. W.
(1978) The language factor in the evaluation of bilingual education. In J. E. Alatis (Ed.), Georgetown University round table on languages and linguistics (pp.410–422). Washington, DC: Georgetown University Press.Google Scholar
Perry, F.
(2011) Research in applied linguistics: Becoming a discerning consumer (2nd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
Peters, C. C., & Van Voorhis, W. R.
(1940) Statistical procedures and their mathematical bases. New York, NY: McGraw-Hill.DOI logoGoogle Scholar
Pliatsikas, C., & Marinis, T.
(2013) Processing empty categories in a second language: When naturalistic exposure fills the (intermediate) gap. Bilingualism: Language and Cognition, 16(1), 167–182.DOI logoGoogle Scholar
Plonsky, L.
(2013) Study quality in SLA. Studies in Second Language Acquisition, 35, 655–687.DOI logoGoogle Scholar
(2015) Statistical power, p-values, descriptive statistics, and effect sizes: A “back-to-basics” approach to advancing quantitative methods in L2 research. In L. Plonsky (Ed.), Advancing quantitative methods in second language research (pp.23–45). New York, NY: Routledge.Google Scholar
Rampton, B.
(1990) Displacing the ‘native speaker’: Expertise, affiliation and inheritance. ELT Journal, 44(2), 97–101.DOI logoGoogle Scholar
Reichle, R., & Birdsong, D.
(2014) Processing focus structure in L1 and L2 French. Studies in Second Language Acquisition, 36(3), 535–564.DOI logoGoogle Scholar
Rossi, S., Gugler M. F., Friederici, A. D., & Hahne, A.
(2006) The impact of proficiency on syntactic second-language processing of German and Italian: Evidence from event-related potentials. Journal of Cognitive Neuroscience 18, 2030–2048.DOI logoGoogle Scholar
Rothman, J.
(2008) Why all counter-evidence to the Critical Period Hypothesis in second language acquisition is not equal or problematic. Language and Linguistics Compass, 2(6), 1063–1088.DOI logoGoogle Scholar
Scarcella, R.
(2003) Academic English: A conceptual framework. Los Angeles, CA: University of California Language Minority Research Institute.Google Scholar
Shentu, Y., & Xie, M.
(2010) A note on dichotomization of continuous response variable in the presence of contamination and model misspecification. Statistics in Medicine, 29(21), 2200–2214.DOI logoGoogle Scholar
Skehan, P.
(2009) Modelling second language performance: Integrating complexity, accuracy, fluency, and lexis. Applied Linguistics, 30(4), 510–532.DOI logoGoogle Scholar
Skutnabb-Kangas, T., & P. Toukomaa
(1976) Teaching migrant children and learning the language of the host country in the context of the socio-cultural situation of the migrant family. Helsinki: The Finnish National UNESCO.Google Scholar
Slabakova, R.
(2015) The effect of construction frequency and native transfer on second language knowledge of the syntax-discourse interface. Applied Psycholinguistics, 36(3), 671–699.DOI logoGoogle Scholar
Steinhauer, K., White, E. J., & Drury, J. E.
(2009) Temporal dynamics of late second language acquisition: Evidence from event-related brain potentials. Second Language Research, 25(1), 13–41.DOI logoGoogle Scholar
Tanner, D., McLaughlin, J., Herschensohn, J., & Osterhout, L.
(2013) Individual differences reveal stages of L2 grammatical acquisition: ERP evidence. Bilingualism: Language and Cognition, 16(2), 367–382.DOI logoGoogle Scholar
Thomas, M.
(1994) Assessment of L2 proficiency in second language acquisition research. Language Learning, 44, 307–336.DOI logoGoogle Scholar
(2006) Research synthesis and historiography: The case of assessment in second language proficiency. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching, (pp.279–298). Amsterdam: John Benjamins.DOI logoGoogle Scholar
Tokowicz, N., & MacWhinney, B.
(2005) Implicit and explicit measures of sensitivity to violations in second language grammar: An event-related potential investigation. Studies in Second Language Acquisition, 27(2), 173–204.DOI logoGoogle Scholar
Treffers-Daller, J., & Silva-Corvalán, C.
(Eds.) (2015) Language dominance in bilinguals: Issues of measurement and operationalization. Cambridge: Cambridge University Press.Google Scholar
Tremblay, A.
(2011) Proficiency assessment standards in second language acquisition research. Studies in Second Language Acquisition, 33(3), 339–372.DOI logoGoogle Scholar
Valdés, G.
(2004) Between support and marginalisation: The development of academic language in linguistic minority children. International Journal of Bilingual Education and Bilingualism, 7(2–3), 102–132.DOI logoGoogle Scholar
Velicer, W. F., & Fava, J. L.
(1998) Effects of variable and subject sampling on factor pattern recovery. Psychological Methods, 3(2), 231–251.DOI logoGoogle Scholar
Verhoeven, L., & de Jong, J. H.
(Eds.) (1992) The construct of language proficiency: Applications of psychological models to language assessment. Amsterdam: John Benjamins.DOI logoGoogle Scholar
Whelan, R.
(2008) Effective analysis of reaction time data. The Psychological Record, 58(3), 475–482.DOI logoGoogle Scholar
White, L.
(2000) Second language acquisition: From initial state to final state. In J. Archibald (Ed.), Second language acquisition and linguistic theory (pp.130–155). Oxford: Blackwell.Google Scholar
(2003) Second language acquisition and universal grammar. Cambridge: Cambridge University Press.DOI logoGoogle Scholar
Wolfe-Quintero, K., Inagaki, S., & Kim, H. Y.
(1998) Second language development in writing: Measures of fluency, accuracy, & complexity. National Foreign Language Resource Center.Google Scholar
Wu, S. L., & Ortega, L.
(2013) Measuring global oral proficiency in SLA research: A new elicited imitation test of L2 Chinese. Foreign Language Annals, 46(4), 680–704.DOI logoGoogle Scholar
Zhang, R.
(2015) Measuring university-level L2 learners’ implicit and explicit linguistic knowledge. Studies in Second Language Acquisition, 37(3), 457–486.DOI logoGoogle Scholar
Cited by

Cited by 12 other publications

Alzahrani, Alaa
2023. LexArabic: A receptive vocabulary size test to estimate Arabic proficiency. Behavior Research Methods DOI logo
Awwad, Anas & Parvaneh Tavakoli
2022. Task complexity, language proficiency and working memory: Interaction effects on second language speech performance. International Review of Applied Linguistics in Language Teaching 60:2 ► pp. 169 ff. DOI logo
Cho, Jacee
2022. Online processing and offline judgments of L2-English articles. Linguistic Approaches to Bilingualism 12:3 ► pp. 280 ff. DOI logo
Cho, Jacee
2024. Scalar implicatures in adult L2 learners: A self-paced reading study. Second Language Research 40:2 ► pp. 327 ff. DOI logo
Fang, Shaohua & Zhiyi Wu
2022. Syntactic prediction in L2 learners: evidence from English disjunction processing. International Review of Applied Linguistics in Language Teaching 0:0 DOI logo
Gudmestad, Aarnes, Amanda Edmonds & Thomas Metzger
2019. Using Variationism and Learner Corpus Research to Investigate Grammatical Gender Marking in Additional Language Spanish. Language Learning 69:4 ► pp. 911 ff. DOI logo
Leal, Tania & Bradley Hoot
2022. L2 representation and processing of Spanish focus. Language Acquisition 29:4 ► pp. 410 ff. DOI logo
Menke, Mandy R. & Paul A. Malovrh
2021. Chapter 2. The (limited) contributions of proficiency assessments in defining advancedness. In Advancedness in Second Language Spanish [Issues in Hispanic and Lusophone Linguistics, 31], ► pp. 18 ff. DOI logo
Nadova, Zuzana & María del Pilar García Mayo
2023. Telicity judgments in L2 English by L1 Slovak speakers. Linguistic Approaches to Bilingualism DOI logo
Perez-Cortes, Silvia
2023. Obviating the mood, but mostly under control: Spanish heritage speakers’ acquisition of the binding constraints of desiderative complements. Language Acquisition ► pp. 1 ff. DOI logo
Sequeros-Valle, Jose, Bradley Hoot & Jennifer Cabrelli
2020. Clitic-doubled left dislocation in L2 Spanish: The effect of processing load at the syntax-discourse interface. Language Acquisition 27:3 ► pp. 306 ff. DOI logo
Slabakova, Roumyana, Tania Leal, Amber Dudley & Micah Stack
2020. Generative Second Language Acquisition, DOI logo

This list is based on CrossRef data as of 22 april 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.