Article published In:
ITL - International Journal of Applied Linguistics
Vol. 174:1 (2023) ► pp.135–165

3K-LEx-MC

The creation and validation of a multiple-choice vocabulary placement test for Spanish language learners

ORCID logo | University of Toronto, Mississauga , Mississauga
ORCID logo | The Human Resources Research Organization (HumRRO)
ORCID logo | University of California, Davis , Davis

This study presents the development and validation of a 132-item Spanish-English bilingual multiple-choice vocabulary test based on the 3,000 most frequent lemmas that distinguishes between North American university students who satisfy the Foreign Language requirement and those who need to complete coursework. 819 students were assigned to one of the two 144-item forms of the preliminary test, which had 72 shared anchor items and other 72 form-specific items. Factor analysis was used to evaluate dimensionality and the Rasch model was used to select the items that best differentiated between these two student populations. This final form was administered to 213 students. Results showed high levels of unidimensionality, and the final form provided a Rasch reliability coefficient of 0.97.

Keywords: vocabulary test, Spanish, Rasch model, multiple-choice, lexical frequency
Article outline
  • High frequency vocabulary in Spanish
  • Spanish vocabulary tests
  • The Rasch model
  • Methods
    • Participants
    • Instruments
    • Procedure
  • Results
    • Calibration and item bank development
    • Unidimensionality
    • Item level fit statistics
    • Construction of final form
    • Participants
    • Explanatory IRT
  • Conclusions
  • Notes
  • References
Published online: 25 October 2022
DOI logo
https://doi.org/10.1075/itl.22008.rob
References
Beavers, A. S., Lounsbury, J. W., Richards, J. K., Huck, S. W., Skolits, G. J., & Esquivel, S. L.
(2013) Practical considerations for using exploratory factor analysis in educational research. Practical Assessment, Research, and Evaluation, 18 (6), 1–13. DOI logoGoogle Scholar
Beglar, D.
(2010) A Rasch-based validation of the vocabulary size test. Language Testing, 27 (1), 101–118. DOI logoGoogle Scholar
Berendt, E., Kurosaki, E., Maeda, A., Matsui, K., & Ochi, N.
(2006) English loan words of Japanese elementary school children and their mental lexicon. Asian Englishes, 8 (2), 26–45. DOI logoGoogle Scholar
Biemiller, A.
(2003) Vocabulary: Needed if more children are to read well. Reading Psychology, 24 (3–4), 323–335. DOI logoGoogle Scholar
Bond, T. G., & Fox, C. M.
(2015) Applying the Rasch model: Fundamental measurement in the human sciences. Routledge. DOI logoGoogle Scholar
Bürkner, P. C.
(2017) brms: An R package for bayesian multilevel models using Stan. Journal of Statistical Software, 80 (1), 1–28. DOI logoGoogle Scholar
Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddel, A.
(2017) Stan: A probabilistic programming language. Journal of Statistical Software, 76 (1), 1–32. DOI logoGoogle Scholar
Casado, M. A., & Dereshiwsky, M. I.
(2001) Foreign language anxiety of university students. College Student Journal, 35 (4), 539–552.Google Scholar
Chalmers, R. P.
(2012) mirt: A Multidimensional item response theory package for the R environment. Journal of Statistical Software, 48 (6), 1–29. DOI logoGoogle Scholar
Davies, M.
(2005) Vocabulary range and text coverage: Insights from the forthcoming Routledge frequency dictionary of Spanish. In D. Eddington (Ed.), Selected proceedings of the 7th Hispanic Linguistics Symposium (pp. 106–115). Somerville, MA: Cascadilla Proceedings Project.Google Scholar
Davies, M., & Hayward Davies, K.
(2017) A frequency dictionary of Spanish: Core vocabulary for learners (2nd ed.). Routledge. DOI logoGoogle Scholar
de Ayala, R. J.
(2009) The theory and practice of item response theory. The Guilford Press.Google Scholar
De Boeck, P., & Wilson, M.
(2004) Explanatory item response models: A generalized linear and nonlinear approach. Springer. DOI logoGoogle Scholar
Desjardins, C. D., & Bulut, O.
(2018) Handbook of educational measurement and psychometrics using R. CRC Press. DOI logoGoogle Scholar
Dorans, N. J., Moses, T. P., & Eignor, D. R.
(2010) Principles and practices of test score equating. ETS Research Report Series 2010(2), i–41. DOI logoGoogle Scholar
Frantzen, D., & Magnan, S. S.
(2005) Anxiety and the true beginner – false beginner dynamic in beginning French and Spanish classes. Foreign Language Annals, 38 (2), 171–186. DOI logoGoogle Scholar
Floyd, F. J., & Widaman, K. F.
(1995) Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7 (3), 286–299. DOI logoGoogle Scholar
Gairns, R., & Redman, S.
(1986) Working with words. A guide to teaching and learning vocabulary. Cambridge University Press.Google Scholar
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B.
(2013) Bayesian data analysis (3rd ed.). CRC Press. DOI logoGoogle Scholar
Gyllstad, H., Vilkaitė, L., & Schmitt, N.
(2015) Assessing vocabulary size through multiple-choice formats: Issues with guessing and sampling rates. International Journal of Applied Linguistics, 166 (2), 278–306. DOI logoGoogle Scholar
Hambleton, R. K., Swaminathan, H., & Rogers, H. J.
(1991) Fundamentals of item response theory. SAGE Publications, Inc.Google Scholar
Horn, J. L.
(1965) A rationale and test for the number of factors in factor analysis. Psychometrika, 30 (2), 179–185. DOI logoGoogle Scholar
Horwitz, E.
(1990) Attending to the affective domain in the foreign language classroom. In S. Magnan (Ed.), Shifting the instructional focus to the learner (pp. 15–33). Northeast Conference on the Teaching of Foreign Languages.
Hu, M., & Nation, I. S. P.
(2000) Vocabulary density and reading comprehension. Reading in a Foreign Language, 23 (1), 4031–430.Google Scholar
Institute for Test Research and Test Development
(2013) Assessing evidence of validity of the ACTFL reading proficiency test (RPT).Google Scholar
Izura, C., Cuetos, F., & Brysbaert, M.
(2014) Lextale-Esp: A test to rapidly and efficiently assess the Spanish vocabulary size. Psicológica, 35 (1), 49–66.Google Scholar
Jones, R., & Tschirner, E.
(2006) A frequency dictionary of German: Core vocabulary for learners. Routledge.Google Scholar
Kaplan, R. M., & Saccuzzo, D. P.
(2009) Psychological testing principles, applications, and issues (7th ed.). Wadsworth.Google Scholar
Kruschke, J. K.
(2015) Doing Bayesian data analysis: A tutorial with R, JAGS, and Stan (2nd ed.). Elsevier.Google Scholar
Laufer, B., & Aviad-Levitzky, A.
(2017) What type of vocabulary knowledge predicts reading comprehension: Word meaning recall or word meaning recognition? The Modern Language Journal, 101 (4), 729–741. [URL]Google Scholar
Laufer, B., & McLean, S.
(2016) Loanwords and vocabulary size test scores: A case of different estimates for different L1 learners. Language Assessment Quarterly, 13 (3), 202–217. DOI logoGoogle Scholar
Liu, N., & Nation, I. S. P.
(1985) Factors affecting guessing vocabulary in context. RELC Journal, 16 (1), 33–42. DOI logoGoogle Scholar
Looney, D., & Lusin, N.
(2019) Enrollments in languages other than English in United States institutions of higher education, summer 2016 and fall 2016: Final report. Modern Language Association of America. [URL]
McElreath, R.
(2020) Statistical rethinking (2nd ed.). CRC Press. DOI logoGoogle Scholar
McLean, S., Kramer, B., & Beglar, D.
(2015a) The creation and validation of a listening vocabulary levels test. Language Teaching Research, 19 (6), 741–760. DOI logoGoogle Scholar
McLean, S., Kramer, B., & Stewart, J.
(2015b) An empirical examination of the effect of guessing on vocabulary size test scores. Vocabulary Learning and Instruction, 4 (1), 26–35. DOI logoGoogle Scholar
McLean, S., Stewart, J., & Batty, A. O.
(2020) Predicting L2 reading proficiency with modalities of vocabulary knowledge: A bootstrapping approach. Language Testing, 37 (3), 389–411. DOI logoGoogle Scholar
McNamara, T., & Knoch, U.
(2012) The Rasch wars: The emergence of Rasch measurement in language testing. Language Testing, 29 (4), 555–576. DOI logoGoogle Scholar
Meara, P.
(2010) EFL vocabulary test (2nd ed.). Centre for Applied Language Studies.Google Scholar
Meara, P., & Milton, J.
(2003) The Swansea levels test. Express.Google Scholar
Milton, J.
(2009) Measuring second language vocabulary acquisition. Multilingual Matters. DOI logoGoogle Scholar
Nation, I. S. P.
(2006) How large a vocabulary is needed for reading and listening? Canadian Modern Language Review, 63 (1), 591–82. DOI logoGoogle Scholar
Nation, I. S. P., & Beglar, D.
(2007) A vocabulary size test. The Language Teacher, 31 (7), 96–13.Google Scholar
Nation, I. S. P., & Webb, S. A.
(2011) Researching and analyzing vocabulary. Cengage.Google Scholar
Nguyen, L. T. C., & Nation, I. S. P.
(2011) A bilingual vocabulary size test of English for Vietnamese learners. RELC Journal, 42 (1), 86–99. DOI logoGoogle Scholar
O’Connor, B. P.
(2000) SPSS and BAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instruments, & Computers, 32 (3), 396–402. DOI logoGoogle Scholar
Pellicer-Sánchez, A., & Schmitt, N.
(2012) Scoring Yes–No vocabulary tests: Reaction time vs. nonword approaches. Language Testing, 29 (4), 489–509. DOI logoGoogle Scholar
R Core Team
(2019) R: A language and environment for statistical computing. R Foundation for Statistical Computing. [URL]
Revelle, W.
(2020) Psych: Procedures for psychological, psychometric, and personality research. Northwestern University. [URL]
Robles-García, P.
(2020) 3K-LEx: Desarrollo y validación de una prueba de amplitud Léxica en español. Journal of Spanish Language Teaching, 7 (1), 64–76. DOI logoGoogle Scholar
(2022) Receptive vocabulary knowledge in L2 learners of Spanish: The role of high-frequency words. Foreign Language Annals, 1–22. DOI logoGoogle Scholar
Schmith, R.
(2002) Qualtrics [survey software]. Utah, U.S. available at [URL]
Schmitt, N., Cobb, T., Horst, M., & Schmitt, D.
(2017) How much vocabulary is needed to use English? Replication of van Zeeland & Schmitt (2012), Nation (2006) and Cobb (2007). Language Teaching, 50 (2), 212–226. DOI logoGoogle Scholar
Schmitt, N., Jiang, X., & Grabe, W.
(2011) The percentage of words known in a text and reading comprehension. Modern Language Journal, 95 (1), 26–43. DOI logoGoogle Scholar
Schmitt, N. & Schmitt, D.
(2014) A reassessment of frequency and vocabulary size in L2 vocabulary teaching. Language Teaching, 47 (4), 484–503. DOI logoGoogle Scholar
Schmitt, N., Schmitt, D., & Clapham, C.
(2001) Developing and exploring the behaviour of two new versions of the vocabulary levels test. Language Testing, 18 (1), 55–88. DOI logoGoogle Scholar
Stoeckel, T.
(2018) High-frequency and academic English vocabulary growth among first-year students at UNP. Journal of International Studies and Regional Development, 9 1, 15–30.Google Scholar
Stoeckel, T., McLean, S., & Nation, I. S. P.
(2021) Limitations of size and levels tests of written receptive vocabulary knowledge. Studies in Second Language Acquisition, 43 (1), 181–203. DOI logoGoogle Scholar
Van Zeeland, H., & Schmitt, N.
(2012) Lexical coverage in L1 and L2 listening comprehension: The same or different from reading comprehension? Applied Linguistics, 34 (4), 457–479. DOI logoGoogle Scholar
Webb, S. A., & Chang, A. C.-S.
(2012) Second language vocabulary growth. RELC Journal, 43 (1), 113–126. DOI logoGoogle Scholar
Webb, S., Sasao, Y., & Ballance, O.
(2017) The updated vocabulary levels test: Developing and validating two new forms of the VLT. International Journal of Applied Linguistics, 168 (1), 34–70. DOI logoGoogle Scholar
Wilson, M., De Boeck, P., & Carstensen, C. H.
(2008) Explanatory item response models: A brief introduction. In M. Wilson & P. De Boeck (Eds.), Explanatory Item Response Models (2nd ed.). Springer.Google Scholar
Zhang, S., & Zhang, X.
(2020) The relationship between vocabulary knowledge and L2 reading/listening comprehension: A meta-analysis. Language Teaching Research, 26 (4), 696–725. DOI logoGoogle Scholar
Zhang, X.
(2013) The I don’t know option in the vocabulary size test. TESOL Quarterly, 47 (4), 790–811. DOI logoGoogle Scholar
Zipf, G.
(1935) The psychobiology of language: An introduction to dynamic philology. MIT Press.Google Scholar