Article published In:
ITL - International Journal of Applied LinguisticsVol. 166:2 (2015) ► pp.278–306
Vol. 166:2 (2015) ► pp.278–306
Assessing vocabulary size through multiple-choice formats
Issues with guessing and sampling rates
In most tests of vocabulary size, knowledge is assessed through multiple-choice formats. Despite advantages such as ease of scoring, multiple-choice tests (MCT) are accompanied with problems. One of the more central issues has to do with guessing and the presence of other construct-irrelevant strategies that can lead to overestimation of scores. A further challenge when designing vocabulary size tests is that of sampling rate. How many words constitute a representative sample of the underlying population of words that the test is intended to measure? This paper addresses these two issues through a case study based on data from a recent and increasingly used MCT of vocabulary size: the Vocabulary Size Test. Using a criterion-related validity approach, our results show that for multiple-choice items sampled from this test, there is a discrepancy between the test scores and the scores obtained from the criterion measure, and that a higher sampling rate would be needed in order to better represent knowledge of the underlying population of words. We offer two main interpretations of these results, and discuss their implications for the construction and use of vocabulary size tests.
References
Alderson, J.C
(2005) Diagnosing foreign language proficiency: The interface between learning and assessment. London: Continuum.
Bachman, L.F
(1990) Fundamental considerations in language testing. Oxford: Oxford University Press.
Beglar, D
(2010) A Rasch-based validation of the Vocabulary Size Test. Language Testing, 27(1), 101–118. 
Elgort, I
(2011) Deliberate learning and vocabulary acquisition in a second language. Language Learning, 61(2), 367–413.
(2013) Effects of L1 definitions and cognate status of test items on the Vocabulary Size Test. Language Testing, 30(2), 253–272.
Embretson, S.E
(1996) The new rules of measurement. Psychological Assessment, 8(4), 341–349.
Eyckmans, J
(2004) Measuring receptive vocabulary size: Reliability and validity of the yes/no vocabulary test for French-speaking learners of Dutch. Utrecht: LOT.
Gyllstad, H
(2012) Validating the Vocabulary Size Test. A classical test theory approach. Poster presented at The Ninth Annual Conference of EALTA, Innsbruck, Austria, 31 May – 3 June. Retrieved from: [URL]
Horst, M., Cobb, T., & Meara, P
(1998) Beyond A Clockwork Orange: Acquiring second language vocabulary through reading. Reading in a Foreign Language, 111, 207–223.
Huibregtse, I., Admiraal, W., & Meara, P
(2002) Scores on a yes-no vocabulary test: Correction for guessing and response style. Language Testing, 19(3), 227–245.
Kamimoto, T
(2008) Nation’s vocabulary levels test and its successors: A re-appraisal. Unpublished PhD Thesis. University of Wales, Swansea.
Kane, M.T
(2013) Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.
Laufer, B., & Goldstein, Z
(2004) Testing vocabulary knowledge: Size, strength, and computer adaptiveness. Language Learning, 54(3), 399–436.
Meara, P
(1992) EFL vocabulary tests. ERIC Clearinghouse.
(1996) The dimensions of lexical competence. In G. Brown, K. Malmkjaer, & J. Williams (Eds.), Performance and competence in second language acquisition (pp. 35–53). Cambridge: Cambridge University Press.
Meara, P., & Buxton, B
(1987) An alternative to multiple choice vocabulary tests. Language Testing, 4(2), 142–154.
Messick, S
(1995) Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.
Nation, I.S.P
(2006) How large a vocabulary is needed for reading and listening? Canadian Modern Language Review/La Revue Canadienne Des Langues Vivantes, 63(1), 59–82.
(1990) Teaching and learning vocabulary. Heinle & Heinle.
Nation, I.S.P., & Beglar, D
(2007) A vocabulary size test. The Language Teacher, 31(7), 9–13.
Nation, I.S.P., & Gu, P.Y
(2007) Focus on vocabulary. Sydney: NCELTR Publications.
Nation, P
(1993) Using dictionaries to estimate vocabulary size: essential, but rarely followed, procedures. Language Testing, 10(1), 27–40.
Nation, I.S.P
(2012) Vocabulary size test instructions and description. Retrieved from [URL]. Last revised on October 23, 2012.
Paul, P.V., Stallman, A.C., & O’Rourke, J.P
(1990) Using three test formats to assess good and poor reader’s word knowledge. Technical Report No. 509 of the Center for the Study of Reading. University of Illinois. Retrieved from [URL]
Pellicer-Sánchez, A., & Schmitt, N
(2012) Scoring Yes–No vocabulary tests: Reaction time vs. nonword approaches. Language Testing, 29(4), 489–509.
Read, J
(1998) Validating a test to measure depth of vocabulary knowledge. In A.J. Kunnan (Ed.) Validation in language assessment (pp.41–61). Mahwah, NJ: Lawrence Erlbaum Associates.
(2000) Assessing vocabulary. Cambridge: Cambridge University Press.
Rupp, A.A., Ferne, T., & Choi, H
(2006) How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing, 23(4), 441–474.
Saragi, T., Nation, I.S.P., & Meister, F
(1978) Vocabulary learning and reading. System, 61, 72–78.
Schmitt, N
(2010) Researching vocabulary: A vocabulary research manual. Houndmills: Palgrave Macmillan.
Schmitt, N., & Meara, P
(1997) Researching vocabulary through a word knowledge framework: Word associations and verbal suffixes. Studies in Second Language Acquisition, 191, 17–36.
Schmitt, N., Ng, J.W.C., & Garras, J
(2011) The word associates format: Validation evidence. Language Testing, 28(1), 105–126.
Schmitt, N., & Schmitt, D
(2014) A reassessment of frequency and vocabulary size in L2 vocabulary teaching. Language Teaching, 47(4), 484–503.
Schmitt, N., Schmitt, D., & Clapham, C
(2001) Developing and exploring the behaviour of two new versions of the Vocabulary Levels Test. Language Testing, 18(1), 55–88.
Stenius Staehr, L
(2009) Vocabulary knowledge and advanced listening comprehension in English as a foreign language. Studies in Second Language Acquisition, 311, 577–607.
Stewart, J
(2012) A multiple-choice test of active vocabulary knowledge. Vocabulary Learning and Instruction, 1(1), 53–59.
(2014) Do multiple-choice options inflate estimates of vocabulary size on the VST? Language Assessment Quarterly, 11(3), 271–282.
Stewart, J., & White, D.A
(2011) Estimating guessing effects on the vocabulary levels test for differing degrees of word knowledge. TESOL Quarterly, 45(2), 370–380.
Uden, J., Schmitt, D., & Schmitt, N
(2014) Jumping from the highest graded readers to ungraded novels: Four case studies. Reading in a Foreign Language, 26(1), 1–28.
Waring, R., & Takaki, M
(2003) At what rate do learners learn and retain new vocabulary from reading a graded reader. Reading in a Foreign Language, 15(2), 130–163.
Weir, C.J
(2005) Language testing and validation. Houndmills: Palgrave McMillan.
Wesche, M., & Paribakht, T.S
(1996) Assessing second language vocabulary knowledge: Depth versus breadth. Canadian Modern Language Review, 53(1), 13–40.
Wood, B
(1928) New York experiments with new-type modern language tests. New York, NY: Macmillan.
Zhang, X
(2013) The I don’t know option in the Vocabulary Size Test. TESOL Quarterly, 47(4), 790–811.
Cited by
Cited by 66 other publications
Alzahrani, Alaa
2023. LexArabic: A receptive vocabulary size test to estimate Arabic proficiency. Behavior Research Methods 
Aviad-Levitzky, Tami, Batia Laufer & Zahava Goldstein
2019. The New Computer Adaptive Test of Size and Strength (CATSS): Development and Validation. Language Assessment Quarterly 16:3 ► pp. 345 ff.
B. Silva, Breno
2022. The Importance of General and Academic Vocabulary Learning. In Writing to Learn Academic Words [Second Language Learning and Teaching, ], ► pp. 1 ff.
B. Silva, Breno
2022. Study 1—The Assessment of Academic Vocabulary: Developing a Reliable Academic Placement Tool. In Writing to Learn Academic Words [Second Language Learning and Teaching, ], ► pp. 81 ff.
Bardel, Camilla, Henrik Gyllstad & Jörgen Tholin
2023. Research on foreign language learning, teaching, and assessment in Sweden 2012–2021. Language Teaching 56:2 ► pp. 223 ff.
Busby, Nicole Louise
Dang, Thi Ngoc Yen, Duy Van Vu & Thi My Hang Nguyen
2023. Researching Vocabulary: Mixed Methods Research. In The Encyclopedia of Applied Linguistics, ► pp. 1 ff.
De Wilde, Vanessa
2023. The auditory picture vocabulary test for English L2: A spoken receptive meaning-recognition test intended for Dutch-speaking L2 learners of English. Language Teaching Research ► pp. 136216882211474 ff.
Dong, Yang, Yi Tang, Sammy Xiao-Ying Wu, Wei-Yang Dong & Zhen Li
2020. Longitudinal Effects of Mediums of Word Explanation on L2 Vocabulary Learning Strategies Among Chinese Grade-7 Students. Frontiers in Psychology 11
Duff, Dawna & Melissa Brydon
2020. Estimates of individual differences in vocabulary size in English: how many words are needed to ‘close the vocabulary gap’?. Journal of Research in Reading 43:4 ► pp. 454 ff.
Ha, Hung Tan
2021. A Rasch-based validation of the Vietnamese version of the Listening Vocabulary Levels Test. Language Testing in Asia 11:1
İnözü, Julide & Sehnaz Sahinkarakas
2016. Exploring Relations between Pre-Service English Teachers’ Psychological and Social Well-Being and their Perceptions of Positive Classroom Environment. Eurasian Journal of Applied Linguistics 2:2 ► pp. 79 ff.
Janebi Enayat, Mostafa & Esmat Babaii
2018. Reliable predictors of reduced redundancy test performance: The interaction between lexical bonds and test takers’ depth and breadth of vocabulary knowledge. Language Testing 35:1 ► pp. 121 ff.
Koizumi, Rie & Yo In’nami
2020. Structural Equation Modeling of Vocabulary Size and Depth Using Conventional and Bayesian Methods. Frontiers in Psychology 11
Kremmel, Benjamin
2016. Word Families and Frequency Bands in Vocabulary Tests: Challenging Conventions. TESOL Quarterly 50:4 ► pp. 976 ff.
LAUFER, BATIA & TAMI AVIAD–LEVITZKY
2017. What Type of Vocabulary Knowledge Predicts Reading Comprehension: Word Meaning Recall or Word Meaning Recognition?. The Modern Language Journal 101:4 ► pp. 729 ff.
Mainz, Nina, Zeshu Shao, Marc Brysbaert & Antje S. Meyer
2017. Vocabulary Knowledge Predicts Lexical Processing: Evidence from a Group of Participants with Diverse Educational Backgrounds. Frontiers in Psychology 8
Masrai, Ahmed
2019. Vocabulary and Reading Comprehension Revisited: Evidence for High-, Mid-, and Low-Frequency Vocabulary Knowledge. SAGE Open 9:2 ► pp. 215824401984518 ff.
Masrai, Ahmed
2022. The Development and Validation of a Lemma-Based Yes/No Vocabulary Size Test. SAGE Open 12:1 ► pp. 215824402210743 ff.
Masrai, Ahmed & James Milton
2018. Measuring the contribution of academic and general vocabulary knowledge to learners' academic achievement. Journal of English for Academic Purposes 31 ► pp. 44 ff.
Masrai, Ahmed & James Milton
2019. How many words do you need to speak Arabic? An Arabic vocabulary size test. The Language Learning Journal 47:5 ► pp. 519 ff.
McLean, Stuart
2018. Evidence for the Adoption of the Flemma as an Appropriate Word Counting Unit. Applied Linguistics 39:6 ► pp. 823 ff.
Milliner, Brett, Kriss Lange, Joshua Matthews & Riko Umeki
2024. Examining EFL learners’ comprehension of derivational forms: The role of overlap with base word knowledge, word frequency, and contextual support. Language Teaching Research
Nall, Matthew, Ari Huhta & Behzad Mansouri
2022. Close‐Ended Vocabulary Assessments. In The TESOL Encyclopedia of English Language Teaching, ► pp. 1 ff.
Peters, Elke, Tom Velghe & Tinne Van Rompaey
Peters, Elke & Stuart Webb
2018. INCIDENTAL VOCABULARY ACQUISITION THROUGH VIEWING L2 TELEVISION AND FACTORS THAT AFFECT LEARNING. Studies in Second Language Acquisition 40:3 ► pp. 551 ff.
Petrescu, Maria Claudia & Rena Helms-Park
2018. The Lexical Development of Canadian-Born Romanian L1 Bilingual Kindergarteners. Languages 3:3 ► pp. 33 ff.
Puimège, Eva & Elke Peters
2019. Learning L2 vocabulary from audiovisual input: an exploratory study into incidental learning of single words and formulaic sequences. The Language Learning Journal 47:4 ► pp. 424 ff.
Qi, Shiwei, Mark Feng Teng & Ailan Fu
2024. LexCH: a quick and reliable receptive vocabulary size test for Chinese Learners. Applied Linguistics Review 15:2 ► pp. 643 ff.
Reynolds, Barry Lee, Ying-Chun Shih & Wei-Hua Wu
2018. Modeling Taiwanese adolescent learners' English vocabulary acquisition and retention: The washback effect of the College Entrance Examination Center's reference word list. English for Specific Purposes 52 ► pp. 47 ff.
Reynolds, Barry Lee, Wei-Hua Wu & Ying-Chun Shih
2020. Which Elements Matter? Constructing Word Cards for English Vocabulary Growth. SAGE Open 10:2 ► pp. 215824402091951 ff.
Robles-García, Pablo
2020. 3K-LEx. Desarrollo y validación de una prueba de amplitud léxica en español. Journal of Spanish Language Teaching 7:1 ► pp. 64 ff.
Robles‐García, Pablo
2022. Receptive vocabulary knowledge in L2 learners of Spanish: The role of high‐frequency words. Foreign Language Annals 55:4 ► pp. 963 ff.
Robles-García, Pablo, Stuart McLean, Jeffrey Stewart, Ji-young Shin & Claudia Helena Sánchez-Gutiérrez
2024. The Development and Initial Validation of O-WSVLT, a Meaning-Recall Online L2 Spanish Vocabulary Levels Test. Language Assessment Quarterly ► pp. 1 ff.
Robles-García, Pablo, Glen H. Wallace & Claudia Helena Sánchez-Gutiérrez
Roche, Thomas & Michael Harrington
2018. Offshore and onsite placement testing for English pathway programmes. Journal of Further and Higher Education 42:3 ► pp. 415 ff.
SALMELA, ROSA, MINNA LEHTONEN, STEFANO GARUSI & RAYMOND BERTRAM
2021. Lexize: A test to quickly assess vocabulary knowledge in Finnish. Scandinavian Journal of Psychology 62:6 ► pp. 806 ff.
Schmitt, Norbert, Karen Dunn, Barry O’Sullivan, Laurence Anthony & Benjamin Kremmel
2021. Introducing Knowledge‐based Vocabulary Lists (KVL). TESOL Journal 12:4
Schmitt, Norbert, Paul Nation & Benjamin Kremmel
2020. Moving the field of vocabulary assessment forward: The need for more rigorous test development and validation. Language Teaching 53:1 ► pp. 109 ff.
Schmitt, Norbert & Diane Schmitt
2020. Vocabulary in Language Teaching,
Shafi, Sehrish & Sam Hellmuth
2024. Stress Placement in English Loanwords by Speakers of Mirpur Pahari in the UK. In The Phonetics and Phonology of Heritage Languages, ► pp. 107 ff.
Silva, Breno B. & Agnieszka Otwinowska
2019. VST as a reliable academic placement tool despite cognate inflation effects. English for Specific Purposes 54 ► pp. 35 ff.
Stewart, Jeffrey, Henrik Gyllstad, Christopher Nicklin & Stuart McLean
2024. Establishing meaning recall and meaning recognition vocabulary knowledge as distinct psychometric constructs in relation to reading proficiency. Language Testing 41:1 ► pp. 89 ff.
Stewart, Jeffrey, Stuart McLean & Brandon Kramer
2017. A Response to Holster and Lake Regarding Guessing and the Rasch Model. Language Assessment Quarterly 14:1 ► pp. 69 ff.
Stewart, Jeffrey, Tim Stoeckel, Stuart McLean, Paul Nation & Geoffrey G. Pinchbeck
2021. WHAT THE RESEARCH SHOWS ABOUT WRITTEN RECEPTIVE VOCABULARY TESTING. Studies in Second Language Acquisition 43:2 ► pp. 462 ff.
Stoeckel, Tim, Phil Bennett & Stuart Mclean
2016. Is “I Don't Know” a Viable Answer Choice on the Vocabulary Size Test?. TESOL Quarterly 50:4 ► pp. 965 ff.
Stoeckel, Tim, Tomoko Ishii, Young Ae Kim, Hung Tan Ha, Nam Thi Phuong Ho & Stuart McLean
2023. A comparison of contextualized and non-contextualized meaning-recall vocabulary test formats. Research Methods in Applied Linguistics 2:3 ► pp. 100075 ff.
Stoeckel, Tim, Stuart McLean & Paul Nation
2021. LIMITATIONS OF SIZE AND LEVELS TESTS OF WRITTEN RECEPTIVE VOCABULARY KNOWLEDGE. Studies in Second Language Acquisition 43:1 ► pp. 181 ff.
Stoeckel, Tim, Jeffrey Stewart, Stuart McLean, Tomoko Ishii, Brandon Kramer & Yuko Matsumoto
2019. The relationship of four variants of the Vocabulary Size Test to a criterion measure of meaning recall vocabulary knowledge. System 87 ► pp. 102161 ff.
Stoeckel, Tim & Tomomi Sukigara
2018. A Serial Multiple‐Choice Format Designed to Reduce Overestimation of Meaning‐Recall Knowledge on the Vocabulary Size Test. TESOL Quarterly 52:4 ► pp. 1050 ff.
Sun, Ting & Hyunjeong Nam
2023. Variations in Vocabulary Depth Among Second Language Learners with Comparable Vocabulary Breadths: An Empirical Investigation. Journal of Psycholinguistic Research 52:6 ► pp. 2473 ff.
Tam, Ho I (Anna) & Barry Lee Reynolds
2023. The relationship between extramural English engagement and the vocabulary size of L1 Cantonese speakers in Macau. ITL - International Journal of Applied Linguistics 174:1 ► pp. 49 ff.
Urdaniz, Raquel Perez & Sophia Skoufaki
2022. Spanish L1 EFL learners’ recognition knowledge of English academic vocabulary: The role of cognateness, word frequency and length. Applied Linguistics Review 13:4 ► pp. 661 ff.
Vilkaitė, Laura
2017. Incidental acquisition of collocations in L2. ITL - International Journal of Applied Linguistics 168:2 ► pp. 248 ff.
Warnby, Marcus
2022. Receptive academic vocabulary knowledge and extramural English involvement – is there a correlation?. ITL - International Journal of Applied Linguistics 173:1 ► pp. 120 ff.
Warnby, Marcus
2023. Academic vocabulary knowledge among adolescents in university preparatory programmes. Journal of English for Academic Purposes 61 ► pp. 101203 ff.
Warnby, Marcus, Hans Malmström & Kajsa Yang Hansen
2023. Linking scores from two written receptive English academic vocabulary tests—The VLT-Ac and the AVT. Language Testing 40:3 ► pp. 548 ff.
Yunjiu, Luo, Wei Wei & Ying Zheng
2022. Artificial Intelligence-Generated and Human Expert-Designed Vocabulary Tests: A Comparative Study. SAGE Open 12:1 ► pp. 215824402210821 ff.
Z Szabo, Csaba
2016. Exploring the Mental Lexicon of the Multilingual: Vocabulary Size, Cognate Recognition and Lexical Access in the L1, L2 and L3. Eurasian Journal of Applied Linguistics 2:2 ► pp. 1 ff.
Zeng, Yuntao, Qiuxia Lu, Matthew P. Wallace, Yawei Guo, Chun-Wai Fan & Xiaofei Chen
2022. Understanding Sustainable Development of English Vocabulary Acquisition: Evidence from Chinese EFL Learners. Sustainability 14:11 ► pp. 6532 ff.
Zhang, Haiwei, Peijian Paul Sun, Xueyan Zhang, Silin Fan & Yunji Zhang
2023. A Systematic Review of Chinese Character Size Tests From 1930 to 2021. SAGE Open 13:3
Zhang, Haiwei, Peng Sun, Yaowaluk Bianglae & Winda Widiawati
2024. The development of a Chinese vocabulary proficiency test (CVPT) for learners of Chinese as a second/foreign language . Language Testing 41:2 ► pp. 412 ff.
Zhang, Pengchong & Suzanne Graham
2020. Vocabulary learning through listening: Comparing L2 explanations, teacher codeswitching, contrastive focus-on-form and incidental learning. Language Teaching Research 24:6 ► pp. 765 ff.
Zhang, Songshan & Xian Zhang
2022. The relationship between vocabulary knowledge and L2 reading/listening comprehension: A meta-analysis. Language Teaching Research 26:4 ► pp. 696 ff.
Zhang, Xian, Jianda Liu & Haiyang Ai
2020. Pseudowords and guessing in the Yes/No format vocabulary test. Language Testing 37:1 ► pp. 6 ff.
[no author supplied]
This list is based on CrossRef data as of 1 april 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.