This paper investigated the effect of English-Hebrew loanwords on English vocabulary test scores when the number of loanwords in the test is random and when it is representative of their proportion in the vocabulary lists from which the test items were taken. 303 EFL learners, speakers of Hebrew as L1, at three L2 proficiency levels, received tests with no loanwords, with a representative number of loanwords and with a random number of loanwords in four modalities: word form recall, word meaning recall, word form recognition, word meaning recognition. Though different effects were found for different modalities and different language proficiencies, the score increases from the representative loanword test version to the random loanword version were low and the effect sizes of the differences were very low. It is suggested that the inclusion of loanwords in vocabulary tests may not inflate the true vocabulary knowledge score. Keywords: cognates, loanwords, representative number of loanwords, random number of loanwords, L2 vocabulary testing, vocabulary size tests, loanword effect on vocabulary test