Article published In:
ITL - International Journal of Applied Linguistics: Online-First Articles

Development and initial validation of a yes/no vocabulary test for North Sámi

Drawing on item response theory and signal detection theory

ORCID logo | Lund University
 | University of Konstanz | UiT The Arctic University of Norway | Cluster of Excellence “The Politics of Inequality”
 | University of Konstanz | Cluster of Excellence “The Politics of Inequality”
This paper accounts for the development and initial validation of a yes/no vocabulary test of North Sámi called North Sámi Vocabulary Test (NSVT). North Sámi (NS) is an Indigenous language spoken in northern Scandinavia. Being an endangered language, NS is in need of revitalisation efforts. One contribution is the provision of proficiency assessment tools. We administered a 75-item NSVT version (50 real verbs and 25 pseudoverbs) to users of North Sámi in Sweden and Norway (N = 289). Evidence of high item- and person-based reliability and separation indices were observed, as well as support for a number of validity facets. High correlations existed between NSVT scores and self-reported and rated spoken proficiency. The NSVT provides quick and reliable assessment of vocabulary knowledge and an indication of overall language proficiency in North Sámi.
Keywords: North Sámi, language proficiency, Indigenous language, vocabulary, yes/no test, Item Response Theory, Signal Detection Theory, Rasch
Article outline
  • Introduction
  • Background
    • Testing vocabulary knowledge
    • Psychometric theories and techniques
    • Aims and research questions
  • Developing the NSVT
    • Item construction and piloting
    • Item selection
    • Main administration of the NSVT in Sweden and Norway
  • Rasch and signal detection theory analyses
    • Summary statistics from the Rasch model and variable map
    • A SDT-based score calculation
    • Probing the relation between NSVT scores and spoken ability in North Sámi
  • Discussion
    • Reliability and validity
    • Recommendation for future implementations of the NSVT and future directions
    • Challenges
  • Conclusions
  • Notes
  • References
Available under the Creative Commons Attribution (CC BY) 4.0 license.

For any use beyond this license, please contact the publisher at [email protected].

Published online: 5 August 2024
DOI logo
https://doi.org/10.1075/itl.23005.gyl
References (75)
References
Aikio-Puoskari, U. (2018). Revitalization of Sámi languages in three nordic countries: Finland, Norway, and Sweden. In L. Hinton, L. Huss, & G. Roche (Eds.), The Routledge Handbook of Language Revitalization (pp. 355–363). Routledge. DOI logoGoogle Scholar
Albury, N. J. (2016). Holding them at arm’s length: A critical review of Norway’s policy on Sámi language maintenance. Journal of Home Language Research, 1 1, 1–16. DOI logoGoogle Scholar
Alderson, C. (2005). Diagnosing foreign language proficiency. Continuum.Google Scholar
Amenta, S., Badan, L., & Brysbaert, M. (2021). LexITA: A quick and reliable assessment tool for Italian L2 receptive vocabulary size. Applied Linguistics, 42 (2), 292–314. DOI logoGoogle Scholar
Aryadoust, V., Ng, L. Y., & Sayama, H. (2021). A comprehensive review of Rasch measurement in language assessment: Recommendations and guidelines for research. Language Testing, 38 (1), 6–40. DOI logoGoogle Scholar
Beeckmans, R., Eyckmans, J., Janssens, V., Dufranne, M., & Van de Velde, H. (2001). Examining the yes/no vocabulary test: Some methodological issues in theory and practice. Language Testing, 18 (3), 235–274. DOI logoGoogle Scholar
Beglar, D. (2010). A Rasch-based validation of the Vocabulary Size Test. Language Testing, 27 (1), 101–118. DOI logoGoogle Scholar
Belančić, K. (2020). Language policy and Sámi education in Sweden: ideological and implementational spaces for Sámi language use [Unpublished PhD thesis]. Umeå University.
Belančić, K., & Lindgren, E. (2017). Discourses of functional bilingualism in the Sámi curriculum in Sweden. International Journal of Bilingual Education and Bilingualism, 23 (5), 601–616. DOI logoGoogle Scholar
Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Lawrence Erlbaum Associates.Google Scholar
Brysbaert, M. (2013). Lextale_FR a fast, free, and efficient test to measure language proficiency in French. Psychologica Belgica, 53 (1), 23–37. DOI logoGoogle Scholar
Duncan, P. W., Bode, R. K., Lai, S. M., & Perera, S. (2003). Rasch analysis of a new stroke-specific outcome scale: The stroke impact scale. Archives of Physical Medicine and Rehabilitation, 84 1, 950–963. DOI logoGoogle Scholar
Eberhard, D. M., Simons, G. F., & Fennig, C. D. (2022). Ethnologue: Languages of the World. 25th Edition. SIL International. Available at: [URL]
Eyckmans, J. (2004). Measuring receptive vocabulary size. LOT.Google Scholar
Ferin, M., Gyllstad, H., Venagli, I., Zordan, A., & Kupisch, T. (2023). LexVEN: A quick vocabulary test for proficiency in Venetan. Isogloss, 9 (1), 1–31. DOI logoGoogle Scholar
Giellatekno – Saami Language Technology, UiT The Arctic University of Norway and The Divvun group at UiT The Arctic University of Norway (2015). SIKOR North Saami free corpus, Common Language Resources and Technology Infrastructure Norway (CLARINO) Bergen Repository. [URL]
Green, R. (2013). Statistical analyses for language testers. Routledge. DOI logoGoogle Scholar
Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York: Wiley.Google Scholar
Grenoble, L. A., & Osipov, B. (2023). The dynamics of bilingualism in language shift ecologies. Linguistic Approaches to Bilingualism, 13 (1), 1–39. DOI logoGoogle Scholar
Gyllstad, H. (2007). Testing English collocations: Developing receptive tests for use with advanced Swedish learners [Unpublished PhD Thesis]. Lund University.
(2013). Looking at L2 vocabulary knowledge dimensions from an assessment perspective: Challenges and potential solutions. In C. Bardel, C. Lindqvist, & B. Laufer (Eds.), L2 vocabulary acquisition, knowledge and use. New perspectives on assessment and corpus analysis (pp. 11–28). Eurosla Monograph Series 2. European Association of Second Language Acquisition.Google Scholar
(2020). Measuring knowledge of multiword items. In S. Webb (Ed.), The Routledge Handbook of Vocabulary Studies (pp. 387–405). Routledge.Google Scholar
Gyllstad, H., Vilkaitė, L., & Schmitt, N. (2015). Assessing vocabulary size through multiple-choice formats: Issues with guessing and sampling rates. ITL International Journal of Applied Linguistics, 166 (2), 278–306. DOI logoGoogle Scholar
Gyllstad, H., McLean, S., & Stewart, J. (2021). Using confidence intervals to determine adequate item sample sizes for vocabulary tests: An essential but overlooked practice. Language Testing, 38 (4), 558–579. DOI logoGoogle Scholar
Harrington, M., & Carey, M. (2009). The on-line Yes/No test as a placement tool. System, 37 (4), 614–626. DOI logoGoogle Scholar
Hashimoto, B. J. (2021). Is frequency enough?: The frequency model in vocabulary size testing. Language Assessment Quarterly, 18 (2), 171–187. DOI logoGoogle Scholar
Henning, G. (1987). A guide to language testing. Heinle and Heinle.Google Scholar
Hirvonen, V. (2008). ‘Out on the fells, I feel like a Sámi’: Is There Linguistic and Cultural Equality in the Sámi School? In N. Hornberger (Ed.), Can schools save indigenous languages? (pp. 15–41). Palgrave Macmillan. DOI logoGoogle Scholar
Huibregtse, I., Admiraal, W., & Meara, P. (2002). Scores on a yes–no vocabulary test: Correction for guessing and response style. Language Testing, 19 (3), 227–245. DOI logoGoogle Scholar
Izura, C., Cuetos, F., & Brysbaert, M. (2014). Lextale-Esp: A test to rapidly and efficiently assess the Spanish vocabulary size. Psicológica, 35 (1), 49–66.Google Scholar
Knoch, U., & McNamara, T. (2015). Rasch analysis. In L. Plonsky (Ed.), Advancing quantitative methods in second language research (pp. 275–304). Routledge. DOI logoGoogle Scholar
Kupisch, T., Arona, S., Besler, A., Cruschina, S., Ferin, M., Gyllstad, H., & Venagli, I. (2023). LexSIC: A quick vocabulary test for proficiency in Sicilian. Isogloss, 9 (1), 1–24. DOI logoGoogle Scholar
Leiner, D. J. (2014). SoSci Survey (Version 2.4.00-i). [Computer software].Google Scholar
Lemhöfer, K., & Broersma, M. (2012). Introducing LexTALE: A quick and valid lexical test for advanced learners of English. Behavior Research Methods, 44(2), 325–343. DOI logoGoogle Scholar
Linacre, J. M. (2002). What do infit and outfit, mean-square and standardized mean. Rasch Measurement Transactions, 16 (2), 878.Google Scholar
(2018, 3 Sep.). Detecting multidimensionality in Rasch data using Winsteps Table 23 1 [Video]. YouTube. [URL]
(2021a). Winsteps® Rasch measurement computer program (Version 5.1.6). [URL]
(2021b). A user’s guide to Winsteps® Ministep Rasch-model computer programs (Program manual 5.1.6). [URL]
Lloyd-Smith, A., Bergman, F., Sapir, Y., Yasar, R., & Kupisch, T. (2023a). Why language matters. Inequality perceptions among the in Sweden and Norway. Policy Paper 11, Policy Papers of the Cluster “The Politics of Inequality”.Google Scholar
Lloyd-Smith, A., Bergman, F., Hund, L., & Kupisch, T. (2023b). Can policies improve language vitality? The Sámi languages in Sweden and Norway. Frontiers in Psychology, 14 1. DOI logoGoogle Scholar
McIvor, O. (2020). Indigenous Language Revitalization and Applied Linguistics: Parallel Histories, Shared Futures? Annual Review of Applied Linguistics, 401, 78–96. DOI logoGoogle Scholar
McIvor, O., & Jacobs, P. (2016). NETOLNEW Language Learning Assessment Tool (v3.1). Retrieved from: [URL]
McLean, S., Kramer, B., & Beglar, D. (2015). The creation and validation of a listening vocabulary levels test. Language Teaching Research, 19 (6), 741–760. DOI logoGoogle Scholar
Meara, P. (1996). The dimensions of lexical competence. In G. Brown, K. Malmkjær, & J. Williams (Eds.), Performance and competence in second language acquisition (pp. 33–55). Cambridge University Press.Google Scholar
Meara, P. M., & Buxton, B. (1987). An alternative to multiple choice vocabulary tests. Language Testing, 4 (2), 142–154. DOI logoGoogle Scholar
Meara, P., & Miralpeix, I. (2017). Tools for researching vocabulary. Multilingual Matters.Google Scholar
Mercator European Research Centre on Multilingualism and Language Learning (2020). SÁMI. The Sámi language in education in Sweden.Google Scholar
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13–104). American Council on education and Macmillan.Google Scholar
(1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50 (9), 741–749. DOI logoGoogle Scholar
Milton, J., Wade, J., & Hopkins, N. (2010). Aural word recognition and oral competence in English as a foreign language. In R. Chacón-Beltrán, C. Abello-Contesse, & M. del Mar Torreblanca-López (Eds.), Insights into non-native vocabulary teaching and learning (pp. 83–98). Multilingual Matters. DOI logoGoogle Scholar
Min, S., & Aryadoust, V. (2021). A systematic review of item response theory in language assessment: Implications for the dimensionality of language ability. Studies in Educational Evaluation, 68 1, 100963. DOI logoGoogle Scholar
Minde, H. (2003). Assimilation of the Sami: Implementation and consequences. Acta Borealia, 20 1, 121–146. DOI logoGoogle Scholar
Mochida, K., & Harrington, M. (2006). The Yes/No test as a measure of receptive vocabulary knowledge. Language Testing, 23 (1), 73–98. DOI logoGoogle Scholar
Nakata, T., Tamura, Y., & Aubrey, S. (2020). Examining the validity of the LexTALE test for Japanese college students. The journal of Asia TEFL, 17 (2), 335–348.Google Scholar
Nicklin, C., & Vitta, J. P. (2022). Assessing Rasch measurement estimation methods across R packages with yes/no vocabulary test data. Language Testing, 39 (4), 513–540. DOI logoGoogle Scholar
Nation, P., & Beglar, D. (2007). A vocabulary size test. The Language Teacher, 31 (7), 9–13.Google Scholar
Ockey, G. (2017). Item response theory. In G. Fulcher, & F. Davidson (Eds.), The Routledge Handbook of Language Testing (pp. 336–349). Routledge.Google Scholar
Outakoski, H. (2015). Multilingual Literacy among Young Learners of North Sámi: Contexts, Complexity and Writing in Sápmi. Department of language studies, Umeå University.Google Scholar
Pellicer-Sánchez, A., & Schmitt, N. (2012). Scoring Yes–No vocabulary tests: Reaction time vs. nonword approaches. Language Testing, 29 (4), 489–509. DOI logoGoogle Scholar
Plonsky, L. (2023). Sampling and generalizability in Lx research: A second-order synthesis. Languages, 8 (1), 75. DOI logoGoogle Scholar
Plonsky, L., & Oswald, F. L. (2014). How big is “big”? Interpreting effect sizes in L2 research. Language Learning, 64 (4), 878–912. DOI logoGoogle Scholar
Rasmussen, T., & Nolan, J. S. (2011). Reclaiming Sámi languages: Indigenous language emancipation from East to West. International Journal of the Sociology of Language, 2091, 35–55. DOI logoGoogle Scholar
Salminen, T. (2007). Europe and North Asia. In C. Moseley (Ed.), Encyclopedia of the World’s Endangered Languages (pp. 211–281). Routledge.Google Scholar
Salmela, R., Lehtonen, M., Garusi, S., & Bertram, R. (2021). Lexize: A test to quickly assess vocabulary knowledge in Finnish. Scandinavian Journal of Psychology, 62 (6), 806–819. DOI logoGoogle Scholar
Schmitt, N., Schmitt, D., & Clapham, C. (2001). Developing and exploring the behaviour of two new versions of the Vocabulary Levels Test. Language Testing, 18 (1), 55–88. DOI logoGoogle Scholar
Solstad, K. J., Balto, A. M. V., Nygaard, V., Jodesfsen, E., & Solstad, M. (2012). Samisk Sprakundersokelne (7).Google Scholar
Staehr, L. S. (2008). Vocabulary size and the skills of listening, reading and writing. Language Learning Journal, 36 (2), 139–152. DOI logoGoogle Scholar
Stanislaw, H., & Todorov, N. (1999). Calculation of signal detection theory measures. Behavior Research Methods, Instruments and Computers, 31 (1), 137–149. DOI logoGoogle Scholar
Stubbe, R. (2012). Do pseudoword false alarm rates and overestimation rates in yes–no vocabulary tests change with Japanese university students’ ability levels? Language Testing, 29 (4), 471–488. DOI logoGoogle Scholar
Trosterud, T. (2008). Language assimilation during the modernisation process: Experiences from Norway and north-west Russia. Acta Borealia, 25 (2), 93–112. DOI logoGoogle Scholar
Uchihara, T., & Clenton, J. (2020). Investigating the role of vocabulary size in second language speaking ability. Language Teaching Research, 24 (4), 540–556. DOI logoGoogle Scholar
Vangsnes, Ø. (2019, December 13–16). A prognosis for Sámi in Norway: School as key to revitalization [Conference presentation]. Foundation for Endangered Languages XXIII, University of Sydney, Australia. [URL]
Yanagisawa, A., & Webb, S. (2020). Measuring depth of vocabulary knowledge. In S. Webb (Ed.), The Routledge Handbook of Vocabulary Studies (pp. 371–386). Routledge.Google Scholar
Zhang, X., Liu, J., & Ai, H. (2020). Pseudowords and guessing in the Yes/No format vocabulary Test. Language Testing, 37 (1), 6–30. DOI logoGoogle Scholar
Zimmerman, J., Broder, P. K., Shaughnessy, J. J., & Underwood, B. J. (1977). A recognition test of vocabulary using signal-detection measures and some correlates of word and non-word recognition. Intelligence, 1 (1), 5–13. DOI logoGoogle Scholar