Cover not available
Article published In:
International Journal of Corpus Linguistics
Vol. 28:3 (2023) ► pp.318–343

Assessing word commonness

Adding dispersion to frequency

ORCID logo | University of Bergen

The article investigates the two main corpus indicators of word commonness, frequency and dispersion, through a cross-validation analysis of frequency and four dispersion measures (‘Range’, ‘Chi-squared’, ‘Deviation of Proportions’ and ‘Juilland’s D’). The approach provides an estimation of the capacity of the named measures to predict the distribution of corpus items in an extracted language sample. Based on a dataset of 273 Norwegian compounds, the results show that especially Deviation of Proportions is a robust measure of dispersion that can be used in conjunction with frequency to substantiate assertions of word commonness based on corpus data. In addition, dispersion measures do not only reflect what sort of distribution the frequency statistic is generated from, but also how reliable the frequency estimation in the corpus sample is in terms of giving an accurate representation of frequency in the language variety that the corpus is sampled from.

Keywords: dispersion, frequency, word commonness, cross-validation, lexicography
Article outline
  • 1.Introduction
  • 2.Dictionaries and distributions
    • 2.1Dictionaries and core vocabulary
    • 2.2The case of compounds in Norwegian
    • 2.3Disadvantages of frequency, advantages of dispersion
    • 2.4Comparing measures
      • 2.4.1Frequency
      • 2.4.2Relative range
      • 2.4.3Chi squared χ2
      • 2.4.4Deviation of proportions (DP)
      • 2.4.5Juilland’s D (D_uneq)
      • 2.4.6Which one(s) to choose?
    • 2.5What are distributions in corpora supposed to tell us?
  • 3.Methodology
    • 3.1Cross-validation and correlation analysis
  • 4.Results and analysis
    • 4.1Frequency
    • 4.2Range
    • 4.3Chi squared
    • 4.4Deviation of proportions
    • 4.5Juilland’s D
  • 5.Summary and concluding discussion
  • Notes
  • References
Published online: 25 November 2022
DOI logo
https://doi.org/10.1075/ijcl.21037.eke
References
Baayen, R. H.
(2008) Analyzing Linguistic Data: A Practical Introduction to Statistics Using R. Cambridge University Press. DOI logoGoogle Scholar
Bakken, K.
(1998) Leksikalisering av sammensetninger: en studie av leksikaliseringsprosessen belyst ved et gammelnorsk diplommateriale fra 1300-tallet [Lexicalisation of compounds: A study of the process of lexicalisation in light of a Norse diploma from 14th century]. [Doctoral dissertation, University of Oslo]. Acta Humaniora.
Balota, D. A., & Spieler, D. H.
(1998) The utility of item-level analyses in model evaluation: A reply to Seidenberg and Plaut. Psychological Science, 9 (3), 238–240. DOI logoGoogle Scholar
Biber, D., Reppen, R., Schnur, E., & Ghanem, R.
(2016) On the (non)utility of Juilland’s D to measure lexical dispersion in large corpora. International Journal of Corpus Linguistics, 21 (4), 439–464. DOI logoGoogle Scholar
CLARINO UiB Portal
(2020) Norwegian Newspaper Corpus Bokmål. Created by Norsk aviskorpus. Retrieved February 23, 2021, from [URL]
Durkin, P.
(2016) Introduction. In P. Durkin (Ed.), The Oxford Handbook of Lexicography. Oxford University Press.Google Scholar
Egbert, J., Burch, B., & Biber, D.
(2020) Lexical dispersion and corpus design. International Journal of Corpus Linguistics, 25 (1), 89–115. DOI logoGoogle Scholar
Fellbaum, C. D.
(2015) The treatment of multi-word units in lexicography. In P. Durkin (Ed.), The Oxford Handbook of Lexicography (pp. 411–424). Oxford University Press. DOI logoGoogle Scholar
Fjeld, R. V., Nøklestad, A., & Hagen, K.
(2020) Leksikografisk bokmålskorpus (LBK) – bakgrunn og bruk [Lexicographic corpus of Bokmål (LBK) – background and usage]. In J. B. Johannessen & K. Hagen (Eds.), Leksikografi og korpus. En hyllest til Ruth Vatvedt Fjeld, Oslo Studies in Language 11 (1) (pp. 47–59). DOI logoGoogle Scholar
Glossa
(2020)  Leksikografisk bokmålskorpus (LBK). Retrieved February 23, 2021, from [URL]
Gries, S. T.
(2008) Dispersions and adjusted frequencies in corpora. International Journal of Corpus Linguistics, 13 (4), 403–437. DOI logoGoogle Scholar
(2010) Dispersions and adjusted frequencies in corpora: Further explorations. In S. T. Gries, S. Wulff, & M. Davies (Eds.), Corpus Linguistic Applications: Current Studies, New Directions (pp. 197–112). Rodopi. DOI logoGoogle Scholar
(2020) Analyzing dispersion. In M. Paquot & S. T. Gries (Eds.), A Practical Handbook of Corpus Linguistics. Springer. DOI logoGoogle Scholar
Lyne, A. A.
(1985) The Vocabulary of French Business Correspondence: Word Frequencies, Collocations and Problems of Lexicometric Method. Slatkine-Champion.Google Scholar
R Core Team
(2020) R: A language and environment for statistical computing (Version 4.1.1). R Foundation for Statistical Computing. [URL]
Savický, P., & Hlavácová, J.
(2002) Measures of word commonness. Journal of Quantitative Linguistics, 9 (3), 215–231. DOI logoGoogle Scholar
Stefanowitsch, A.
(2020) Corpus Linguistics: A Guide to the Methodology. Language Science Press.Google Scholar