Key words when text forms the unit of study
Sizing up the effects of different measures
Stephen Jeaco | Xi’an Jiaotong-Liverpool University
Throughout the social sciences, there has been growing pressure to present effect sizes when publishing empirical data (see American Psychological Association, 2001; Parsons & Nelson, 2004). While it seems indisputable that for the majority of quantitative research foci, effect size is an essential element of statistical analysis, this paper argues that specifically for key word analysis in corpus linguistics, the means of reporting effect size must depend on the level of the unit of study of each investigation (single text, collection or large corpus). After exploring some main criticisms of the log-likelihood measure, this paper unpacks the parameters of different measures for keyness and how they might address underlying concerns. It maintains that for the exploration of foregrounded/deviant/salient/marked features in text, the use of log-likelihood scores to rank the results is still fit for purpose and coupled with Bayes Factors is a solid approach for key word analyses.
Keywords: keyness, effect size, key word analysis, log-likelihood, ranking
Published online: 28 August 2020
https://doi.org/10.1075/ijcl.18053.jea
https://doi.org/10.1075/ijcl.18053.jea
References
[ p. 152 ]References
Anthony, L.
(2019) AntConc (Version 3.5.8) [Computer software]. Waseda University. https://www.laurenceanthony.net/software
American Psychological Association
Baker, P.
Baker, P., Gabrielatos, C., Khosravinik, M., Krzyżanowski, M., McEnery, T., & Wodak, R.
Brezina, V., McEnery, T., & Wattam, S.
Cobb, T.
(2000) The Compleat Lexical Tutor (Version 8.3) [Computer software]. Retrieved November, 2019, from http://www.lextutor.ca
Croft, W. B., Metzler, D., & Strohman, T.
Dunning, T.
Egbert, J., & Biber, D.
Gabrielatos, C.
Gabrielatos, C., & Marchi, A.
(2012) Keyness: Appropriate metrics and practical issues [Paper presentation]. CADS International Conference 2012, University of Bologna, Italy. https://www.researchgate.net/publication/261708842_Keyness_Appropriate_metrics_and_practical_issues
Gabrielatos, C., Torgersen, E. N., Hoffmann, S., & Fox, S.
Grissom, R. J., & Kim, J. J.
Hardie, A.
(2014a) Log Ratio – an informal introduction. ESRC Centre for Corpus Approaches to Social Science (CASS). http://cass.lancs.ac.uk/?p=1133
(2014b) Statistical identification of keywords, lockwords and collocations as a two-step procedure [Paper presentation]. ICAME 35 Conference, University of Nottingham, Nottingham, UK.
Jeaco, S.
Johnston, J. E., Berry, K. J., & Mielke Jr, P. W.
Kass, R. E., & Raftery, A. E.
Kilgarriff, A., Rychly, P., Smrz, P., & Tugwell, D.
(2004) The Sketch Engine [Paper presentation]. The 2003 International Conference on Natural Language Processing and Knowledge Engineering, Beijing, China.
Lee, D. Y. W.
Leech, G. N., Hundt, M., Mair, C., & Smith, N.
Leech, G. N., & Short, M. H.
Lexical Computing Ltd
(2014) Statistics used in the Sketch Engine. https://www.sketchengine.eu/wp-content/uploads/ske-statistics.pdf
Mahlberg, M., Stockwell, P., de Joode, J., Smith, C., & O’Donnell, M. B.
Parsons, T. D., & Nelson, N. W.
Partington, A.
Plonsky, L., & Oswald, F. L.
Raftery, A. E.
Rayson, P.
n.d.). UCREL Log-likelihood and effect size calculator. Retrieved November, 2019, from http://ucrel.lancs.ac.uk/llwizard.html
Rayson, P., Berridge, D., & Francis, B.
[ p. 154 ]
(2004) Extending the Cochran rule for the comparison of word frequencies between corpora [Paper presentation]. The 7th International Conference on Statistical Analysis of Textual Data, Louvain-la-Neuve, Belgium. https://eprints.lancs.ac.uk/id/eprint/12424/1/rbf04_jadt.pdf
Rayson, P., & Garside, R.
(2000) Comparing corpora using frequency profiling [Paper presentation]. The Workshop on Comparing Corpora, Hong Kong University of Science and Technology, Hong Kong. https://eprints.lancs.ac.uk/id/eprint/11882/1/rg_acl2000.pdf
Rayson, P., Leech, G., & Hodges, M.
Read, T. R. C., & Cressie, N. A. C.
(2019a) WordSmith Tools online manual “KeyWords: Calculation”. Retrieved November, 2019, from https://lexically.net/downloads/version7/HTML/keywords_calculate_info.html
(2019b) WordSmith Tools online manual “KeyWords”. Retrieved November, 2019, from https://lexically.net/downloads/version7/HTML/keywords2.html
(2019c) WordSmith Tools online manual “KeyWords: Thinking about keyness”. Retrieved November, 2019, from https://lexically.net/downloads/version7/HTML/thinking_about_keyness.html
(2019d) WordSmith Tools online manual “KeyWords: Keyness definition”. Retrieved November, 2019, from https://lexically.net/downloads/version7/HTML/keyness_definition.html
Scott, M., & Tribble, C.
Wilson, A.
Cited by
Cited by 1 other publications
This list is based on CrossRef data as of 08 february 2021. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.