Open-source Corpora
Using the net to fish for linguistic data
Serge Sharoff | University of Leeds
The paper proposes a methodology for collecting “open-source” corpora, i.e. corpora that are automatically collected from the Internet and distributed in the form of a list of links with open-source software for recreating their full text. The result is a random snapshot of Internet pages which contain stretches of connected text in a given language. The paper discusses a methodology for acquiring such corpora, two ways of documenting them (using a set of metatextual categories and by comparison to frequency lists from existing corpora) and their function as benchmarks for comparing results of linguistic inquiry. Experiments with a variety of languages show that Internet-derived corpora can be successfully used in the absence of large representative corpora that are rare and expensive to build.
Keywords: representative corpora, frequency lists, Internet, corpus composition
Published online: 08 December 2006
https://doi.org/10.1075/ijcl.11.4.05sha
https://doi.org/10.1075/ijcl.11.4.05sha
Cited by
Cited by 28 other publications
Atwell, Eric Steven
2018.
In Diachronic Corpora, Genre, and Language Change [Studies in Corpus Linguistics, 85],
► pp. 65 ff. 
Biber, Douglas & Jesse Egbert
2016. Register Variation on the Searchable Web.
Journal of English Linguistics 44:2 ► pp. 95 ff.
Biber, Douglas & Jesse Egbert
2018.
In Register Variation Online,
Biber, Douglas, Jesse Egbert & Mark Davies
2015. Exploring the composition of the searchable web: a corpus-based taxonomy of web registers.
Corpora 10:1 ► pp. 11 ff.
Chang, Ching-Yun & Stephen Clark
2014. Practical Linguistic Steganography using Contextual Synonym Substitution and a Novel Vertex Coding Method.
Computational Linguistics 40:2 ► pp. 403 ff.
Coupé, Christophe, Yoon Mi Oh, Dan Dediu & François Pellegrino
2019. Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche.
Science Advances 5:9
Dash, Niladri Sekhar & S. Arulmozi
2018.
In History, Features, and Typology of Language Corpora,
► pp. 125 ff.
De Belder, Jan & Marie-Francine Moens
2012.
In Computational Linguistics and Intelligent Text Processing [Lecture Notes in Computer Science, 7182],
► pp. 426 ff.
DELIN, J., S. SHAROFF, S. LILLFORD & C. BARNES
2007. Linguistic support for concept selection decisions.
Artificial Intelligence for Engineering Design, Analysis and Manufacturing 21:2 ► pp. 123 ff.
Egbert, Jesse, Douglas Biber & Mark Davies
2015. Developing a bottom-up, user-based method of web register classification.
Journal of the Association for Information Science and Technology 66:9 ► pp. 1817 ff.
Fletcher, William H.
2012.
In The Encyclopedia of Applied Linguistics,
Ibrahim, Anna, Patricia E. Cowell & Rosemary A. Varley
2017. Word frequency predicts translation asymmetry.
Journal of Memory and Language 95 ► pp. 49 ff.
Kajiwara, Tomoyuki, Daiki Nishihara, Tomonori Kodaira & Mamoru Komachi
2020. Language Resources for Japanese Lexical Simplification.
Journal of Natural Language Processing 27:4 ► pp. 801 ff.
Kehoe, Andrew
2020.
In A Practical Handbook of Corpus Linguistics,
► pp. 329 ff.
Kilgarriff, Adam, Frieda Charalabopoulou, Maria Gavrilidou, Janne Bondi Johannessen, Saussan Khalil, Sofie Johansson Kokkinakis, Robert Lew, Serge Sharoff, Ravikiran Vadlapudi & Elena Volodina
2014. Corpus-based vocabulary lists for language learners for nine languages.
Language Resources and Evaluation 48:1 ► pp. 121 ff.
Masrai, Ahmed & James Milton
2019. How many words do you need to speak Arabic? An Arabic vocabulary size test.
The Language Learning Journal 47:5 ► pp. 519 ff.
McCarthy, Diana
2007. Computers getting the drift.
Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 365:1861 ► pp. 3019 ff.
McCarthy, Diana
2009. Word Sense Disambiguation: An Overview.
Language and Linguistics Compass 3:2 ► pp. 537 ff.
McCarthy, Diana
2011.
In Computational Linguistics and Intelligent Text Processing [Lecture Notes in Computer Science, 6608],
► pp. 238 ff.
McCarthy, Diana & Roberto Navigli
2009. The English lexical substitution task.
Language Resources and Evaluation 43:2 ► pp. 139 ff.
McCarthy, Diana, Ravi Sinha & Rada Mihalcea
2013. The cross-lingual lexical substitution task.
Language Resources and Evaluation 47:3 ► pp. 607 ff.
Moon, Taesun & Katrin Erk
2013. An inference-based model of word meaning in context as a paraphrase distribution.
ACM Transactions on Intelligent Systems and Technology 4:3 ► pp. 1 ff.
Schaeffer-Lacroix, Eva
2020. Les corpus web à travers le prisme de l’ALMT.
Corpus :20
Song, Jiayin, Jingyue Hu, Leung-Pun Wong, Lap-Kei Lee & Tianyong Hao
2020.
In Database Systems for Advanced Applications. DASFAA 2020 International Workshops [Lecture Notes in Computer Science, 12115],
► pp. 80 ff.
Song, Jiayin, Yingshan Shen, John Lee & Tianyong Hao
2020.
In Natural Language Processing and Chinese Computing [Lecture Notes in Computer Science, 12430],
► pp. 132 ff.
TALALAKINA, EKATERINA, DENIS STUKAL & MIKHAIL KAMROTOV
2020. Developing and Validating an Academic Vocabulary List in Russian: A Computational Approach.
The Modern Language Journal 104:3 ► pp. 618 ff.
Voelkel, Svenja & Franziska Kretzschmar
2021.
In Introducing Linguistic Research,
Wild, Kate, Andrew Church, Diana McCarthy & Jacquelin Burgess
2013. Quantifying lexical usage: vocabulary pertaining to ecosystems and the environment.
Corpora 8:1 ► pp. 53 ff.
This list is based on CrossRef data as of 05 january 2022. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.