Article published In:
International Journal of Corpus LinguisticsVol. 11:4 (2006) ► pp.435–462
Open-source Corpora
Using the net to fish for linguistic data
The paper proposes a methodology for collecting “open-source” corpora, i.e. corpora that are automatically collected from the Internet and distributed in the form of a list of links with open-source software for recreating their full text. The result is a random snapshot of Internet pages which contain stretches of connected text in a given language. The paper discusses a methodology for acquiring such corpora, two ways of documenting them (using a set of metatextual categories and by comparison to frequency lists from existing corpora) and their function as benchmarks for comparing results of linguistic inquiry. Experiments with a variety of languages show that Internet-derived corpora can be successfully used in the absence of large representative corpora that are rare and expensive to build.
Keywords: representative corpora, frequency lists, Internet, corpus composition
Cited by
Cited by 41 other publications
Alarcon, Rodrigo, Lourdes Moreno, Paloma Martínez & Natalia Grabar
2023. EASIER corpus: A lexical simplification resource for people with cognitive impairments. PLOS ONE 18:4 ► pp. e0283622 ff. 
Alrwaita, Najla, Lotte Meteyard, Carmel Houston-Price & Christos Pliatsikas
2022. Is There an Effect of Diglossia on Executive Functions? An Investigation among Adult Diglossic Speakers of Arabic. Languages 7:4 ► pp. 312 ff.
Alzahrani, Alaa
2023. LexArabic: A receptive vocabulary size test to estimate Arabic proficiency. Behavior Research Methods
Atwell, Eric Steven
2018. Classical and modern Arabic corpora. In Diachronic Corpora, Genre, and Language Change [Studies in Corpus Linguistics, 85], ► pp. 65 ff.
Biber, Douglas & Jesse Egbert
2016. Register Variation on the Searchable Web. Journal of English Linguistics 44:2 ► pp. 95 ff.
Biber, Douglas & Jesse Egbert
2018. Register Variation Online,
Biber, Douglas, Jesse Egbert & Mark Davies
2015. Exploring the composition of the searchable web: a corpus-based taxonomy of web registers. Corpora 10:1 ► pp. 11 ff.
Casanueva, Annalí
2021. Can Chants in the Street Change Parliament’s Tune? The Effects of the 15M Social Movement on Spanish Elections. SSRN Electronic Journal
Chang, Ching-Yun & Stephen Clark
2014. Practical Linguistic Steganography using Contextual Synonym Substitution and a Novel Vertex Coding Method. Computational Linguistics 40:2 ► pp. 403 ff.
Coupé, Christophe, Yoon Mi Oh, Dan Dediu & François Pellegrino
2019. Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche. Science Advances 5:9
Dash, Niladri Sekhar & S. Arulmozi
2018. Web Text Corpus. In History, Features, and Typology of Language Corpora, ► pp. 125 ff.
De Belder, Jan & Marie-Francine Moens
2012. A Dataset for the Evaluation of Lexical Simplification. In Computational Linguistics and Intelligent Text Processing [Lecture Notes in Computer Science, 7182], ► pp. 426 ff.
DELIN, J., S. SHAROFF, S. LILLFORD & C. BARNES
2007. Linguistic support for concept selection decisions. Artificial Intelligence for Engineering Design, Analysis and Manufacturing 21:2 ► pp. 123 ff.
Egbert, Jesse, Douglas Biber & Mark Davies
2015. Developing a bottom‐up, user‐based method of web register classification. Journal of the Association for Information Science and Technology 66:9 ► pp. 1817 ff.
El-Farahaty, Hanem, Nouran Khallaf & Amani Alonayzan
2023. Building the Leeds Monolingual and Parallel Legal Corpora of Arabic and English Countries’ Constitutions: Methods, Challenges and Solutions. Corpus Pragmatics 7:2 ► pp. 103 ff.
Fletcher, William H.
2012. Corpus Analysis of the World Wide Web. In The Encyclopedia of Applied Linguistics,
Ibrahim, Anna, Patricia E. Cowell & Rosemary A. Varley
2017. Word frequency predicts translation asymmetry. Journal of Memory and Language 95 ► pp. 49 ff.
jaschke, philipp, Sulin Sardoschau & Marco Tabellini
2021. Scared Straight? Threat and Assimilation of Refugees in Germany. SSRN Electronic Journal
Jaschke, Philipp, Sulin Sardoschau & Marco Tabellini
2022. Scared Straight? Threat and Assimilation of Refugees in Germany. SSRN Electronic Journal
Kajiwara, Tomoyuki, Daiki Nishihara, Tomonori Kodaira & Mamoru Komachi
2020. Language Resources for Japanese Lexical Simplification. Journal of Natural Language Processing 27:4 ► pp. 801 ff.
Kehoe, Andrew
2020. Web Corpora. In A Practical Handbook of Corpus Linguistics, ► pp. 329 ff.
Kilgarriff, Adam, Frieda Charalabopoulou, Maria Gavrilidou, Janne Bondi Johannessen, Saussan Khalil, Sofie Johansson Kokkinakis, Robert Lew, Serge Sharoff, Ravikiran Vadlapudi & Elena Volodina
2014. Corpus-based vocabulary lists for language learners for nine languages. Language Resources and Evaluation 48:1 ► pp. 121 ff.
Li, Jiaxuan, Jinghua Ou & Ming Xiang
2023. Context-specific effects of violated expectations: ERP evidence. Cognition 241 ► pp. 105628 ff.
Li, Rongying, Wenxiu Xie, John Lee & Tianyong Hao
2023. SCA-CLS: A New Semantic-Context-Aware Framework for Community-Oriented Lexical Simplification. In Natural Language Processing and Chinese Computing [Lecture Notes in Computer Science, 14302], ► pp. 69 ff.
Li, Rongying, Wenxiu Xie, Jiayin Song, Leung-Pun Wong, Fu Lee Wang & Tianyong Hao
2022. 2022 IEEE International Symposium on Product Compliance Engineering - Asia (ISPCE-ASIA), ► pp. 1 ff.
Masrai, Ahmed & James Milton
2019. How many words do you need to speak Arabic? An Arabic vocabulary size test. The Language Learning Journal 47:5 ► pp. 519 ff.
McCarthy, Diana
2007. Computers getting the drift. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 365:1861 ► pp. 3019 ff.
McCarthy, Diana
2009. Word Sense Disambiguation: An Overview. Language and Linguistics Compass 3:2 ► pp. 537 ff.
McCarthy, Diana
2011. Measuring Similarity of Word Meaning in Context with Lexical Substitutes and Translations. In Computational Linguistics and Intelligent Text Processing [Lecture Notes in Computer Science, 6608], ► pp. 238 ff.
McCarthy, Diana & Roberto Navigli
2009. The English lexical substitution task. Language Resources and Evaluation 43:2 ► pp. 139 ff.
McCarthy, Diana, Ravi Sinha & Rada Mihalcea
2013. The cross-lingual lexical substitution task. Language Resources and Evaluation 47:3 ► pp. 607 ff.
Moon, Taesun & Katrin Erk
2013. An inference-based model of word meaning in context as a paraphrase distribution. ACM Transactions on Intelligent Systems and Technology 4:3 ► pp. 1 ff.
Schaeffer-Lacroix, Eva
2020. Les corpus web à travers le prisme de l’ALMT. Corpus :20
Sharoff, Serge, Reinhard Rapp & Pierre Zweigenbaum
2023. Building Comparable Corpora. In Building and Using Comparable Corpora for Multilingual Natural Language Processing [Synthesis Lectures on Human Language Technologies, ], ► pp. 17 ff.
Sharoff, Serge, Reinhard Rapp & Pierre Zweigenbaum
2023. Introduction. In Building and Using Comparable Corpora for Multilingual Natural Language Processing [Synthesis Lectures on Human Language Technologies, ], ► pp. 1 ff.
Song, Jiayin, Jingyue Hu, Leung-Pun Wong, Lap-Kei Lee & Tianyong Hao
2020. A New Context-Aware Method Based on Hybrid Ranking for Community-Oriented Lexical Simplification. In Database Systems for Advanced Applications. DASFAA 2020 International Workshops [Lecture Notes in Computer Science, 12115], ► pp. 80 ff.
Song, Jiayin, Yingshan Shen, John Lee & Tianyong Hao
2020. A Hybrid Model for Community-Oriented Lexical Simplification. In Natural Language Processing and Chinese Computing [Lecture Notes in Computer Science, 12430], ► pp. 132 ff.
TALALAKINA, EKATERINA, DENIS STUKAL & MIKHAIL KAMROTOV
2020. Developing and Validating an Academic Vocabulary List in Russian: A Computational Approach. The Modern Language Journal 104:3 ► pp. 618 ff.
Voelkel, Svenja & Franziska Kretzschmar
2021. Introducing Linguistic Research,
Wild, Kate, Andrew Church, Diana McCarthy & Jacquelin Burgess
2013. Quantifying lexical usage: vocabulary pertaining to ecosystems and the environment. Corpora 8:1 ► pp. 53 ff.
Wilkens, Rodrigo, Alessandro Dalla Vecchia, Marcely Zanon Boito, Muntsa Padró & Aline Villavicencio
2014. Size Does Not Matter. Frequency Does. A Study of Features for Measuring Lexical Complexity. In Advances in Artificial Intelligence -- IBERAMIA 2014 [Lecture Notes in Computer Science, 8864], ► pp. 129 ff.
This list is based on CrossRef data as of 31 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.