Comparing Corpora
Adam Kilgarriff | ITRI, University of Brighton
Corpus linguistics lacks strategies for describing and comparing corpora. Currently most descriptions of corpora are textual, and questions such as ‘what sort of a corpus is this?’, or ‘how does this corpus compare to that?’ can only be answered impressionistically. This paper considers various ways in which different corpora can be compared more objectively. First we address the issue, ‘which words are particularly characteristic of a corpus?’, reviewing and critiquing the statistical methods which have been applied to the question and proposing the use of the Mann-Whitney ranks test. Results of two corpus comparisons using the ranks test are presented. Then, we consider measures for corpus similarity. After discussing limitations of the idea of corpus similarity, we present a method for evaluating corpus similarity measures. We consider several measures and establish that a\chi\tsup{2}-based one performs best. All methods considered in this paper are based on word and ngram frequencies; the strategy is defended.
Keywords: similarity, homogeneity, word frequency
Published online: 17 December 2001
https://doi.org/10.1075/ijcl.6.1.05kil
https://doi.org/10.1075/ijcl.6.1.05kil
Cited by
Cited by 143 other publications
No author info given
Adamov, Abzetdin Z.
Adamov, Abzetdin Z.
Ahmed, Saifuddin, Kokil Jaidka & Jaeho Cho
Ancarno, Clyde
Anthony, L.
Anthony, L.
Babych, Bogdan, Fangzhong Su, Anthony Hartley, Ahmet Aker, Monica Lestari Paramita, Paul Clough & Robert Gaizauskas
Baker, Paul
Benko, Vladimír
Bentum, Martijn, Louis ten Bosch, Antal van den Bosch & Mirjam Ernestus
Bernardini, Silvia & Adriano Ferraresi
Billot, Jennie & Virginia King
Bittar, André, Sumithra Velupillai, Angus Roberts & Rina Dutta
Bradlow, Eric
Brezina, Vaclav & Miriam Meyerhoff
Brooks, Penelope A. & Atif M Memon
Brunzel, Marko & Myra Spiliopoulou
Buckley, Kevin & Carl Vogel
Buhler, Thomas & Richard Stephenson
Buzila, Eduard
Carpuat, Marine, Pascale Fung & Grace Ngai
Carter, Pelham, Matt Gee, Hollie McIlhone, Harkeeret Lally & Robert Lawson
Chatzitheodorou, Konstantinos & Vassilios Kappatos
Chen, Xiao Xiao, Shi Li Ge & Min Lin
Cislaru, Georgeta & Frédérique Sitri
Conrad, Susan M. & Kimberly R. LeVelle
Cook, Paul & Laurel J. Brinton
Crawford, Lynn, Julien Pollack & David England
Crawford, Lynn, Julien Pollack & David England
Curtotti, Michael & Eric McCreath
Cvrček, Václav, Zuzana Komrsková, David Lukeš, Petra Poukarová, Anna Řehořková, Adrian Jan Zasina & Vladimír Benko
Dag, J.N., V. Gervasi, S. Brinkkemper & B. Regnell
Davalos, Sergio & Ehsan H. Feroz
De Groot, Elizabeth, Catherine Nickerson, Hubert Korzilius & Marinel Gerritsen
Desagulier, Guillaume
Desagulier, Guillaume
Desagulier, Guillaume
Dias, Gaël & Špela Vintar
Dunn, Jonathan
Dyevre, Arthur & Nicolas Lampach
Eiteljoerge, Sarah F. V., Nausicaa Pouscoulous & Elena V. M. Lieven
Evans, Roger, Alexander Gelbukh, Gregory Grefenstette, Patrick Hanks, Miloš Jakubíček, Diana McCarthy, Martha Palmer, Ted Pedersen, Michael Rundell, Pavel Rychlý, Serge Sharoff & David Tugwell
Fitz, Hartmut & Franklin Chang
Gablasova, Dana, Vaclav Brezina & Tony McEnery
García Salido, Marcos & Marcos Garcia
Ghazzawi, Nizar, Benoît Robichaud, Patrick Drouin & Fatiha Sadat
Gries, Stefan Th.
Grimm, Stephan, Andreas Abecker, Johanna Völker & Rudi Studer
Hafsa, Fatima, Nicole Darnall & Stuart Bretschneider
Hahn, Udo & Joachim Wermter
Holtz, Peter, Emanuel Deutschmann & Henrik Dobewall
Horák, Aleš, Vít Baisa, Adam Rambousek & Vít Suchomel
Jamatia, Anupam, Björn Gambäck & Amitava Das
Jordanous, Anna
Jordanous, Anna, Bill Keller & Peter Csermely
Kettunen, Kimmo
Kettunen, Kimmo & Matti La Mela
Kilgarriff, Adam
Kilgarriff, Adam, Vít Baisa, Jan Bušta, Miloš Jakubíček, Vojtěch Kovář, Jan Michelfeit, Pavel Rychlý & Vít Suchomel
Kilgarriff, Adam & Gregory Grefenstette
Koplenig, Alexander
Koplenig, Alexander
Koplenig, Alexander
Koplenig, Alexander
Kuosmanen, Sonja
La, Hanh Luong & Rudi Bekkers
Lam, Phoenix
Lebedeva, Maria, Tatyana Veselovskaya, Olga Kupreshchenko & Antonina Laposhina
Lee, Thomas
Lefer, Marie-Aude
LI, BO, ERIC GAUSSIER & DAN YANG
Li, Paul Luo, Connie Yang, Sophia Liu & Mary Hu
Lien, Hsin-Yi
Lindemann, David & Iñaki San Vicente
Lippincott, Thomas, Diarmuid Ó Séaghdha & Anna Korhonen
Liu, Xiao-Yue & Chunyu Kit
Liu, Yong, Pavel Dmitriev, Yifei Huang, Andrew Brooks, Li Dong, Mengyue Liang, Zvi Boshernitzan, Jiwei Cao & Bobby Nguy
Mitchell, Andrew S.
Moadel‐Attie, Roxanne, Sheri R. Levy, Bonita London & Rami Al‐Rfou
Nakamata, Naoki
Natt och Dag, Johan & Vincenzo Gervasi
Neumann, Stella & Silvia Hansen-Schirra
Nishino, Ryutaro & Kayoko Nohara
O'Boyle, Aisling
Padó, Sebastian & Mirella Lapata
Pal, Samiran, Avinash Singh, Soham Datta, Sangameshwar Patil, Indrajit Bhattacharya & Girish Palshikar
Palander-Collin, Minna & Minna Nevala
Panunzi, A., M. Fabbri & M. Moneglia
Paquot, Magali & Luke Plonsky
Paulett, John M. & Curtis P. Langlotz
PEIRSMAN, YVES, DIRK GEERAERTS & DIRK SPEELMAN
Pinnis, Mārcis, Nikola Ljubešić, Dan Ştefănescu, Inguna Skadiņa, Marko Tadić, Tatjana Gornostaja, Špela Vintar & Darja Fišer
Pojanapunya, Punjaporn & Richard Watson Todd
Ponomareva, Natalia & Mike Thelwall
Remus, Robert
Rietveld, Toni, Roeland Van hout & Mirjam Ernestus
Rodríguez-Puente, Paula
Salido, Marcos García, Paula Lorente & Almudena Basanta
Santini, Marina, Arne Jönsson, Wiktor Strandqvist, Gustav Cederblad, Mikael Nyström, Marjan Alirezaie, Leili Lind, Eva Blomqvist, Maria Lindén & Annica Kristoffersson
Santini, Marina, Wiktor Strandqvist, Mikael Nyström, Marjan Alirezai & Arne Jönsson
Savage, Saiph, Andres Monroy-Hernandez & Tobias Höllerer
Savoy, Jacques
Schröter, Julian, Keli Du, Julia Dudar, Cora Rok & Christof Schöch
Schäfer, Roland & Felix Bildhauer
Shaikina, Alevtina A. & Anastasia A. Funkner
Sharma, Raksha, Dibyendu Mondal & Pushpak Bhattacharyya
Sharon, Aviv J. & Ayelet Baram-Tsabari
Siegert, Ingo, Ronald Böck & Andreas Wendemuth
Sierra, Gerardo, Tonatiuh Hernández-García, Helena Gómez-Adorno, Gemma Bel-Enguix, David Pinto, Beatriz Beltrán & Vivek Singh
Skadiņa, Inguna, Robert Gaizauskas, Andrejs Vasiļjevs & Monica Lestari Paramita
Smith, Catherine, Svenja Adolphs, Kevin Harvey & Louise Mullany
Sokolova, Marina, Mohak Shah & Stan Szpakowicz
Sokolova, Marina & Stan Szpakowicz
Staab, S.
Stanislav, Petr, Jan Švec & Luboš Šmídl
Subašić, Ilija & Bettina Berendt
Supran, Geoffrey & Naomi Oreskes
Swanson, Zane L., Edward Walker, Louise Miller & Richard Green
Tay, Dennis
Taylor, Charlotte
Tellez, Fernando Perez, David Pinto, John Cardiff & Paolo Rosso
Th. Gries, Stefan & Martin Hilpert
Underwood, Ted, Michael L. Black, Loretta Auvil & Boris Capitanu
Uribe, Diego
VERHAGEN, VÉRONIQUE, MARIA MOS, AD BACKUS & JOOST SCHILPEROORD
Vogel, Carl & Gerard Lynch
Vogel, Carl, Gerard Lynch & Jerom Janssen
Wirén, Mats
Woźniak, Michał, Agnieszka Wołos, Urszula Modrzyk, Rafał L. Górski, Jan Winkowski, Michał Bajczyk, Sara Szymkuć, Bartosz A. Grzybowski & Maciej Eder
Šemelík, Martin
Švec, Jan, Jan Hoidekr, Daniel Soutner & Jan Vavruška
Švec, Jan, Jan Lehečka, Pavel Ircing, Lucie Skorkovská, Aleš Pražák, Jan Vavruška, Petr Stanislav & Jan Hoidekr
This list is based on CrossRef data as of 15 april 2022. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.