Chapter published in:
Crossroads Semantics: Computation, experiment and grammarEdited by Hilke Reckman, Lisa Lai-Shen Cheng, Maarten Hijzelendoorn and Rint Sybesma
[Not in series 210] 2017
► pp. 23–37
Chapter 2Experimental research
Problems and opportunities in the big-data era
Henk Cremers | University of Amsterdam
Experimental research in psychology, psycholinguistics or medicine provides quantitative and therefore seemingly conclusive and trustworthy evidence. However, it has been convincingly shown that most research findings are actually false. This has hardly influenced the dominant scientific evaluation system which reflects a continued trust in the unbiasedness of data by a strong reliance on simple quantifications of scientific quality and productivity, such as number of publications and number of citations. This state of affairs is remarkable in the light of a long history of strong criticism of commonly used inference methods and scientific evaluation systems, which is now backed by large-scale research projects directly questioning the reproducibility of scientific findings. This way, the large amounts of data – “big-data” – have helped to uncover some of these problematic issues, but also provided a more open attitude towards data and code sharing. In addition, novel analytic frameworks may help to better integrate empirical data with computational models.
Article outline
- 1.Introduction
- 2.The questionable empirical toolbox
- 2.1Bias
- 2.2Null hypothesis testing
- 2.3Theory testing
- 3.Scientific publications and evaluation
- 3.1The economy of the publication and evaluation systems
- 3.2Alternatives for the evaluation system
- 4.More data, more problems?
-
Notes -
References
Published online: 12 April 2017
https://doi.org/10.1075/z.210.02cre
https://doi.org/10.1075/z.210.02cre
References
References
Bargh, John A., Mark Chen & Lara Burrows
Bornmann, Lutz, Rüdiger Mutz & Hans-Dieter Daniel
Breiman, Leo
Button, Katherine S., John P. A. Ioannidis, Claire Mokrysz, Brian A. Nosek, Jonathan Flint, Emma S. J. Robinson & Marcus R. Munafò
Carp, Joshua
Collobert, Ronan & Jason Weston
Coyne, James C & Jacob N de Voogd
Doyen, Stéphane, Olivier Klein, Cora-Lise Pichon & Axel Cleeremans
Dreber, Anna, Thomas Pfeiffer, Johan Almenberg, Siri Isaksson, Brad Wilson, Yiling Chen, Brian A. Nosek & Magnus Johannesson
Falk, Ruma & Charles W. Greenbaum
Fanelli, Daniele
Fugelsang, Jonathan A., Courtney B. Stein, Adam E. Green & Kevin N. Dunbar
Gilbert, Daniel T., Gary King, Stephen Pettigrew & Timothy D. Wilson
Greenberg, Steven A.
Greenwald, Anthony G.
Hill, Ray
Ioannidis, John P. A.
Ioannidis, John P A.
Ioannidis, John P.A.
Ioannidis, John P. A., Kevin W. Boyack, Henry Small, Aaron A. Sorensen & Richard Klavans
Klein, Richard A., Kate A. Ratliff, Michelangelo Vianello, Reginald B. Adams, Štěpán Bahník, Michael J. Bernstein, Konrad Bocian, et al.
Kramer, Adam D. I., Jamie E. Guillory & Jeffrey T. Hancock
Kriegeskorte, Nikolaus
Kriegeskorte, Nikolaus, Marieke Mur & Peter Bandettini
Lazer, David, Ryan Kennedy, Gary King & Alessandro Vespignani
Lindley, Dennis V.
Loftus, Geoffrey R.
Maxwell, Scott E.
Meehl, Paul E.
Meehl, PE.
Miller, G.
Nickerson, R. S.
Nuijten, Michèle B., Chris H. J. Hartgerink, Marcel A. L. M. van Assen, Sacha Epskamp & Jelte M. Wicherts
Open Science Collaboration
Pfeiffer, Thomas, Lars Bertram & John P. A. Ioannidis
Rodriguez-Esteban, Raul, Ivan Iossifov & Andrey Rzhetsky
Rosenthal, Robert
Rosenthal, Robert & John Gaito
Rutledge, Robb B., Nikolina Skandali, Peter Dayan & Raymond J. Dolan
Simmons, Joseph P., Leif D. Nelson & Uri Simonsohn
Simonsohn, Uri, Leif D. Nelson & Joseph P. Simmons
Thompson, William C. & Edward L. Schumann
Tversky, Amos & Daniel Kahneman
Tyler, Lorraine K, Teresa P. L. Cheung, Barry J. Devereux & Alex Clarke
Wicherts, Jelte M. & Marcel A. L. M. van Assen
Yarkoni, Tal, Russell A. Poldrack, Thomas E. Nichols, David C. Van Essen & Tor D. Wager