Cover not available
Part of
Language and Text: Data, models, information and applications
Edited by Adam Pawłowski, Jan Mačutek, Sheila Embleton and George Mikros
[Current Issues in Linguistic Theory 356] 2021
► pp. 145–162

From distinguishability to informativity

A quantitative text model for detecting random texts

| Goethe University Frankfurt
| Goethe University Frankfurt
| Goethe University Frankfurt
| Goethe University Frankfurt
We present a study of the distinctiveness of random and non-random texts based on text characteristics of quantitative linguistics. We additionally experiment with text features that evaluate contiguity associations among sentences by means of BERT (Bidirectional Encoder Representations from Transformers). To this end, we experiment with generative models for random texts as currently discussed in the context of neural networks. The chapter contributes to the clarification of deficits of existing random text models and of the informativeness of quantitative text features.
Keywords: random text, quantitative text characteristics, text classification, BERT
Article outline
  • 1.Introduction
  • 2.Text corpora and their quantification
    • 2.1Quantification
    • 2.2Text corpora and their randomization
    • 2.3Classification and evaluation methods
  • 3.Results
  • 4.Discussion
  • 5.Conclusion
  • Notes
  • References
Published online: 22 December 2021
DOI logo
https://doi.org/10.1075/cilt.356.10kon
References (43)
References
Altmann, Gabriel. 1988. Wiederholungen in Texten. Bochum: Brockmeyer.Google Scholar
Baayen, Harald, Hans van Halteren & Fiona Tweedie. 1996. Outside the cave of shadows: Using syntactic annotation to enhance authorship attribution. Literary and Linguistic Computing 11(3). 121–131. DOI logoGoogle Scholar
Bahdanau, Dzmitry, Kyunghyun Cho & Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.Google Scholar
Bengio, Yoshua, Réjean Ducharme, Pascal Vincent & Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research 3. 1137–1155.Google Scholar
Biemann, Chris. 2007. A random text model for the generation of statistical language invariants. In Candace Sidner, Tanja Schultz, Matthew Stone & ChengXiang Zhai (eds.), Human language technologies 2007: The conference of the North American chapter of the association for computational linguistics; proceedings of the main conference, 105–112. Rochester, NY: Association for Computational Linguistics.Google Scholar
Boser, Bernhard E., Isabelle M. Guyon & Vladimir N. Vapnik. 1992. A training algorithm for optimal margin classifiers. In David Haussler (ed.), Proceedings of the fifth annual workshop on computational learning theory, 144–152. New York: Association for Computing Machinery. DOI logoGoogle Scholar
Breiman, Leo. 2001. Random forests. Machine Learning 45(1). 5–32. DOI logoGoogle Scholar
Campolongo, Francesca, Jessica Caribon & Andrea Saltelli. 2007. An effective screening design for sensitivity analysis of large models. Environmental Modelling & Software 22(10). 1509–1518. DOI logoGoogle Scholar
Čech, Radek. 2015. Text length and the lambda frequency structure of a text. In George K. Mikros & Ján Macutek (eds.), Sequences in language and text, 71–88. Berlin: De Gruyter Mouton. DOI logoGoogle Scholar
Čech, Radek, Ioan-Iovitz Popescu & Gabriel Altmann. 2013. Methods of analysis of a thematic concentration of the text. Czech and Slovak Linguistic Review 3. 4–21.Google Scholar
Chang, Chih-Chung & Chih-Jen Lin. 2011. Libsvm: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) 2(3). 1–27. DOI logoGoogle Scholar
Cheng, Jianpeng, Li Dong & Mirella Lapata. 2016. Long short-term memory-networks for machine reading. In Jian Su, Kevin Duh & Xavier Carreras (eds.), Proceedings of the 2016 conference on empirical methods in natural language processing, 551–561. Austin, TX: Association for Computational Linguistics. DOI logoGoogle Scholar
Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement 20(1). 37–46. DOI logoGoogle Scholar
Devlin, Jacob, Ming-Wei Chang, Kenton Lee & Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.Google Scholar
Maria Dolores Esteban & Domingo Morales. 1995. A summary on entropy statistics. Kybernetika 31(4). 337–346.Google Scholar
Gabrilovich, Evgeniy & Shaul Markovitch. 2006. Overcoming the brittleness bottleneck using Wikipedia: Enhancing text categorization with encyclopedic knowledge. In Proceedings of the twenty-first national conference on artificial intelligence, 2006 Jul 16 (Vol. 6, pp. 1301–1306) Boston, MA: AAAI Press.Google Scholar
Hirsch, Jorge E. 2005. An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences 102(46). 16569–16572. DOI logoGoogle Scholar
Hochreiter, Sepp & Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9(8). 1735–1780. DOI logoGoogle Scholar
Joachims, Thorsten. 2002. Learning to classify text using support vector machines. Boston: Kluwer. DOI logoGoogle Scholar
Kubát, Miroslav, Vladimír Matlach & Radek Čech. 2014. Quita. Quantitative Index Text Analyzer. Lüdenscheid: RAM-Verlag.Google Scholar
McIntosh, Robert P. 1967. An index of diversity and the relation of certain concepts to diversity. Ecology 48(3). 392–404. DOI logoGoogle Scholar
Mehler, Alexander. 2005. Eigenschaften der textuellen Einheiten und Systeme [Properties of textual units and systems]. In Reinhard Köhler, Gabriel Altmann & Rajmund G. Piotrowski (eds.), Quantitative linguistik. ein internationales handbuch / quantitative linguistics. An international handbook, 325–348. Berlin: De Gruyter.Google Scholar
Mehler, Alexander, Peter Geibel & Olga Pustylnikov. 2007. Structural classifiers of text types: Towards a novel model of text representation. Journal for Language Technology and Computational Linguistics (JLCL) 22(2). 51–66.Google Scholar
Mehler, Alexander, Wahed Hemati, Rüdiger Gleim & Daniel Baumartz. 2018. VienNA: Auf dem Weg zu einer Infrastruktur für die verteilte interaktive evolutionäre Verarbeitung natürlicher Sprache. In Henning Lobin, Roman Schneider & Andreas Witt (eds.), Forschungsinfrastrukturen und digitale Informationssysteme in der germanistischen Sprachwissenschaft, Volume 6, 149–176). Berlin: De Gruyter.Google Scholar
Mehler, Alexander, Wahed Hemati, Tolga Uslu & Andy Lücking. 2018. A multidimensional model of syntactic dependency trees for authorship attribution. In Jingyang Jiang & Haitao Liu (eds.), Quantitative analysis of dependency structures, 315–348. Berlin: De Gruyter. DOI logoGoogle Scholar
Metz, Luke, Ben Poole, David Pfau & Jascha Sohl-Dickstein. 2016. Unrolled generative adversarial networks. arXiv preprint arXiv:1611.02163.Google Scholar
Morris, Max D. 1991. Factorial sampling plans for preliminary computational experiments. Technometrics 33(2). 161–174. DOI logoGoogle Scholar
Parzen, Emanuel. 1963. On spectral analysis with missing observations and amplitude modulation. Sankhyā: The Indian Journal of Statistics, Series A, 383–392.Google Scholar
Popescu, Ioan-Iovitz. 2009. Word frequency studies, Volume 64. Berlin: Walter de Gruyter.Google Scholar
Popescu, Ioan-Iovitz & Gabriel Altmann. 2006. Some aspects of word frequencies. Glottometrics 13. 23–46.Google Scholar
. 2007. Writer’s view of text generation. Glottometrics, 15, 71–81.Google Scholar
. 2011. Thematic concentration in texts. Issues in quantitative linguistics 2. 110–116.Google Scholar
Popescu, Ioan-Iovitz, Radek Čech & Gabriel Altmann. 2011. The lambda-structure of texts. Lüdenscheid: Ram-Verlag.Google Scholar
Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei & Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1(8). 9.Google Scholar
Reiter, Ehud & Robert Dale. 1997. Building applied natural language generation systems. Natural Language Engineering 3(1). 57–87. DOI logoGoogle Scholar
Saltelli, Andrea. 2002. Making best use of model evaluations to compute sensitivity indices. Computer physics communications 145(2). 280–297. DOI logoGoogle Scholar
Saltelli, Andrea, Paola Annoni, Ivano Azzini, Francesca Campolongo, Marco Ratto & Stefano Tarantola. 2010. Variance based sensitivity analysis of model output. design and estimator for the total sensitivity index. Computer Physics Communications 181(2). 259–270. DOI logoGoogle Scholar
Smola, Alex J. & Bernhard Schölkopf. 2004. A tutorial on support vector regression. Statistics and computing 14(3). 199–222. DOI logoGoogle Scholar
Sobol, Ilya M. 2001. Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Mathematics and Computers in Simulation 55(1–3). 271–280. DOI logoGoogle Scholar
Székely, Gábor J., Maria L. Rizzo & Nail K. Bakirov. 2007. Measuring and testing dependence by correlation of distances. The Annals of Statistics 35(6). 2769–2794. DOI logoGoogle Scholar
Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser & Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems (pp. 5998–6008).Google Scholar
Wimmer, Gejza. 2005. The type-token-relation. In Reinhard Köhler, Gabriel Altmann & Rajmund G. Piotrowski (eds.), Quantitative Linguistik: Ein internationales Handbuch [Quantitative linguistics: An international handbook], 361–368. Berlin: De Gruyter.Google Scholar
Zhu, Yukun, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba & Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, 19–27. Cambridge, MA: IEEE. DOI logoGoogle Scholar