Chapter published in:
Corpus Approaches to Social Media
Edited by Sofia Rüdiger and Daria Dayter
[Studies in Corpus Linguistics 98] 2020
► pp. 149–174

Chapter 7
Constructing corpora from images and text

An introduction to Visual Constituent Analysis

| Loughborough University
| Lancaster University
| Lancaster University
Visual analysis represents a significant oversight in the corpus literature, and possibly one that may lead to unintended omissions, particularly when analysing social media. In this chapter we introduce Visual Constituent Analysis (VCA), a method of multimodal corpus construction that allows researchers to construct and analyse visual aspects of online media in large-scale corpora. The chapter addresses the shortcomings of a purely textual approach to discourse analysis when dealing with social media texts and offers a solution using computer ‘Vision’-based image annotation (in our case Google Cloud Vision). Finally, we demonstrate how our approach can be used to analyse a sample of 150,000 micro-blog posts from Twitter and show the difference in level of user interaction with combined image/texts over language-only social media texts.
Keywords: corpus construction, multimodality, images, Twitter, information operations
Article outline
  • 1.Introduction
  • 2.Opportunities and obstacles – Why ‘visual’?
  • 3.Tentative solutions – Constructing ‘constituents’
    • 3.1Vision
    • 3.2Concatenating outputs
  • 4.Analysing T-IRA with VCA
    • 4.1Hostile state information operations
    • 4.2T-IRA – a general overview
  • 5.Quantifying the importance of images
    • 5.1Images, likes and retweets
    • 5.2Text-image overlap
  • 6.T-IRA – A case study
    • 6.1Data
    • 6.2Method
    • 6.3Analysis
      • 6.3.1Image reference (IR)
      • 6.3.2Image and text reference (ITR)
  • 7.Concluding remarks
  • Acknowledgments
  • Notes
  • References
Published online: 04 November 2020
https://doi.org/10.1075/scl.98.07chr
References
Allwood, Jens
2008 Multimodal corpora. In Corpus Linguistics: An International Handbook, Vol. 1 [Handbücher zur Sprach- und Kommunikationswissenschaft 29.1], A. Lüdeling & M. Kytö (eds), 207–225. Berlin: Mouton de Gruyter.Google Scholar
Archer, Dawn, Wilson, Andrew & Rayson, Paul
2002 Introduction to the USAS Category System. Lancaster University. http://​ucrel​.lancs​.ac​.uk​/usas​/usas​_guide​.pdf (15 October 2019).
Baker, Paul
2006 Using Corpora in Discourse Analysis. London: Continuum.Google Scholar
Baker, Paul & McEnery, Tony
2015 Corpora and Discourse Studies: Integrating Discourse and Corpora. London: Palgrave. CrossrefGoogle Scholar
Bateman, John, Wildfeuer, Jamina & Hiippala, Tuomo
2017 Multimodality: Foundations, Research and Analysis – A Problem-oriented Introduction. Berlin: De Gruyter. CrossrefGoogle Scholar
Bednarek, Monika & Caple, Helen
2017 Introducing a new topology for (multimodal) discourse analysis. In Transforming Contexts: Papers from the 44th International Systemic Functional Congress, Phil Chappell & John S. Knox (eds), 19–25. Wollongong: 44th ISFC Organising Committee.Google Scholar
Caple, Helen
2018 Analysing the multimodal text. In Corpus Approaches to Discourse: A Critical Review, Charlotte Taylor & Anna Marchi (eds), 85–109. London: Routledge. CrossrefGoogle Scholar
2019 “Lucy says today she is a Labordoodle”: How the dogs-of-Instagram reveal voter preferences. Social Semiotics 29(4): 427–447. CrossrefGoogle Scholar
Chang, Yan-Shou
2017 Fine-grained attention for image caption generation. Multimedia Tools and Applications 77(7): 2959–2971.Google Scholar
Chen, Jianfu, Kuznetsova, Polina, Warren, David S. & Choi, Yejin
2015 Deja image-captions: A corpus of expressive descriptions in repetition. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Matt Post (ed.), 504–514. Denver CO: ACL.Google Scholar
CNSSI 4009–2015
CNSSI 4009 Committee on National Security Systems (CNSS) Glossary. Strategic Environmental Research and Development Program (SERDP), Committee on National Security Systems. https://​www​.serdp​-estcp​.org​/content​/download​/47576​/453617​/file​/CNSSI%204009%20Glossary%202015​.pdf (20 October 2019).
Deighton-Smith, Nova & Bell, Beth T.
2018 Objectifying fitness: A content and thematic analysis of #Fitspiration images on social media. Psychology of Popular Media Culture 7(4): 1–41. CrossrefGoogle Scholar
Fanelli, Gabriele, Gall, Juergen, Romsdorfer, Harald, Weise, Thibaut & van Gool, Luc
2010 3D vision technology for capturing multimodal corpora: Chances and challenges. In LREC Workshop on Multimodal Corpora, Rada Mihalcea, Joyce Chai & Anoop Sarkar (eds), 70–73. Valletta: European Language Resources Association (ELRA).Google Scholar
Firth, John R.
1957 Papers in Linguistics, 1934–1951. London: OUP.Google Scholar
Gatt, Albert, Tanti, Marc, Muscat, Adrian, Paggio, Patrizia, Farrugia, Reuben A., Borg, Claudia, Camilleri, Kenneth, Rosner, Micahel & van der Plas, Lonneke
2018 Face2Text: Collecting an annotated image description corpus for the generation of rich face descriptions. In Proceedings of the 11th edition of the Language Resources and Evaluation Conference, Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, Takenobu Tokunaga (eds), 3323–3328. Miazaki: European Language Resources Association (ELRA).Google Scholar
Jewitt, Carey
2015 Multimodal analysis. In The Routledge Handbook of Language and Digital Communication, Alexandra Georgakopoulou & Tereza Spilioti (eds), 69–85. London: Routledge.Google Scholar
Knight, Dawn
2015 e-Language: Communication in the digital age. In Corpora and Discourse Studies, Paul Baker & Tony McEnery (eds), 20–40. London: Palgrave Macmillan. CrossrefGoogle Scholar
Kress, Gunther & van Leeuwen, Theo
2006 Reading Images: The Grammar of Visual Design, 2nd edn. London: Routledge. CrossrefGoogle Scholar
Kuznetsova, Polina, Ordonez, Vicente, Berg, Alexander, Berg, Tamara & Choi, Yejin
2013 Generalizing image captions for image-text parallel corpus. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, Hinrich Schuetze, Pascale Fung & Massimo Poesio (eds), 790–796. Sofia: Association for Computational Linguists (ACL).Google Scholar
Liu, Jeffrey, Weinert, Andrew & Amin, Saurabh
2018 Semantic topic analysis of traffic camera images. In 21st IEEE International Conference on Intelligent Transportation Systems (ITSC), Wei-Bin Zhang, Alexandre M. Bayen, Javier J. Sánchez Medina & Matthew J. Barth (eds), 568–574. Maui: Institute of Electical and Electronics Engineers (IEEE).Google Scholar
McGowan, Michael
2018 ‘Just not blond’: Fake Black Lives Matter Facebook page run by Australian union official – Report, The Guardian. 10 April 2018, https://​www​.theguardian​.com​/us​-news​/2018​/apr​/10​/fake​-black​-lives​-matter​-facebook​-page​-run​-by​-australian​-union​-official​-report (28 November 2019).
Mitchell, William J. T.
1994 Picture Theory: Essays on Verbal and Visual Representation. Chicago IL: University of Chicago Press.Google Scholar
Ordonez, Vicente, Kulkarni, Girish & Berg, Tamara L.
2011 Im2Text: Describing images using 1 million captioned photographs. In Advances in Neural Information Processing Systems, John Shawe-Taylor, Richard Zemel, Peter Bartlett, Fernando Pereira & Kilian Weinberger (eds), 1143–1151. Granada: Neural Information Processing Systems (NIPS).Google Scholar
Pastra, Katerina & Wilks, Yorick
2004 Vision-language integration in AI: A reality check. In Proceedings of the 16th European Conference in Artifical Intelligence, Ramón López de Mántaras & Lorenza Saitta (eds), 937–941. Valencia: IOS Press.Google Scholar
Pew Research Center
2019 State of the Union 2019: How Americans see major national issues. 4 February 2019, https://​www​.pewresearch​.org​/fact​-tank​/2019​/02​/04​/state​-of​-the​-union​-2019​-how​-americans​-see​-major​-national​-issues/ (28 October 2019).
Rayson, Paul
2009 Wmatrix: A web-based corpus processing environment. Computing Department, Lancaster University. http://​ucrel​.lancs​.ac​.uk​/wmatrix/ (5 November 2019).
Rayson, Paul, Archer, Dawn, Piao, Scott & McEnery, Tony
2004 The UCREL semantic analysis system. In Proceedings of the LREC-04 Workshop, beyond Named Entity Recognition Semantic Labelling for NLP Tasks, Lisbon, Portugal, Maria Teresa Lino, Maria Francisca Xavier, Fátima Ferreira, Rute Costa & Raquel Silva (eds), 7–12. Lisbon: European Language Resource Association (ELRA).Google Scholar
Seo, Hyunjin
2014 Visual propaganda in the age of social media: An empirical analysis of Twitter images during the 2012 Israeli–Hamas conflict. Visual Communication Quarterly 21(3): 150–161. CrossrefGoogle Scholar
Twitter
2015 Hearts on Twitter. 3 November 2015, https://​blog​.twitter​.com​/official​/en​_us​/a​/2015​/hearts​-on​-twitter​.html (3 February 2020).
2019a What is a retweet? Twitter Help Center. https://​help​.twitter​.com​/en​/using​-twitter​/retweet​-faqs (3 February 2020).
2019b How to like a tweet. Twitter Help Center. https://​help​.twitter​.com​/en​/using​-twitter​/liking​-tweets​-and​-moments (3 February 2020).
2020 Pricing: API access that scales with you and your solution. https://​developer​.twitter​.com​/en​/pricing (10 February 2020).
United States Department of Justice
2018 Case 1:18-cr-00032-DLF: UNITED STATES OF AMERICA v. INTERNET RESEARCH AGENCY LLC. 16 January 2018, https://​www​.justice​.gov​/file​/1035477​/download (10 December 2019).
United States Joint Chiefs of Staff
2014 Information operations. Homeland Security Digital Library. https://​www​.hsdl​.org​/?view​&did​=759867 (10 December 2019).
Zappavigna, Michele & Martin, James R.
2018 #Communing affiliation: Social tagging as a resource for aligning around values in social media. Discourse, Context & Media 22: 4–12. CrossrefGoogle Scholar
Zappavigna, Michele
2012 Discourse of Twitter and Social Media. London: Continuum.Google Scholar