Cover not available
Article published In:
Pragmatics, Humour and the Internet
Edited by Francisco Yus
[Internet Pragmatics 4:1] 2021
► pp. 150–175

On the order of processing of humorous tweets with visual and verbal elements

| University of Burgos
| Texas A&M University-Commerce
| Kent State University
| Texas A&M University-Commerce
In this paper we examine the order of processing of multimodal tweets (text + image). Using an eye tracker, we collected a sample of 36 participants reading 25 humorous tweets. Our conclusions show that the processing of multimodal humorous tweets is in line with the processing of other multimodal texts. The participants were significantly more likely to start from the image, followed by the caption. Other elements, such as the tweet’s “author” (the user name) or elements outside the tweet’s frame, attracted significantly less and later attention. The participants spent significantly more time gazing at the caption, before moving on to another area. The longer the participants spent looking at the tweet, the less predictable their gaze direction became.
Keywords: Twitter, multimodality, eye tracking, processing order, image, caption
Article outline
  • 1.Introduction
  • 2.Multimodal texts in the eye tracker
  • 3.Attractors
  • 4.Method
    • 4.1Participants
    • 4.2Stimuli
    • 4.3Procedure
  • 5.Results
    • 5.1Descriptive analysis
    • 5.2Inferential analysis
  • 6.Analysis of the results
    • 6.1Summary of multilevel regression modeling
    • 6.2Gaze duration
    • 6.3Conclusion
  • 7.General conclusions
  • Notes
  • References
Published online: 28 August 2020
DOI logo
https://doi.org/10.1075/ip.00060.sim
References
Añaños, Elena, and Anna Astals Serés
2013 “¿Imagen o texto? El poder de captar la atención visual de los elementos gráficos analizado con el Eye tracker [Image or text? The power to capture the visual attention of the graphic elements analyzed with the eye tracker].” Grafica 1(2): 87–98.Google Scholar
Attardo, Salvatore
2020 The Linguistics of Humor: An Introduction. Oxford: Oxford University Press. DOI logoGoogle Scholar
Attardo, Salvatore, and Jean-Charles Chabanne
1992 “Jokes as a text type.” HUMOR: International Journal of Humor Research 5(1–2): 165–176. DOI logoGoogle Scholar
Attardo, Salvatore, Lucy Pickering, Fofo Lomotey, and Shigehito Menjo
2013 “Multimodality in conversational humor.” Review of Cognitive Linguistics 11(2): 402–416. DOI logoGoogle Scholar
Attardo, Salvatore, and Victor Raskin
1991 “Script theory revis(it)ed: Joke similarity and joke representation model.” HUMOR: International Journal of Humor Research 4(3–4): 293–347. DOI logoGoogle Scholar
Boerman, Sophie C., Edith G. Smit, and Lex van Meurs
2011 “Attention battle; the abilities of brand, visual, and text characteristics of the ad to draw attention versus the diverting power of the direct magazine context.” In Advances in Advertising Research, vol.2, ed. by Shintaro Okazaki, 295–310. Wiesbaden: Gabler Verlag. DOI logoGoogle Scholar
Borodo, Michał
2015 “Multimodality, translation and comics.” Perspectives 23(1): 22–41. DOI logoGoogle Scholar
Canestrari, Carla
2018 “The interplay between visual and verbal language in ‘famous last words’ cartoons.” In The Languages of Humor: Verbal, Visual, and Physical Humor, ed. by Arie Sover, 159–166. London: Bloomsbury Academic.Google Scholar
Chiaro, Delia
2005 “Verbally expressed humor and translation: An overview of a neglected field.” HUMOR: International Journal of Humor Research 18(2): 135–145. DOI logoGoogle Scholar
Foulsham, Tom, Dean Wybrow, and Neil Cohn
2016 “Reading without words: Eye movements in the comprehension of comic strips.” Applied Cognitive Psychology 30(4): 566–579. DOI logoGoogle Scholar
Gerhardt, Cornelia
2009 “Multimodal and intertextual humor in the media reception situation.” In Humor in Interaction, ed. by Neal R. Norrick, and Delia Chiaro, 79–100. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Hamrick, Phillip, and Nick B. Pandža
2020 “Contributions of semantic and contextual diversity to the word frequency effect in L2 lexical access.” Canadian Journal of Experimental Psychology 74(1): 25–34. DOI logoGoogle Scholar
Hegarty, Mary
1992 “Mental animation: Inferring motion from static displays of mechanical systems.” Journal of Experimental Psychology: Learning, Memory, and Cognition 18(5): 1084–1102.Google Scholar
Hernández-Méndez, Janet, and Francisco Muñoz-Leiva
2015 “What type of online advertising is most effective for eTourism 2.0? An eye tracking study based on the characteristics of tourists.” Computers in Human Behavior 501: 618–625. DOI logoGoogle Scholar
Kaindl, Klaus
2004 “Multimodality in the translation of humour in comics.” In Perspectives on Multimodality, ed. by Elija Ventola, Cassily Charles, and Martin Kaltenbacher, 173–192. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Kress, Gunther, and Theo Van Leeuwen
2001 Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.Google Scholar
Li, Qian, Zhouwei Huang, and Kiel Christianson
2016 “Visual attention toward tourism photographs with text: An eye-tracking study.” Tourism Management 541: 243–258. DOI logoGoogle Scholar
Liechty, John, Rik Pieters, and Michel Wedel
2003 “Global and local covert visual attention: Evidence from a Bayesian hidden Markov model.” Psychometrika 68(4): 519–541. DOI logoGoogle Scholar
Lohse, Gerald L.
1997 “Consumer eye movement patterns on yellow pages advertising.” Journal of Advertising 26(1): 61–73. DOI logoGoogle Scholar
Pieters, Rik, and Michel Wedel
2004 “Attention capture and transfer in advertising: Brand, pictorial, and text-size effects.” Journal of Marketing 68(2): 36–50. DOI logoGoogle Scholar
2007 “Goal control of attention to advertising: The Yarbus implication.” Journal of Consumer Research 34(2): 224–233. DOI logoGoogle Scholar
R: The R Project for Statistical Computing
The R Foundation.” [URL]
Radach, Ralph, Stefanie Lemmer, Christian Vorstius, Dieter Heller, and Karina Radach
2003 “Eye movements in the processing of print advertisements.” In The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, ed. by Jukka Hyönä, Ralph Radach, and Heiner Deubel, 609–623. Amsterdam: Elsevier. DOI logoGoogle Scholar
Raskin, Victor
1985 Semantics Mechanisms of Humor. Dordrecht: Reidel.Google Scholar
Rayner, Keith, Caren M. Rotello, Andrew J. Stewart, Jessica Keir, and Susan A. Duffy
2001 “Integrating text and pictorial information: Eye movements when looking at print advertisements.” Journal of Experimental Psychology: Applied 7(3): 219–226.Google Scholar
Samson, Andrea C., and Oswald Huber
2007 “The interaction of cartoonist’s gender and formal features of cartoons.” HUMOR: International Journal of Humor Research 20(1): 1–25. DOI logoGoogle Scholar
Simarro Vázquez, María
2016 “Mecanismos de humor verbal en Twitter [Mechanisms of verbal humor on Twitter].” Caracteres: estudios culturales y críticos de la esfera digital 5(2): 32–57.Google Scholar
Tsakona, Villy
2009 “Language and image interaction in cartoons: Towards a multimodal theory of humor.” Journal of Pragmatics 411: 1171–1188. DOI logoGoogle Scholar
Yus, Francisco
2016 Humour and Relevance. Amsterdam: John Benjamins. DOI logoGoogle Scholar
2019a “Multimodality in memes: A cyberpragmatic approach.” In Analyzing Digital Discourse. New Insights and Future Directions, ed. by Patricia Bou-Franch, and Pilar Garcés-Conejos Blitvich, 105–131. Cham: Palgrave Macmillan. DOI logo
2019b Incongruity-Resolution Patterns in Meme Communication. Paper presented at the International Conference on Verbal Humor, University of Alicante, Spain, 23–25 October.
Zanettin, Federico
2010 “Humour in translated cartoons and comics.” In Translation, Humour and the Media, ed. by Delia Chiaro, 34–52. London: Continuum.Google Scholar