Cover not available
Article published In:
Pragmatics, Humour and the Internet
Edited by Francisco Yus
[Internet Pragmatics 4:1] 2021
► pp. 150–175

On the order of processing of humorous tweets with visual and verbal elements

| University of Burgos
| Texas A&M University-Commerce
| Kent State University
| Texas A&M University-Commerce
In this paper we examine the order of processing of multimodal tweets (text + image). Using an eye tracker, we collected a sample of 36 participants reading 25 humorous tweets. Our conclusions show that the processing of multimodal humorous tweets is in line with the processing of other multimodal texts. The participants were significantly more likely to start from the image, followed by the caption. Other elements, such as the tweet’s “author” (the user name) or elements outside the tweet’s frame, attracted significantly less and later attention. The participants spent significantly more time gazing at the caption, before moving on to another area. The longer the participants spent looking at the tweet, the less predictable their gaze direction became.
Keywords: Twitter, multimodality, eye tracking, processing order, image, caption
Article outline
  • 1.Introduction
  • 2.Multimodal texts in the eye tracker
  • 3.Attractors
  • 4.Method
    • 4.1Participants
    • 4.2Stimuli
    • 4.3Procedure
  • 5.Results
    • 5.1Descriptive analysis
    • 5.2Inferential analysis
  • 6.Analysis of the results
    • 6.1Summary of multilevel regression modeling
    • 6.2Gaze duration
    • 6.3Conclusion
  • 7.General conclusions
  • Notes
  • References
Published online: 28 August 2020
DOI logo
https://doi.org/10.1075/ip.00060.sim
References (32)
References
Añaños, Elena, and Anna Astals Serés. 2013. “¿Imagen o texto? El poder de captar la atención visual de los elementos gráficos analizado con el Eye tracker [Image or text? The power to capture the visual attention of the graphic elements analyzed with the eye tracker].” Grafica 1(2): 87–98.Google Scholar
Attardo, Salvatore. 2020. The Linguistics of Humor: An Introduction. Oxford: Oxford University Press. DOI logoGoogle Scholar
Attardo, Salvatore, and Jean-Charles Chabanne. 1992. “Jokes as a text type.” HUMOR: International Journal of Humor Research 5(1–2): 165–176. DOI logoGoogle Scholar
Attardo, Salvatore, Lucy Pickering, Fofo Lomotey, and Shigehito Menjo. 2013. “Multimodality in conversational humor.” Review of Cognitive Linguistics 11(2): 402–416. DOI logoGoogle Scholar
Attardo, Salvatore, and Victor Raskin. 1991. “Script theory revis(it)ed: Joke similarity and joke representation model.” HUMOR: International Journal of Humor Research 4(3–4): 293–347. DOI logoGoogle Scholar
Boerman, Sophie C., Edith G. Smit, and Lex van Meurs. 2011. “Attention battle; the abilities of brand, visual, and text characteristics of the ad to draw attention versus the diverting power of the direct magazine context.” In Advances in Advertising Research, vol.2, ed. by Shintaro Okazaki, 295–310. Wiesbaden: Gabler Verlag. DOI logoGoogle Scholar
Borodo, Michał. 2015. “Multimodality, translation and comics.” Perspectives 23(1): 22–41. DOI logoGoogle Scholar
Canestrari, Carla. 2018. “The interplay between visual and verbal language in ‘famous last words’ cartoons.” In The Languages of Humor: Verbal, Visual, and Physical Humor, ed. by Arie Sover, 159–166. London: Bloomsbury Academic.Google Scholar
Chiaro, Delia. 2005. “Verbally expressed humor and translation: An overview of a neglected field.” HUMOR: International Journal of Humor Research 18(2): 135–145. DOI logoGoogle Scholar
Foulsham, Tom, Dean Wybrow, and Neil Cohn. 2016. “Reading without words: Eye movements in the comprehension of comic strips.” Applied Cognitive Psychology 30(4): 566–579. DOI logoGoogle Scholar
Gerhardt, Cornelia. 2009. “Multimodal and intertextual humor in the media reception situation.” In Humor in Interaction, ed. by Neal R. Norrick, and Delia Chiaro, 79–100. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Hamrick, Phillip, and Nick B. Pandža. 2020. “Contributions of semantic and contextual diversity to the word frequency effect in L2 lexical access.” Canadian Journal of Experimental Psychology 74(1): 25–34. DOI logoGoogle Scholar
Hegarty, Mary. 1992. “Mental animation: Inferring motion from static displays of mechanical systems.” Journal of Experimental Psychology: Learning, Memory, and Cognition 18(5): 1084–1102.Google Scholar
Hernández-Méndez, Janet, and Francisco Muñoz-Leiva. 2015. “What type of online advertising is most effective for eTourism 2.0? An eye tracking study based on the characteristics of tourists.” Computers in Human Behavior 501: 618–625. DOI logoGoogle Scholar
Kaindl, Klaus. 2004. “Multimodality in the translation of humour in comics.” In Perspectives on Multimodality, ed. by Elija Ventola, Cassily Charles, and Martin Kaltenbacher, 173–192. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Kress, Gunther, and Theo Van Leeuwen. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.Google Scholar
Li, Qian, Zhouwei Huang, and Kiel Christianson. 2016. “Visual attention toward tourism photographs with text: An eye-tracking study.” Tourism Management 541: 243–258. DOI logoGoogle Scholar
Liechty, John, Rik Pieters, and Michel Wedel. 2003. “Global and local covert visual attention: Evidence from a Bayesian hidden Markov model.” Psychometrika 68(4): 519–541. DOI logoGoogle Scholar
Lohse, Gerald L. 1997. “Consumer eye movement patterns on yellow pages advertising.” Journal of Advertising 26(1): 61–73. DOI logoGoogle Scholar
Pieters, Rik, and Michel Wedel. 2004. “Attention capture and transfer in advertising: Brand, pictorial, and text-size effects.” Journal of Marketing 68(2): 36–50. DOI logoGoogle Scholar
. 2007. “Goal control of attention to advertising: The Yarbus implication.” Journal of Consumer Research 34(2): 224–233. DOI logoGoogle Scholar
R: The R Project for Statistical Computing. “The R Foundation.” [URL]
Radach, Ralph, Stefanie Lemmer, Christian Vorstius, Dieter Heller, and Karina Radach. 2003. “Eye movements in the processing of print advertisements.” In The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, ed. by Jukka Hyönä, Ralph Radach, and Heiner Deubel, 609–623. Amsterdam: Elsevier. DOI logoGoogle Scholar
Raskin, Victor. 1985. Semantics Mechanisms of Humor. Dordrecht: Reidel.Google Scholar
Rayner, Keith, Caren M. Rotello, Andrew J. Stewart, Jessica Keir, and Susan A. Duffy. 2001. “Integrating text and pictorial information: Eye movements when looking at print advertisements.” Journal of Experimental Psychology: Applied 7(3): 219–226.Google Scholar
Samson, Andrea C., and Oswald Huber. 2007. “The interaction of cartoonist’s gender and formal features of cartoons.” HUMOR: International Journal of Humor Research 20(1): 1–25. DOI logoGoogle Scholar
Simarro Vázquez, María. 2016. “Mecanismos de humor verbal en Twitter [Mechanisms of verbal humor on Twitter].” Caracteres: estudios culturales y críticos de la esfera digital 5(2): 32–57.Google Scholar
Tsakona, Villy. 2009. “Language and image interaction in cartoons: Towards a multimodal theory of humor.” Journal of Pragmatics 411: 1171–1188. DOI logoGoogle Scholar
Yus, Francisco. 2016. Humour and Relevance. Amsterdam: John Benjamins. DOI logoGoogle Scholar
. 2019a. “Multimodality in memes: A cyberpragmatic approach.” In Analyzing Digital Discourse. New Insights and Future Directions, ed. by Patricia Bou-Franch, and Pilar Garcés-Conejos Blitvich, 105–131. Cham: Palgrave Macmillan. DOI logo
. 2019b. Incongruity-Resolution Patterns in Meme Communication. Paper presented at the International Conference on Verbal Humor, University of Alicante, Spain, 23–25 October.
Zanettin, Federico. 2010. “Humour in translated cartoons and comics.” In Translation, Humour and the Media, ed. by Delia Chiaro, 34–52. London: Continuum.Google Scholar
Cited by (4)

Cited by four other publications

Geise, Stephanie & Yi Xu
2024. Effects of Visual Framing in Multimodal Media Environments: A Systematic Review of Studies Between 1979 and 2023. Journalism & Mass Communication Quarterly DOI logo
Yus, Francisco
2023. Meme-Mediated Humorous Communication. In Pragmatics of Internet Humour, ► pp. 245 ff. DOI logo
Yus, Francisco
2023. Internet Humour. In Pragmatics of Internet Humour, ► pp. 59 ff. DOI logo
[no author supplied]
2022. The Multimodal Performance of Conversational Humor [Figurative Thought and Language, 13], DOI logo

This list is based on CrossRef data as of 5 august 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.