On the order of processing of humorous tweets with visual and verbal elements
In this paper we examine the order of processing of multimodal tweets (text + image). Using an eye tracker, we collected a
sample of 36 participants reading 25 humorous tweets. Our conclusions show that the processing of multimodal humorous tweets is in line with
the processing of other multimodal texts. The participants were significantly more likely to start from the image, followed by the caption.
Other elements, such as the tweet’s “author” (the user name) or elements outside the tweet’s frame, attracted significantly less and later
attention. The participants spent significantly more time gazing at the caption, before moving on to another area. The longer the
participants spent looking at the tweet, the less predictable their gaze direction became.
Article outline
- 1.Introduction
- 2.Multimodal texts in the eye tracker
- 3.Attractors
- 4.Method
- 4.1Participants
- 4.2Stimuli
- 4.3Procedure
- 5.Results
- 5.1Descriptive analysis
- 5.2Inferential analysis
- 6.Analysis of the results
- 6.1Summary of multilevel regression modeling
- 6.2Gaze duration
- 6.3Conclusion
- 7.General conclusions
- Notes
-
References
References
Añaños, Elena, and Anna Astals Serés
2013 “
¿Imagen o texto? El poder de captar la atención visual de los elementos gráficos analizado con el Eye tracker [Image or text? The power to capture the visual attention of the graphic elements analyzed with the eye tracker].”
Grafica 1(2): 87–98.
Attardo, Salvatore
2020 The Linguistics of Humor: An Introduction. Oxford: Oxford University Press.
Attardo, Salvatore, and Jean-Charles Chabanne
1992 “
Jokes as a text type.”
HUMOR: International Journal of Humor Research 5(1–2): 165–176.
Attardo, Salvatore, Lucy Pickering, Fofo Lomotey, and Shigehito Menjo
Attardo, Salvatore, and Victor Raskin
1991 “
Script theory revis(it)ed: Joke similarity and joke representation model.”
HUMOR: International Journal of Humor Research 4(3–4): 293–347.
Boerman, Sophie C., Edith G. Smit, and Lex van Meurs
2011 “
Attention battle; the abilities of brand, visual, and text characteristics of the ad to draw attention versus the diverting power of the direct magazine context.” In
Advances in Advertising Research, vol.2, ed. by
Shintaro Okazaki, 295–310. Wiesbaden: Gabler Verlag.
Borodo, Michał
2015 “
Multimodality, translation and comics.”
Perspectives 23(1): 22–41.
Canestrari, Carla
2018 “
The interplay between visual and verbal language in ‘famous last words’ cartoons.” In
The Languages of Humor: Verbal, Visual, and Physical Humor, ed. by
Arie Sover, 159–166. London: Bloomsbury Academic.
Chiaro, Delia
2005 “
Verbally expressed humor and translation: An overview of a neglected field.”
HUMOR: International Journal of Humor Research 18(2): 135–145.
Foulsham, Tom, Dean Wybrow, and Neil Cohn
2016 “
Reading without words: Eye movements in the comprehension of comic strips.”
Applied Cognitive Psychology 30(4): 566–579.
Hamrick, Phillip, and Nick B. Pandža
2020 “
Contributions of semantic and contextual diversity to the word frequency effect in L2 lexical access.”
Canadian Journal of Experimental Psychology 74(1): 25–34.
Hegarty, Mary
1992 “
Mental animation: Inferring motion from static displays of mechanical systems.”
Journal of Experimental Psychology: Learning, Memory, and Cognition 18(5): 1084–1102.
Hernández-Méndez, Janet, and Francisco Muñoz-Leiva
2015 “
What type of online advertising is most effective for eTourism 2.0? An eye tracking study based on the characteristics of tourists.”
Computers in Human Behavior 501: 618–625.
Kress, Gunther, and Theo Van Leeuwen
2001 Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.
Li, Qian, Zhouwei Huang, and Kiel Christianson
2016 “
Visual attention toward tourism photographs with text: An eye-tracking study.”
Tourism Management 541: 243–258.
Liechty, John, Rik Pieters, and Michel Wedel
2003 “
Global and local covert visual attention: Evidence from a Bayesian hidden Markov model.”
Psychometrika 68(4): 519–541.
Lohse, Gerald L.
1997 “
Consumer eye movement patterns on yellow pages advertising.”
Journal of Advertising 26(1): 61–73.
Pieters, Rik, and Michel Wedel
2004 “
Attention capture and transfer in advertising: Brand, pictorial, and text-size effects.”
Journal of Marketing 68(2): 36–50.
Pieters, Rik, and Michel Wedel
2007 “
Goal control of attention to advertising: The Yarbus implication.”
Journal of Consumer Research 34(2): 224–233.
R: The R Project for Statistical Computing
Radach, Ralph, Stefanie Lemmer, Christian Vorstius, Dieter Heller, and Karina Radach
2003 “
Eye movements in the processing of print advertisements.” In
The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, ed. by
Jukka Hyönä,
Ralph Radach, and
Heiner Deubel, 609–623. Amsterdam: Elsevier.
Raskin, Victor
1985 Semantics Mechanisms of Humor. Dordrecht: Reidel.
Rayner, Keith, Caren M. Rotello, Andrew J. Stewart, Jessica Keir, and Susan A. Duffy
2001 “
Integrating text and pictorial information: Eye movements when looking at print advertisements.”
Journal of Experimental Psychology: Applied 7(3): 219–226.
Samson, Andrea C., and Oswald Huber
2007 “
The interaction of cartoonist’s gender and formal features of cartoons.”
HUMOR: International Journal of Humor Research 20(1): 1–25.
Simarro Vázquez, María
2016 “
Mecanismos de humor verbal en Twitter [Mechanisms of verbal humor on Twitter].”
Caracteres: estudios culturales y críticos de la esfera digital 5(2): 32–57.
Tsakona, Villy
2009 “
Language and image interaction in cartoons: Towards a multimodal theory of humor.”
Journal of Pragmatics 411: 1171–1188.
Yus, Francisco
2019a “
Multimodality in memes: A cyberpragmatic approach.” In
Analyzing Digital Discourse. New Insights and Future Directions, ed. by
Patricia Bou-Franch, and
Pilar Garcés-Conejos Blitvich, 105–131. Cham: Palgrave Macmillan.
Yus, Francisco
2019b Incongruity-Resolution Patterns in Meme Communication. Paper presented at the International Conference on Verbal Humor, University of Alicante, Spain, 23–25 October.
Zanettin, Federico
2010 “
Humour in translated cartoons and comics.” In
Translation, Humour and the Media, ed. by
Delia Chiaro, 34–52. London: Continuum.
Cited by
Cited by 3 other publications
Yus, Francisco
2023.
Meme-Mediated Humorous Communication. In
Pragmatics of Internet Humour,
► pp. 245 ff.
Yus, Francisco
2023.
Internet Humour. In
Pragmatics of Internet Humour,
► pp. 59 ff.
This list is based on CrossRef data as of 4 july 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.