1.Introduction

The emergence and rapid spread of computer-mediated communication (CMC) in the last few decades has enabled new ways of expressing emotions, including the use of traditional keyboard-based symbols such as emoticons (e.g. :D for a smile), the use of asterisks (e.g. *joy*), acronyms (e.g. LOL), and typographical emphasis, such as the use of uppercase and bold. Others ways of expressing emotions include a visual presentation through digital pictograms or graphic images, such as emoji, stickers or GIFs. In this study, I will focus on emoji and reaction GIFs.

With their increasing prevalence, there is a rich body of work on the meaning of emoji11.Emoji may be facial or non-facial. In this study, I only look at facial emoji. (see Bai et al. 2019Bai, Qiyu, Qi Dan, Zhe Mu, and Maokun Yang 2019 “A Systematic Review of Emoji: Current Research and Future Perspectives.” Frontiers in Psychology 10: 2221. for a full systematic review). Emoji are generally considered “to be a substitute for facial expressions, gestures, and other nonverbal cues” (Scott 2022Scott, Kate 2022 Pragmatics Online. London: Routledge. , 89). Scholars often focus on the semantic properties or encoded meanings of emoji, their paralinguistic function, and pragmatic functions such as communication of speech act information and visual rhetoric (see Jibril and Abdullah 2013Jibril, Ahmed Tanimu, and Mardziah Haytati Abdullah 2013 “Relevance of Emoticons in Computer-Mediated Communication Contexts: An Overview.” Asian Social Science 9: 201. ; López and Cap 2017López, Rebecca Padilla, and Fabienne Cap 2017 “Did You Ever Read about Frogs Drinking Coffee? Investigating the Compositionality of Multi-Emoji Expressions.” Paper Presented at the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis . ; Khandekar et al. 2019Sujay, Khandekar, Chae Won Ryu, Joseph Higgs, Jerry O. Talton, Yuanzhe Bian, and Ranjitha Kumar 2019 “Opico: A Study of Emoji-First Communication in a Mobile Social App.” Paper presented at the Companion of the World Wide Web Conference . ). Other aspects of emoji that have attracted scholarly attention include their emotional and linguistic functions and attributes (Danesi 2017Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London: Bloomsbury. ; Riordan 2017Riordan, Monica A. 2017 “Emojis as Tools for Emotion Work: Communicating Affect in Text Messages.” Journal of Language and Social Psychology 36: 549–567. ), factors that influence users’ preferences, such as individual characteristics (see Tossell et al. 2012Tossell, Chad C., Philip Kortum, Clayton Shepard, Laura H. Barg-Walkow, Ahmad Rahmati, and Lin Zhong 2012 “A Longitudinal Study of Emoticon Use in Text Messaging from Smartphones.” Computer in Human Behavior 28: 659–663. ; Hall and Pennington 2013Hall, Jeffrey. A., and Natalie Pennington 2013 “Self-Monitoring, Honesty, and Cue Use on Facebook: The Relationship with User Extraversion and Conscientiousness.” Computers in Human Behavior 29: 1556–1564. ; Settanni and Marengo 2015Settanni, Michele, and Davide Marengo 2015 “Sharing Feelings Online: Studying Emotional Well-Being Via Automated Text Analysis of Facebook Posts.” Frontiers in Psychology 6: 1045. ; Prada et al. 2018Prada, Marília, David L. Rodrigues, Margarida V. Garrido, Diniz Lopes, Bernardo Cavalheiro, and Rui Gaspar 2018 “Motives, Frequency and Attitudes toward Emoji and Emoticon Use.” Telematics and Informatics 35: 1925–1934. ), cultural background or system platform (see Gülşen 2016Gülşen, Tüge. T. 2016 “You Tell Me in Emojis.” In Computational and Cognitive Approaches to Narratology, ed. by T. Ogata and T. Akimoto, 354–375. IGI Global. ; Cheng 2017Cheng, Lifen 2017 “Do I Mean What I Say and Say What I Mean? A Cross-Cultural Approach to the Use of Emoticons & Emojis in CMC Messages.” FONSECA: Journal of Communication 15: 199–217. ; Jaeger and Ares 2017Jaeger, Sara R., and Gastón Ares 2017 “Dominant Meanings of Facial Emoji: Insights from Chinese Consumers and Comparison with Meanings from Internet Resources.” Food Quality and Preference 62: 275–283. ; López and Cap 2017López, Rebecca Padilla, and Fabienne Cap 2017 “Did You Ever Read about Frogs Drinking Coffee? Investigating the Compositionality of Multi-Emoji Expressions.” Paper Presented at the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis . ; Hjartstrom et al. 2019Hjartstrom, Hanna, Daniel E. Sörman, and Jessica K. Ljungberg 2019 “Distraction and Facilitation: The Impact of Emotional Sounds in an Emoji Oddball Task.” PsyCh Journal 8: 180–186. ; Jaeger et al. 2019Jaeger, Sara R., Christina M. Roigard, David Jin, Leticia Vidal, and Gastón Ares 2019 “Valence, Arousal and Sentiment Meanings of 33 Facial Emoji: Insights for the Use of Emoji in Consumer Research.” Food Research International. 119: 895–907. ; Sadiq et al. 2019Sadiq, M., and Shahida 2019 “Learning Pakistani Culture through the Namaz Emoji.” 2nd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET): 1–8, ). Emoji are often considered to fill a gap and provide non-verbal cues in CMC, where more traditional non-verbal cues, such as facial expressions or gestures, are not available. As a result, scholars (e.g. Gülşen 2016Gülşen, Tüge. T. 2016 “You Tell Me in Emojis.” In Computational and Cognitive Approaches to Narratology, ed. by T. Ogata and T. Akimoto, 354–375. IGI Global. ; Gibson et al. 2018Gibson, Will, Pingping Huang, and Qianyun Yu 2018 “Emoji and Communicative Action: The Semiotics, Sequence and Gestural Actions of ‘Face Covering Hand’.” Discourse, Context & Media 26: 91–99. ) consider that emoji play an auxiliary role and promote interpersonal communication.

In contrast, GIFs, and reaction GIFs in particular,22.GIFs are generally a short, continuous moving/animated image taken from a longer video, often (but not always) from famous TV programmes or films. Reaction GIFs are a particular use of GIFs, where the communicator sends a GIF as a reply to a text message or a social media posting. have attracted less scholarly attention and, to the best of the author’s knowledge, there is little work from the perspective of pragmatics on reaction GIFs. However, scholars in media studies have discussed their function in communicating emotions as well as demonstrating cultural knowledge (e.g. Veszelszki 2015Veszelszki, Agnes 2015 “Emoticons vs. Reaction-Gifs. Non-Verbal Communication on the Internet from the Aspects of Visuality, Verbality and Time.” In Beyond Words. Pictures, Parables, Paradoxes, Visual Learning series, vol. 5, ed. by András Benedek, and Kristóf Nyíri, 131–145. Frankfurt: Peter Lang.; Tolins and Samermit 2016Tolins, Jackson, and Patrawat Samermit 2016 “Gifs as Embodied Enactments in Text-Mediated Conversation.” Research on Language and Social Interaction 49: 75–91. ; Miltner and Highfield 2017Miltner, Kate M., and Tim Highfield 2017 “Never Gonna GIF You Up: Analyzing the Cultural Significance of the Animated GIF.” Social Media and Society 3.3: 1–11. ). Miltner and Highfield (2017Miltner, Kate M., and Tim Highfield 2017 “Never Gonna GIF You Up: Analyzing the Cultural Significance of the Animated GIF.” Social Media and Society 3.3: 1–11. , 5) argue that by singling out and thus emphasising a particular scene, reaction GIFs can “act as a proxy for, or expression of, emotion and/or affect”.

While these studies provide a rich description of the uses of emoji or GIFs, it is not entirely clear how the addressee/recipient processes and interprets these visual cues. Moreover, despite the similarities in functions between emoji and reaction GIFs, they are often analysed separately, and little work has been done in pragmatics to account for their role in the communication of emotion, and how recipients process such visually oriented ostensive stimuli. Against this backdrop, this study aims to explain the role of emoji and GIFs from the perspective of relevance theory. In particular, this study examines the role of facial emoji and reaction GIFs in terms of perceptual resemblance and the showing–meaning continuum, with a reference to the relevance-theoretic analyses of metaphor and irony. As discussed in Section 3, Deirdre Wilson has done extensive work on metaphor and irony33.See Sperber and Wilson (1986/1995Sperber, Dan, and Deirdre Wilson 1986/1995 Relevance: Communication and Cognition, 2nd ed. Oxford: Blackwell., 1987Sperber, Dan, and Deirdre Wilson 1987 “Précis of Relevance: Communication and Cognition.” Behavioral and Brain Sciences 10: 697–754. , 1998Sperber, Dan, and Deirdre Wilson 1998 “Irony and Relevance: A Reply to Seto, Hamamoto and Yamanashi.” In Relevance Theory. Applications and Implications. ed. by R. Carston, and S. Uchida, 283–293. Amsterdam: John Benjamins. ), Wilson and Sperber (2002Wilson, Deirdre, and Dan Sperber 2002 “Truthfulness and Relevance.” Mind 111 (443): 583–632. , 2004Wilson, Deirdre, and Dan Sperber 2004 “Relevance Theory.” In The Handbook of Pragmatics, ed. by Laurence R. Horn, and Gregory Ward, 607–632. Oxford: Blackwell.), and Wilson and Carston (2006)Wilson, Deirdre, and Robyn Carston 2006 “Metaphor, Relevance and the ‘Emergent Property’ Issue.” Mind & Language 21 (3): 404–433. for fuller accounts of metaphor and irony by Deirdre Wilson. and has shown that both metaphor and irony involve an interpretive relation between the propositional form and some thoughts. Drawing on this valuable contribution to the debate on metaphor and irony, I argue that emoji and GIFs involve the showing aspect of communication, providing a clue to ostension by virtue of perceptual resemblance between what they represent and the communicator’s emotional state. I further argue that both emoji and GIFs can involve echoic use of language, which leads to the communication of their attitude.

In the next section, I present an overview of previous studies. In Section 3, I introduce relevance theory and notions that are relevant to this study. In Section 4, I show how ideas in relevance theory can be used to account for the role of emoji and GIFs. The conclusion is presented in Section 5.

2.Previous studies

2.1The semantics and pragmatics of emoji

As Bai et al. (2019)Bai, Qiyu, Qi Dan, Zhe Mu, and Maokun Yang 2019 “A Systematic Review of Emoji: Current Research and Future Perspectives.” Frontiers in Psychology 10: 2221. show, emoji have attracted huge scholarly attention in many disciplines, including linguistics, marketing, computing, behavioural science, and communication studies. It is generally considered that emoji work in a similar manner to non-verbal cues such as facial expressions and gestures in face-to-face communication and that they convey emotions. As they allow the inclusion in computer-mediated communication of what is often conveyed via non-verbal cues in face-to-face communication, scholars such as Gülşen (2016)Gülşen, Tüge. T. 2016 “You Tell Me in Emojis.” In Computational and Cognitive Approaches to Narratology, ed. by T. Ogata and T. Akimoto, 354–375. IGI Global. and Gibson et al. (2018)Gibson, Will, Pingping Huang, and Qianyun Yu 2018 “Emoji and Communicative Action: The Semiotics, Sequence and Gestural Actions of ‘Face Covering Hand’.” Discourse, Context & Media 26: 91–99. argue that the use of emoji promotes interaction and interpersonal relationships. In linguistics, scholars are often concerned with the semantics and pragmatics of emoji: the types of emotions emoji can signal and the emotional tone emoji can add to communication.

According to Danesi (2017)Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London: Bloomsbury. , who provides a comprehensive overview of the linguistic analysis of emoji, emoji have two important functions, a phatic function and an emotive function. They function to “add emotional tone and to emphasise certain phatic aspects of communication” (Danesi 2017Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London: Bloomsbury. , 100). In particular, Danesi (2017)Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London: Bloomsbury. analyses the semantics of emoji in the framework of Goffman’s notion of framing, which is defined as the presentation of concepts from a particular perspective so that they can be ‘framed’ through the form used (ibid., 56). On this view, emoji convey a frame of mind, or perspective, within which the message is interpreted. Danesi (2017)Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London: Bloomsbury. also argues that emoji are fundamentally metaphorical pictures, where “separate domains of meaning are blended to produce new forms of meaning that amalgamate the various referential domains into one image” (ibid., 66), and each emoji is “a manifestation of a CM [Conceptual Metaphor] in visual form” (ibid., 70). According to Danesi (2017)Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London: Bloomsbury. , ‘blending’ is the fundamental aspect of emoji semantics and assumed to be universal. He demonstrates this using a range of examples including facial emoji, logographs, and a snake emoji.

For Danesi (2017)Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London: Bloomsbury. , pragmatics is about communicative competence (see Hymes 1971Hymes, Dell 1971 On Communicative Competence. Philadelphia: University of Pennsylvania Press.), which is concerned with how to use language. He analyses 323 texts in terms of pragmatic functions and argues that the most important pragmatic functions of emoji are to add tone and to inject a positive mood (Danesi 2017Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London: Bloomsbury. , 95–96). Following Collister (2015)Collister, Lauren 2015 “Textspeak is Modernizing the English Language.” New Republic. https://​newrepublic​.com​/article​/121463​/textspeak​-streamlining​-language​-not​-ruining​-it, Danesi argues that the use of emoji enables the communicator to minimise potential misunderstanding and threat. For example, a smiley emoji could soften a message that is potentially conflictive. Another basic pragmatic function of emoji, according to Danesi (2017)Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London: Bloomsbury. , is salutation, or the opening and ending of messages. He argues that salutation comes under the phatic function, as defined by Roman Jakobson (1960)Jakobson, Roman 1960 “Linguistics and Poetics.” In Style in Language, ed. by Thomas A. Sebeok, 34–45. Cambridge, MA: MIT Press., which is “designed to establish, maintain or assuage social contact” (Danesi 2017Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London: Bloomsbury. , 101). He also shows how emoji can be used as punctuation or as visual discourse particles that “reinforce the various emotional states or moods” (ibid., 107).

While Danesi’s (2017)Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London: Bloomsbury. work, which represents a dominant approach to emoji meaning, provides a rich description of emoji functions, it is not entirely clear how recipients process emoji used independently of, or in conjunction with, other verbal and non-verbal input. It is obvious that emoji indeed have some metaphorical function in that they represent some thoughts from a different domain, and that the communicator can use emoji for salutation or punctuation. However, the question is, how would a recipient choose one interpretation over another, and how would the recipient process or interpret emoji independently or in conjunction with other verbal and non-verbal input?

Another aspect arising from Danesi’s work that needs further discussion is the semantics and pragmatics of emoji. For Danesi (2017)Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London: Bloomsbury. , the role of pragmatics is limited to communicative competence, or ‘how to use language’, which appears to mean discourse management or interpersonal functions. However, as widely acknowledged in relevance theory, pragmatics goes far beyond interpersonal functions. What has been considered a strictly semantic matter, for example, the recovery of explicit (or linguistically encoded) meaning, has been shown to involve inference and therefore is a matter of pragmatics.44.See a series of works by Robyn Carston (1999Carston, Robyn 1999 “The Semantics/Pragmatics Distinction: A View from Relevance Theory.” In The Semantics/Pragmatics Interface from Different Points of View, ed. by Ken Turner, 85–125. Amsterdam: Elsevier., 2002Carston, Robyn 2002 Thoughts and Utterances. Oxford: Blackwell. , 2004Carston, Robyn 2004 “Explicature and Semantics.” In Semantics: A Reader, ed. by S. Davis, and B. Gillon, 1–44. Oxford: Oxford University Press., 2008Carston, Robyn 2008 “Linguistic Communication and the Semantics/Pragmatics Distinctions.” Synthèse 165 (3): 321–345. , 2010Carston, Robyn 2010 “Metaphor: Ad Hoc Concepts, Literal Meaning and Mental Images.” Proceedings of the Aristotelian Society 110 (3): 295–321. ) on linguistic underdeterminacy, where she discusses how the encoded content of an utterance underdetermines the proposition expressed. That is, the intended meaning is always richer than what is encoded in the utterance. In relation to emoji, interpreting metaphor necessarily involves inference and hence is a matter of pragmatics. In contrast, semantics is a matter of coding. Metaphor is not coded. In a similar way, an emoji of a face with rolling eyes does not encode the attitude of condescension. An emoji of a face with fingers across its chin as if stroking it does not encode the act of consideration.55.Some scholars argue that some type of visual representation could be coded (see, for example, Forceville 2020Forceville, Charles 2020 Visual and Multimodal Communication: Applying the Relevance Principle. Oxford: Oxford University Press. ). I will return to this point in Section 3. Of course, these emoji trigger the recovery of these interpretations, but such processes necessarily involve inference and hence are a matter of pragmatics. If it is a matter of pragmatics, then, we must explain how the use of emoji guides the recipient to recover the intended interpretation. This is particularly important, as understanding emoji’s role in the communication and interpretation process will enable us to provide an explanatory account of the role of emoji alongside other communicative tools.

2.2A relevance-theoretic account of emoji

Yus (2019Yus, Francisco 2019 “Emoji: A Full Cyberpragmatic Approach.” Paper presented at the 16th China Pragmatics Conference . Nanchang, China., 2021Yus, Francisco 2021 Smartphone Communication: Interactions in the App Ecosystem. London: Routledge. ) analyses various uses of emoji in terms of relevance theory and shows the range of information that emoji can be used to communicate. Yus (2019Yus, Francisco 2019 “Emoji: A Full Cyberpragmatic Approach.” Paper presented at the 16th China Pragmatics Conference . Nanchang, China., 2021Yus, Francisco 2021 Smartphone Communication: Interactions in the App Ecosystem. London: Routledge. ) argues that emoji are not just a matter of coding and that the interpretation of emoji involves inference. He also emphasises the role of context in achieving a full understanding of emoji meaning, as there is a substantial gap between the emoji’s form and the intended interpretation. Most importantly, unlike other scholars who argue that emoji play an auxiliary role, Yus (2019Yus, Francisco 2019 “Emoji: A Full Cyberpragmatic Approach.” Paper presented at the 16th China Pragmatics Conference . Nanchang, China., 2021Yus, Francisco 2021 Smartphone Communication: Interactions in the App Ecosystem. London: Routledge. ) argues that emoji are essential to the recovering of the intended interpretation of the accompanying text as well as other aspects of the discourse in which emoji are used.

Yus (2019)Yus, Francisco 2019 “Emoji: A Full Cyberpragmatic Approach.” Paper presented at the 16th China Pragmatics Conference . Nanchang, China. shows how emoji can be used (i) to guide the interpretation of the texts which they accompany (emoji within), (ii) independently of other input (emoji without), and (iii) in parallel with the accompanying text and modifying the communicative act itself rather than the verbal input (emoji beyond). For each case, Yus (2019)Yus, Francisco 2019 “Emoji: A Full Cyberpragmatic Approach.” Paper presented at the 16th China Pragmatics Conference . Nanchang, China. presents a range of uses, including providing constituents of the proposition expressed, communicating propositional attitude, conveying emotions, and discourse management/interpersonal use.

Yus’ (2019)Yus, Francisco 2019 “Emoji: A Full Cyberpragmatic Approach.” Paper presented at the 16th China Pragmatics Conference . Nanchang, China. description of emoji’s role and classification offers a rich insight into the use of emoji from the perspective of relevance theory. Unlike most studies that focus solely on the classification of emoji use, Yus’ work acknowledges the role of inference and context in interpreting emoji. Drawing on his claims that interpreting emoji necessarily involves inference and that context plays an important role in this, this study provides an analysis of such inferential processes, rather than a description of the different functions of emoji. Furthermore, I also compare emoji with reaction GIFs, which are also visual-based communicative inputs in digital communication.

2.3Reaction GIFs

‘GIF’ is short for Graphic Interchange Format. GIFs are a very popular tool used in social media posts or in the comment sections of social media sites to express (often emotive) reactions. GIFs allow and play a role in the communication of affect and the demonstration of cultural knowledge, the two core aspects of digital communication (Tolins and Samermit 2016Tolins, Jackson, and Patrawat Samermit 2016 “Gifs as Embodied Enactments in Text-Mediated Conversation.” Research on Language and Social Interaction 49: 75–91. ; Miltner and Highfield 2017Miltner, Kate M., and Tim Highfield 2017 “Never Gonna GIF You Up: Analyzing the Cultural Significance of the Animated GIF.” Social Media and Society 3.3: 1–11. ). GIFs consist of “the endless looping of image sequences” (Miltner and Highfield 2017Miltner, Kate M., and Tim Highfield 2017 “Never Gonna GIF You Up: Analyzing the Cultural Significance of the Animated GIF.” Social Media and Society 3.3: 1–11. , 2), often an isolated snippet of well-known moving images. As Miltner and Highfield (2017Miltner, Kate M., and Tim Highfield 2017 “Never Gonna GIF You Up: Analyzing the Cultural Significance of the Animated GIF.” Social Media and Society 3.3: 1–11. , 5) explain, GIFs, especially reaction GIFs, place “attention on a single visual action, feeling or response”. By putting the focus on one single element, reaction GIFs can “act as a proxy for, or expression of, emotion and/or affect” (ibid.).

Scholars also discuss the polysemic nature of GIFs (Tolins and Samermit 2016Tolins, Jackson, and Patrawat Samermit 2016 “Gifs as Embodied Enactments in Text-Mediated Conversation.” Research on Language and Social Interaction 49: 75–91. ; Miltner and Highfield 2017Miltner, Kate M., and Tim Highfield 2017 “Never Gonna GIF You Up: Analyzing the Cultural Significance of the Animated GIF.” Social Media and Society 3.3: 1–11. ). Miltner and Highfield (2017Miltner, Kate M., and Tim Highfield 2017 “Never Gonna GIF You Up: Analyzing the Cultural Significance of the Animated GIF.” Social Media and Society 3.3: 1–11. , 2) note: “One of the main communicative advantages of the GIF is that it is polysemic, offering different meanings and interpretations to different audiences. This also encourages the diverse contexts in which a specific GIF may be employed”. Furthermore, drawing on Katz and Shifman’s (2017Katz, Yuval, and Limor Shifman 2017 “Making Sense? The Structure and Meanings Of Digital Memetic Nonsense.” Information, Communication & Society 20: 825–842. , 827) argument that “polysemy enhances the popular appeal of texts”, Miltner and Highfield (2017Miltner, Kate M., and Tim Highfield 2017 “Never Gonna GIF You Up: Analyzing the Cultural Significance of the Animated GIF.” Social Media and Society 3.3: 1–11. , 4–5) argue that a GIF, being polysemic, has the potential to appeal to a range of audiences and settings, as it offers “different meanings and interpretations to different audiences”. For example, by depicting an everyday experience, a GIF could communicate a range of emotions. However, to those who can recognise the context, there may be further layers of meaning. Miltner and Highfield (2017Miltner, Kate M., and Tim Highfield 2017 “Never Gonna GIF You Up: Analyzing the Cultural Significance of the Animated GIF.” Social Media and Society 3.3: 1–11. , 9) explain: “the format’s polysemy and affective capacity afford users with the opportunity to provide heightened and layered communication, demonstrate cultural knowledge, and occasionally engage in displays of resistance to certain ideologies and actors”. This polysemic nature is also linked to how GIFs are created. As mentioned previously, GIFs are often a snippet of a famous scene, which makes them a quotation in essence (see Tolins and Samermit 2016Tolins, Jackson, and Patrawat Samermit 2016 “Gifs as Embodied Enactments in Text-Mediated Conversation.” Research on Language and Social Interaction 49: 75–91. ; Miltner and Highfield 2017Miltner, Kate M., and Tim Highfield 2017 “Never Gonna GIF You Up: Analyzing the Cultural Significance of the Animated GIF.” Social Media and Society 3.3: 1–11. ). However, by taking it out of context, the communicator can apply it to any other context, regardless of the audience’s awareness of the original context. They can also add individual commentary or reaction (see Tolins and Samermit 2016Tolins, Jackson, and Patrawat Samermit 2016 “Gifs as Embodied Enactments in Text-Mediated Conversation.” Research on Language and Social Interaction 49: 75–91. ; Miltner and Highfield 2017Miltner, Kate M., and Tim Highfield 2017 “Never Gonna GIF You Up: Analyzing the Cultural Significance of the Animated GIF.” Social Media and Society 3.3: 1–11. ), which adds to the culturally specific nature of GIFs. For this, Sha (2016)Sha 2016 “The Digital Materiality Of Gifs.” Retrieved from http://​digitalmateriality​.com/ argues that GIFs are “a visual language unto themselves, an emotive vocabulary made out of culture”.

Unlike emoji, there are very few studies on GIFs, especially in pragmatics. Most studies are concerned with their history and development (e.g. Eppink 2014Eppink, Jason 2014 “A Brief History of the Gif (So Far).” Journal of Visual Culture 13: 298–306. ), cultural significance (e.g. Miltner and Highfield 2017Miltner, Kate M., and Tim Highfield 2017 “Never Gonna GIF You Up: Analyzing the Cultural Significance of the Animated GIF.” Social Media and Society 3.3: 1–11. ) or communicative affordances (Eppink 2014Eppink, Jason 2014 “A Brief History of the Gif (So Far).” Journal of Visual Culture 13: 298–306. ; Ash 2015Ash, James, and Stephen Wilhite 2015 “Sensation, Networks, and the GIF: Toward an Allotropic Account of Affect.” In Networked Affect, ed. by Ken Hillis, Susanna Paasonen, and Michael Petit, 119–133. Cambridge, MA: MIT Press. ; Cho 2015Cho, Alexander 2015 “Queer Reverb: Tumblr, Affect, Time.” In Networked Affect, ed. by Ken Hillis, Susanna Paasonen, and Michael Petit, 43–58. Cambridge, MA: MIT Press. ; Paasonen et al. 2015Paasonen, Susanna, Ken Hillis, and Michael Petit 2015 “Introduction: Networks of Transmission: Intensity, Sensation, Value.” In Networked affect, ed. by Ken Hillis, Susanna Paasonen, and Michael Petit, 1–24. Cambridge: The MIT Press. , to name but a few), and little is known about how viewers process such culturally specified visual input regardless of their awareness of the original context. The question for pragmatics, therefore, is how viewers interpret GIFs, and how GIFs contribute to communication.

As I have shown, while the literature provides a rich description of the uses, functions, and affordances of emoji and GIFs, there is still room for analysis, particularly from the perspective of pragmatics, as this would offer insight into how readers/recipients process or interpret GIFs and emoji, and how GIFs and emoji contribute to communication. Furthermore, to the best of the author’s knowledge, there is little research into the similarities between the two from the perspective of relevance theory. Do these visually presented communicative inputs contribute to communication in a manner which is different from ordinary communicative acts? Or is the use, and the interpretation, of GIFs and emoji just a standard human behaviour? To answer these questions, I will discuss the role of emoji and GIFs in communication in terms of the relevance-theoretic notions of perceptual resemblance and the showing–meaning continuum and argue that the underlying processing for emoji and GIFs is the same as with any other ostensive stimulus in ordinary communication, in that it triggers the relevance-theoretic comprehension procedure, and they do not need to be treated as a special case. In the next section, I first present notions from relevance theory, and then suggest how ideas from the relevance-theoretic framework enable us to explain the role of emoji and GIFs in communication.

3.Relevance theory

Relevance theory, proposed and developed by Sperber and Wilson (1986/1995)Sperber, Dan, and Deirdre Wilson 1986/1995 Relevance: Communication and Cognition, 2nd ed. Oxford: Blackwell., is a cognitively grounded theory of communication with the notion of relevance at its core. Relevance is defined in terms of an interaction between information and the audience’s cognitive environment:66.In relevance theory, the cognitive environment is defined as “the set of assumptions that are manifest to an individual at a given time” (Carston 2002Carston, Robyn 2002 Thoughts and Utterances. Oxford: Blackwell. , 376). One might not be entertaining all assumptions at that time, but these assumptions are still manifest (or the individual is capable of representing it mentally) to the individual. “Information is relevant to you if it interacts in a certain way with your existing assumptions about the world” (Sperber and Wilson 1987Sperber, Dan, and Deirdre Wilson 1987 “Précis of Relevance: Communication and Cognition.” Behavioral and Brain Sciences 10: 697–754. , 41–42). This can be further explained in terms of processing effort and contextual effects.

Relevance of an input to an individual

1. Other things being equal, the greater the positive cognitive effects achieved by processing an input, the greater the relevance of the input to the individual at that time.

2. Other things being equal, the greater the processing effort expended, the lower the relevance of the input to the individual at that time. (Wilson and Sperber 2004Wilson, Deirdre, and Dan Sperber 2004 “Relevance Theory.” In The Handbook of Pragmatics, ed. by Laurence R. Horn, and Gregory Ward, 607–632. Oxford: Blackwell., 609)

Contextual effects include (i) contextual implication, where new information interacts with existing assumptions to create a new assumption, (ii) contextual strengthening, where new information interacts with existing assumptions and strengthens them, and (iii) contextual contradiction and elimination, where new information contradicts existing assumptions and eliminates them. For example, suppose that Mary thought she would do gardening the following day if the weather was nice. She wakes up in the morning and sees that it is raining. In this context, Mary would conclude that she cannot do gardening that day. This is an example of contextual implication. Similarly, suppose Mary thought it was raining. She opens the curtain and confirm it is indeed raining. This is an example of contextual strengthening. Finally, suppose Mary thought she could hear the sound of rain on the window. However, upon opening her curtains, she sees her neighbour watering plants. In this case, the new information about where the water is coming from (the neighbour) contradicts her existing assumption that it is raining, and this existing assumption will be eliminated. This is an example of contextual contradiction and elimination.

Relevance theory is centred around two principles: the cognitive principle of relevance and the communicative principle of relevance. The first, the cognitive principle of relevance, describes how human cognition is designed: “Human cognition tends to be geared to the maximisation of relevance” (Sperber and Wilson 1986/1995Sperber, Dan, and Deirdre Wilson 1986/1995 Relevance: Communication and Cognition, 2nd ed. Oxford: Blackwell., 260). That is, humans tend to pay attention to whatever might offer a reward, or the cognitive effects which justify the processing effort required. This suggests that if someone tries to draw your attention, then you would expect that your attention will be rewarded. This is captured in the communicative principle of relevance: “Every ostensive stimulus conveys a presumption of its own optimal relevance” (Sperber and Wilson 1986/1995Sperber, Dan, and Deirdre Wilson 1986/1995 Relevance: Communication and Cognition, 2nd ed. Oxford: Blackwell., 158). This principle, in turn, creates a presumption of optimal relevance, where the hearer expects that whatever the speaker is trying to communicate must be worthwhile for them:

Presumption of optimal relevance

1. The ostensive stimulus is relevant enough to be worth the audience’s processing effort.

2. The ostensive stimulus is the most relevant one compatible with the communicator’s abilities and preferences. (Sperber and Wilson 1986/1995Sperber, Dan, and Deirdre Wilson 1986/1995 Relevance: Communication and Cognition, 2nd ed. Oxford: Blackwell., 270)

According to relevance theory, ostensive-inferential communication necessarily involves two layers of intention – the informative intention and the communicative intention:

Ostensive-Inferential Communication

1. The informative intention:

   The intention to inform an audience of something.

2. The communicative intention:

   The intention to inform the audience of one’s informative intention. (Wilson and Sperber 2004Wilson, Deirdre, and Dan Sperber 2004 “Relevance Theory.” In The Handbook of Pragmatics, ed. by Laurence R. Horn, and Gregory Ward, 607–632. Oxford: Blackwell., 611)

As Wharton (2008Wharton, Tim 2008 “ ‘Meaning’ and ‘Showing’: Gricean Intentions and Relevance-Theoretic Intentions.” Intercultural Pragmatics 5 (2): 131–152. , 16) explains, “[t]he first, basic layer is the information being pointed out, and the second is the information that the first layer is being pointed out intentionally”.

Once an audience recognises the communicator’s intentions, they follow the Relevance-Theoretic Comprehension Procedure to recover the intended interpretation:

1. Follow a path of least effort in deriving cognitive effects: test interpretive hypotheses (reference assignments, disambiguations, implicatures etc.) in order of accessibility.

2. Stop when your expectations of relevance are satisfied. (Wilson and Sperber 2004Wilson, Deirdre, and Dan Sperber 2004 “Relevance Theory.” In The Handbook of Pragmatics, ed. by Laurence R. Horn, and Gregory Ward, 607–632. Oxford: Blackwell., 613)

When we process an utterance or any ostensive stimulus through the Relevance-Theoretic Comprehension Procedure, the intended interpretation, or the overall intended meaning of the speaker, could be either an explicature or an implicature. An explicature is an ostensively communicated assumption that is built on a conceptual representation delivered by the linguistic content of an utterance, while an implicature is an assumption communicated ostensively, but solely via inference. See Example (1):

To recover the propositional content of (1), the hearer must inferentially develop the incomplete linguistic representation, or logical form. In (1a), the hearer must fill in where and when it is raining to recover the explicature of this utterance:

Suppose this utterance was produced as a response to Alfie’s question to his mother: Can we go to the playground today? Based on the explicature and on existing assumptions about rain and playgrounds, Alfie would recover implicatures such as:

Notice that not all the implicatures are communicated with the same degree of strength. Some assumptions might be made highly manifest by the speaker while others may not. The implicature in (1c) is strongly communicated and the hearer cannot but recover it. In contrast, implicatures (1d) and (1e) are made manifest only weakly. Such implicatures are called weak implicatures.

Interestingly, emoji can contribute to the recovery of both the explicature and the implicatures. See Examples (2)–(4):

In (2), the use of emoji contributes to the recovery of the explicature, which is today’s lecture was boring, while in (3), Mary’s yawning emoji strongly communicates that she does not want to go to watch the latest Star Wars film. In contrast, in (4), the use of the emoji gives rise to a range of weaker assumptions associated with a boring lecture.

Furthermore, the use of emoji can convey extra layers of weak implicatures. For example, in (2), the use of the emoji will convey a range of feelings associated with a boring lecture while in (3), in addition to the strong implicature that Mary does not want to watch the film, it also gives rise to a range of weak implicatures, such as that Mary does not like Star Wars, Mary is too tired to go out, or Mary is bored with John talking about Star Wars all the time.

Just because the emoji contributes to the recovery of the explicature in (2), it does not necessarily mean that the yawning emoji encodes boredom. In cases such as (2), what the emoji conveys is highly conventionalised and the recipient would have quick access to the intended content. However, this is not the same as saying that the yawning emoji encodes boredom. Interestingly, there is a debate around whether visual cues can involve coding. Forceville (2020)Forceville, Charles 2020 Visual and Multimodal Communication: Applying the Relevance Principle. Oxford: Oxford University Press. , for example, argues that some visual representations, such as traffic signs are coded. This argument has implications for emoji, as some emoji, such as flags, might involve coding. However, it is not clear if facial emoji would also involve coding. Facial expressions as natural signs do not encode the feeling they are associated with. Instead, they are an indicator of the feeling an individual is feeling, and if presented ostensively, they lead the hearer to an intended interpretation. That is, facial expressions, when presented ostensively, point to, rather than encode, a particular emotion.77.See Wharton (2009)Wharton, Tin 2009 Pragmatics and Non-Verbal Communication. Cambridge: Cambridge University Press. for a fuller discussion of the pragmatics of non-verbal communication. In a similar manner, we could argue that the use of facial emoji can lead the recipient to an intended interpretation, which does not necessarily mean that they involve coding.88.The matter of coding and emoji is extremely interesting. However, it is beyond the scope of this study and deserves further and fuller discussion elsewhere.

Finally, the communicator’s intention is not necessarily delivered via linguistic means. In fact, it is explicitly acknowledged in relevance theory that interpretation processes are triggered by any ostensive-communicative behaviour, including naturally occurring behaviour, as long as it is presented to the hearer ostensively (cf. Sperber and Wilson 1986/1995Sperber, Dan, and Deirdre Wilson 1986/1995 Relevance: Communication and Cognition, 2nd ed. Oxford: Blackwell., 1987Sperber, Dan, and Deirdre Wilson 1987 “Précis of Relevance: Communication and Cognition.” Behavioral and Brain Sciences 10: 697–754. ; Wilson and Sperber 2004Wilson, Deirdre, and Dan Sperber 2004 “Relevance Theory.” In The Handbook of Pragmatics, ed. by Laurence R. Horn, and Gregory Ward, 607–632. Oxford: Blackwell.; Wharton 2009Wharton, Tin 2009 Pragmatics and Non-Verbal Communication. Cambridge: Cambridge University Press. ; Scott 2017Scott, Kate 2017 “Prosody, Procedures and Pragmatics.” In Semantics and Pragmatics: Drawing a Line, ed. by I. Depraetere, and R. Salkie, 323–341. Berlin: Springer. ). As Sperber and Wilson (1987Sperber, Dan, and Deirdre Wilson 1987 “Précis of Relevance: Communication and Cognition.” Behavioral and Brain Sciences 10: 697–754. , 39) explain, “[c]ommunication is achieved not by coding and decoding messages, but by providing evidence for an intended hypothesis about the communicator’s intentions”.

This means that the communicator can exploit both coded and non-coded evidence for communication. For example, upon being invited to play a game of tennis, the communicator produces the utterance I have injured my wrist. In this case, the communicator has provided coded evidence for the basic layer of intention, or “the information being pointed out” (Wharton 2008Wharton, Tim 2008 “ ‘Meaning’ and ‘Showing’: Gricean Intentions and Relevance-Theoretic Intentions.” Intercultural Pragmatics 5 (2): 131–152. , 16), and it is a case of meaning. However, if the communicator responds by displaying a bandaged wrist to communicate the same message, they have given direct evidence for the basic layer and hence it is a case of showing. The showing aspect of communication involves presenting evidence that is independent of the verbal input by the communicator. For example, whether the communicator ostensively shows their bandaged wrist or not, the evidence is there. In contrast, in the case of meaning, the evidence is less direct and is dependent on the speaker to produce it.

Note that showing and meaning are not mutually exclusive. The speaker could provide both coded and non-coded evidence at the same time. For example, they might provide coded evidence via the utterance I have injured my wrist while providing non-coded evidence by simultaneously showing their bandaged wrist. Furthermore, unlike Grice (1957), who considers that only non-natural meaning (meaning_NN) involves the recognition of the second layer (the intention to inform the audience of one’s informative intention), relevance theory acknowledges that both showing and meaning_NN involve both informative and communicative intentions and hence, both can be considered as cases of ostensive behaviour. This way, showing–meaning is a continuum, rather than a distinction (Sperber and Wilson 1986/1995Sperber, Dan, and Deirdre Wilson 1986/1995 Relevance: Communication and Cognition, 2nd ed. Oxford: Blackwell.; Wharton 2009Wharton, Tin 2009 Pragmatics and Non-Verbal Communication. Cambridge: Cambridge University Press. ). The notion of the showing–meaning continuum and the idea that communication is not just about providing verbal evidence are particularly important to the current study, as they allow for non-verbal (non-coded) communication, including GIFs and emoji, to be treated in the same way as linguistic input. Additionally, the fact that emoji could contribute to both implicatures and explicatures suggests that we need to approach emoji in a way that would encompass their complex uses. In the next section, I will review how ideas from relevance theory can account for the way emoji and GIFs contribute to relevance.

4.Emoji, GIFs, and relevance

In this section, I discuss how emoji and GIF contribute to communication. We have seen how coded evidence can be used to provide indirect evidence (often in the form of linguistic utterances) for communication. However, this does not necessarily mean that utterances are used only to describe a literal truth about the world. There are many cases where a proposition expressed does not exactly match the thought of the speaker. For example, the communicator exploits evidence by virtue of its resemblance to some other phenomena. This aspect of communication is often overlooked in other approaches in pragmatics, but, as Sperber and Wilson (1986/1995Sperber, Dan, and Deirdre Wilson 1986/1995 Relevance: Communication and Cognition, 2nd ed. Oxford: Blackwell., 227) explain, communication can involve providing evidence via resemblance and “any natural or artificial phenomenon […] can be used as a representation of some other phenomenon that it resembles”.

Resemblance is a relationship between two phenomena. Two representations do not need to be identical and they resemble each other when they share logical or contextual implications, or share certain properties, whether these are visual, phonetic, linguistic, semantic or topological. That is, communication is not just about describing a state of affairs of the world. It can also be about representing some other thoughts. Sperber and Wilson (1986/1995Sperber, Dan, and Deirdre Wilson 1986/1995 Relevance: Communication and Cognition, 2nd ed. Oxford: Blackwell., 228–229) explain:

Any representation with a propositional form, and in particular any utterance, can be used to represent things in two ways. It can represent some state of affairs in virtue of its propositional form being true of that state of affairs; in this case we will say that the representation is a description, or that it is used descriptively. Or it can represent some other representation which also has a propositional form – a thought, for instance – in virtue of a resemblance between the two propositional forms; in this case we will say that the first representation is an interpretation of the second one, or that it is used interpretively.

Examples (5) and (6) illustrate this:

Mary’s response in (5) is an interpretation of what the vet said and hence is a representation of another thought. It resembles the thought of the vet. In contrast, Mary’s response in (6) is her own thought, or her own view of the vet. In this case, her utterance is a case of descriptive use.99.An anonymous reviewer suggests that the distinction between descriptive and interpretive uses of language must be revisited, as it is not clear how this distinction could be maintained when “we do not have any direct access to the thought” (Anonymous reviewer comment). This discussion is beyond the scope of this study and requires a fuller discussion elsewhere.

Wilson (2012Wilson, Deirdre 2012 Metarepresentation in linguistic communication. In Meaning and Relevance, Deidre Wilson, and Dan Sperber, 230–258. Cambridge: Cambridge University Press. , 244) defines interpretive resemblance as “resemblance in content: that is, sharing of implications. Two representations resemble each other (in a context) to the extent that they share logical or contextual implications”. The hearer, following the Relevance-Theoretic Comprehension Procedure, will determine which implications are to be identified as shared between the two phenomena (Sperber and Wilson 1986/1995Sperber, Dan, and Deirdre Wilson 1986/1995 Relevance: Communication and Cognition, 2nd ed. Oxford: Blackwell., 224–231). See Example (7):

Assume that John produced the utterance (7a) when Pat asked him if he wanted to join Pat and others for a dinner out. Reporting back to his friends, Pat might say any of the utterances in (7b) to (7d) to represent John’s original utterance (7a). None of them is a literal reproduction of (7a). However, all three utterances do have properties in common with (7a) and share logical or contextual implications. There are similarities in content between (7a), (7b), and (7c). Similarly, the utterance in (7a) and (7d) share contextual implications. While they are not all the same and none are an identical representation of the original utterance, they still do share certain implications and hence can be used by virtue of resemblance.1010.Non-coded evidence, such as mimicking the act of drinking/making phone calls, could also be used by virtue of its resemblance to other thoughts/phenomena.

While Example (7) is an example that involves interpretive resemblance (resemblance between thoughts), the communicator could provide evidence by virtue of perceptual resemblance. I have demonstrated elsewhere how the use of onomatopoeia involves presenting direct evidence based on phonological resemblance to the source experience (Sasamoto and Jackson 2016Sasamoto, Ryoko and Rebecca Jackson 2016 “Onomatopoeia – Showing-Word or Saying-Word? Relevance Theory, Lexis, and the Communication of Impressions.” Lingua 175: 36–53. ; Sasamoto 2019Sasamoto, Ryoko 2019 Onomatopoeia and Relevance Communication of Impressions via Sound. London: Palgrave MacMillan. ). The point is that propositions expressed are not always used to describe a state of the world. Sperber and Wilson (1986/1995)Sperber, Dan, and Deirdre Wilson 1986/1995 Relevance: Communication and Cognition, 2nd ed. Oxford: Blackwell. indeed explain how acknowledging the gap between the propositional form and the speaker’s thought allows us to account for a wide range of phenomena that were previously treated independently of each other under one theoretical framework, including what has traditionally been considered as figurative use of language such as metaphor and irony.

Metaphor and irony have both traditionally been considered to involve a ‘transfer of meaning’ (see Lakoff 1987Lakoff, George 1987 Women, Fire and Dangerous Things: What Categories Reveal about the Mind. Chicago, IL: University of Chicago Press. , 1994Lakoff, George 1994 “What is Metaphor?” Advances in Connectionist and Neural Computation Theory 3: 203–258.; Fauconnier 1997Fauconnier, Gilles 1997 Mappings in Thought and Language. Cambridge: Cambridge University Press. ; Talmy 2000Talmy, Leonard 2000 Toward a Cognitive Semantics, Vol. 2. Cambridge, MA: MIT Press.). Irony was seen as involving the reversal of meaning, while metaphor was seen as involving a transfer of meaning from one conceptual domain to another. ‘Transfer of meaning’ takes place at the level of semantics, and does not consider contextual constraints, the figurative effects these phenomena are often associated with, or why they exist in the first place (Wilson 2021Wilson, Deirdre 2021 “Explaining Metonymy.” Paper presented at Institute of Humanities Research Seminar, Northumbria University, 17 March 2021.). However, in their series of work, Sperber and Wilson (1986/1995Sperber, Dan, and Deirdre Wilson 1986/1995 Relevance: Communication and Cognition, 2nd ed. Oxford: Blackwell., 1987Sperber, Dan, and Deirdre Wilson 1987 “Précis of Relevance: Communication and Cognition.” Behavioral and Brain Sciences 10: 697–754. , 1998Sperber, Dan, and Deirdre Wilson 1998 “Irony and Relevance: A Reply to Seto, Hamamoto and Yamanashi.” In Relevance Theory. Applications and Implications. ed. by R. Carston, and S. Uchida, 283–293. Amsterdam: John Benjamins. ), Wilson and Sperber (2002Wilson, Deirdre, and Dan Sperber 2002 “Truthfulness and Relevance.” Mind 111 (443): 583–632. , 2004Wilson, Deirdre, and Dan Sperber 2004 “Relevance Theory.” In The Handbook of Pragmatics, ed. by Laurence R. Horn, and Gregory Ward, 607–632. Oxford: Blackwell.), Carston (1997Carston, Robyn 1997 “Enrichment and Loosening: Complementary Processes in Deriving the Proposition Expressed?” Linguistische Berichte 8, Special Issue on Pragmatics: 103–127. , 2002Carston, Robyn 2002 Thoughts and Utterances. Oxford: Blackwell. , 2010Carston, Robyn 2010 “Metaphor: Ad Hoc Concepts, Literal Meaning and Mental Images.” Proceedings of the Aristotelian Society 110 (3): 295–321. ), and Wilson and Carston (2006)Wilson, Deirdre, and Robyn Carston 2006 “Metaphor, Relevance and the ‘Emergent Property’ Issue.” Mind & Language 21 (3): 404–433. show that both metaphor and irony involve an interpretive relation between the propositional form and some thoughts.

Metaphor is a case of interpretive use where the proposition expressed and the speaker’s thought share some implications, and it is explained by the inferential process of ad hoc concept construction. In other words, metaphor involves an interpretive relation between the propositional form of an utterance and the thought of the speaker (see Carston 2002Carston, Robyn 2002 Thoughts and Utterances. Oxford: Blackwell. , 2010Carston, Robyn 2010 “Metaphor: Ad Hoc Concepts, Literal Meaning and Mental Images.” Proceedings of the Aristotelian Society 110 (3): 295–321. ; Wilson and Sperber 2002Wilson, Deirdre, and Dan Sperber 2002 “Truthfulness and Relevance.” Mind 111 (443): 583–632. ; Wilson and Carson 2006Wilson, Deirdre, and Robyn Carston 2006 “Metaphor, Relevance and the ‘Emergent Property’ Issue.” Mind & Language 21 (3): 404–433. ). As Carston (2010Carston, Robyn 2010 “Metaphor: Ad Hoc Concepts, Literal Meaning and Mental Images.” Proceedings of the Aristotelian Society 110 (3): 295–321. , 168) explains, “comprehension of a metaphorical use is a case of ad hoc concept formation and where, crucially, the concept inferred is much broader1111.A concept inferred could also be narrower. For example, in John is a bachelor, uttered by John’s wife, the ad hoc concept BACHELOR* is broader than its lexical meaning, as it now includes a married man. It is also narrower than the lexical concept, as it now excludes some bachelors such as celibate priests. in its denotation than the lexical concept from which it was derived”. See Example (8):

As Carson (2010Carston, Robyn 2010 “Metaphor: Ad Hoc Concepts, Literal Meaning and Mental Images.” Proceedings of the Aristotelian Society 110 (3): 295–321. , 1169) explains, the intended meaning of ‘shark’ in (8), or the ad hoc concept SHARK*, recovered by inference, is much broader than the lexically encoded concept SHARK and it now includes some humans in its denotation. That is, the meaning of SHARK has been adjusted so that the intended concept could be identified. This way, a metaphor acts as a blueprint for the construction of an ad hoc concept that would satisfy the presumption of optimal relevance, and the utterance gives rise to a range of weak implicatures (see Sperber and Wilson 1986/1995Sperber, Dan, and Deirdre Wilson 1986/1995 Relevance: Communication and Cognition, 2nd ed. Oxford: Blackwell., 2008Sperber, Dan, and Deirdre Wilson 2008 “A Deflationary Account of Metaphors.” In The Cambridge Handbook of Metaphor and Thought. ed. by Raymond W. Gibbs, jr., 84–105. Cambridge: Cambridge University Press. ; Carston 2002Carston, Robyn 2002 Thoughts and Utterances. Oxford: Blackwell. , 2010Carston, Robyn 2010 “Metaphor: Ad Hoc Concepts, Literal Meaning and Mental Images.” Proceedings of the Aristotelian Society 110 (3): 295–321. ; Wilson and Carston 2006Wilson, Deirdre, and Robyn Carston 2006 “Metaphor, Relevance and the ‘Emergent Property’ Issue.” Mind & Language 21 (3): 404–433. ).

Verbal irony is also a case of interpretive use of utterances but unlike metaphor, irony involves an interpretive relation between a thought of the speaker and attributed thoughts (thoughts attributed to someone – individuals or groups, including themselves). That is, irony is a case of echoic use,1212. Forceville (2020)Forceville, Charles 2020 Visual and Multimodal Communication: Applying the Relevance Principle. Oxford: Oxford University Press. , in his discussion on visual and multimodal communication, argues that the relevance-theoretic notion of echoic use is the same as what has been discussed as intertextuality in semiotics. Indeed, there are similarities between the two, as both involve references to ‘another text’, or an attributed thought. However, this matter is beyond the scope of this study and requires further discussion. which, according to relevance theory, is a subset of ‘interpretive’ use of language, or communication by virtue of resemblance, which allows the communicator to express, generally tacitly, a range of attitudes1313.Attitude is defined as a subset of non-propositional effects (Sperber and Wilson 2015Sperber, Dan, and Deirdre Wilson 2015 “Beyond Speaker’s Meaning.” Croatian Journal of Philosophy 15 (2): 117–149., 21). to an attributed utterance or thought (Wilson and Sperber 1992Wilson, Deirdre, and Dan Sperber 1992 “On Verbal Irony.” Lingua 87: 53–76. ; Wilson 2006Wilson, Deirdre 2006 “The Pragmatics of Verbal Irony: Echo or Pretence?” Lingua 116: 1722–1743. ). As Wilson (2006Wilson, Deirdre 2006 “The Pragmatics of Verbal Irony: Echo or Pretence?” Lingua 116: 1722–1743. , 1730) explains, “the main point of irony is to dissociate the speaker from an attributed thought or utterance which she wants to suggest is more or less obviously false, irrelevant or under-informative”. As a result, irony often expresses mocking attitude.

Relevance-theoretic accounts of metaphor and irony both involve some type of interpretive relation, or the relation between a propositional form and a thought, which is not necessarily about the literal truth of the world. The differences between the two, crucial for the current study, are that irony necessarily involves the expression of a dissociative attitude via echoic use, and that the attributive thought could be either of the speaker’s own or of someone else’s.

As discussed earlier, emoji are often discussed in relation to metaphor, and GIFs are often considered to communicate an (often dissociative) attitude. The fact that they convey an attitude makes them similar to irony. It is therefore worthwhile to examine the cases of emoji and GIFs in terms of the relation between a ‘proposition’ and a thought. This does not necessarily mean that we can simply draw a parallel between emoji and GIFs in the same way as metaphor and irony. However, it is worth exploring how and why emoji could be seen as metaphor and GIFs are often seen as communicating a mocking attitude.

Also, note that, unlike irony or metaphor, both emoji and GIFs are image-based and there is no proposition expressed per se. What they provide is an input based on an image. This suggests that both emoji and reaction GIFs involve the showing, rather than the meaning, aspect of communication and they are a case of providing direct evidence by virtue of resemblance between its form (i.e. the image of the emoji/GIF) and the emotional state the communicator wishes to convey via an associated facial/bodily expression. In both cases, the relation between the visually presented input and the intended meaning (the emotional state the communicator wishes to convey) provides quick and dirty access to a range of extremely weakly communicated or non-propositional assumptions that form expressive meaning, including attitude and impressions (see Sperber and Wilson 2015Sperber, Dan, and Deirdre Wilson 2015 “Beyond Speaker’s Meaning.” Croatian Journal of Philosophy 15 (2): 117–149.).

Let me start with emoji. Like metaphor, emoji provide a blueprint on which the addressee can build an interpretation of the communicator’s feeling that is particular to a certain context. The use of emoji exploits the relation between the communicator’s state of mind and the visually presented image via an associated facial expression so that the communicator can guide the recipient to use the emoji as a starting point to recover the feeling of the communicator. The recipient would be able to build on the image of the facial expression and recover an impression of, or the range of weak implicatures and non-propositional effects in relation to, the communicator’s feeling or state of mind as a result.

For example, the communicator might use a crying-face emoji 😭 to express their feelings when they write a text to a friend to say their car has been stolen. Here, the emoji is used by virtue of resemblance to the facial expression of crying, which leads to the recovery of associated emotions. The recipient, upon receiving this emoji, recovers an impression of the communicator’s feeling within the context of their car being stolen. If it is something less costly, say, stubbing a little toe against a corner, then the impression of sadness will be somewhat weakened. Similarly, a smiley-face emoji 😄 is used by virtue of its perceptual resemblance to smile, which is often associated with happiness, while the angry emoji 😡 can be used by virtue of its resemblance to the facial expression associated with anger. If the smiley face were to be interpreted in the context of receiving good news about promotion, the degree of the happiness would be intensified, while if it is about finding a small chocolate biscuit when you thought you had run out of biscuits, the degree of happiness would not be as strong. The use of a particular emoji will then guide the recipient to the kind of affect the communicator intends to convey, and indicates that the exchange is to be interpreted in that context. That is, the use of emoji will enable the communicator to create the impression of the feeling in which the exchange is to be interpreted.1414.This function of emoji, to influence the accompanying text, is well documented. See, for example, Scott (2022)Scott, Kate 2022 Pragmatics Online. London: Routledge. for a fuller discussion.

There are also cases where the interpretation of emoji is influenced by the accompanying text.

The yawning emoji could be interpreted either as tiredness or boredom without the accompanying text. However, used with the information provided by the text in (9), the recipient would recover the intended meaning as boredom. This shows that emoji serve as a blueprint or a template on which the intended meaning can be inferred in conjunction with the accompanying text.1515.I would like to thank the anonymous reviewer for pointing out that the interpretation of emoji could also be influenced by the accompanying text. I would also like to thank Kate Scott for these examples.

This understanding of the use of emoji and their role in conveying an impression of the communicator’s feeling enables us to account for the so-called interpersonal function of emoji. As generally acknowledged, emoji can be used to strengthen or weaken the impression of a feeling communicated by other means and can contribute to promoting interpersonal functions and/or discourse management. For example, the use of smiley emoji at the end of a strongly worded email message could guide the recipient to the recovery of a more positive interpretation, in contrast to the severe tone of the email itself. The question is, how does the use of emoji function in this way? It does not matter if the communicator is truly happy or not. The point is that the speaker can make a conscious choice and use the emoji to alter the affect that the recipient would have recovered from the text of the email if it were not for the smiley emoji.1616.Scott (2022Scott, Kate 2022 Pragmatics Online. London: Routledge. , 93) discusses how writers consciously choose to use a particular emoji and unlike facial expressions as natural signals, the use of emoji does not “provide direct evidence of the emotion that the communicator is experiencing as she writes the message”. By using the smiley-face emoji at the end of an angry email, the communicator can soften the tone of the email itself. This does not mean that the smiley emoji at the end of a severe email makes the recipient think that the communicator is in fact happy. Instead, it lessens the intensity of the anger the recipient might have recovered otherwise (see Riordan 2017Riordan, Monica A. 2017 “Emojis as Tools for Emotion Work: Communicating Affect in Text Messages.” Journal of Language and Social Psychology 36: 549–567. ). This way, emoji can be used as part of the communicator’s discourse strategy. The use of emoji, even when the accompanying text conveys a rather strong message, can mitigate what is communicated by the verbal input and help the communicator to maintain the interpersonal relationship. Similarly, if one uses a crying-face emoji at the end of the episode of stubbing their little toe against a corner, the recipient would recover the communicator’s intention that the story is to be taken with sympathy. However, if the episode were accompanied by a laughing emoji, then the recipient would interpret it as less seriously intended. In this way, emoji can help the communicator moderate the discourse (and hence contribute to Yus’ ‘beyond text’ function or other so-called interpersonal functions).

Like emoji, reaction GIFs also involve perceptual resemblance between the form (or the sequence of the image) and the facial (and/or bodily) expression associated with the communicator’s emotional state. The use of a GIF achieves relevance by virtue of resemblance between what it perceptually resembles and the attributed thought. In doing so, it gives rise to a wide range of weak implicatures and non-propositional effects. In addition, like irony, reaction GIFs often involve the communication of an attitude via the echoic use where the communicator expresses their dissociative attitude to the attributed thought delivered by the image of a particular GIF. As pointed out in Section 2, GIFs are often taken from a famous TV programme or film. When the communicator identifies a particular scene as sharing some properties with what they wish to communicate, they use the GIF to echo a thought which is attributed to the original source of the GIF to express a certain attitude (see Wilson and Sperber 2012Wilson, Deirdre, and Dan Sperber 2012 Relevance and Meaning. Cambridge: Cambridge University Press. , 123–146 for a fuller account of irony). Let us examine two famous GIFs: Michael Jackson eating popcorn and Judge Judy’s face palm.

The Michael Jackson eating popcorn GIF involves a sequence of images of the late singer Michael Jackson sitting in a cinema seat, grinning while eating popcorn. This GIF is often used in the comment sections of social media posts where people are arguing in a heated exchange, to express excitement for an upcoming online debate or argument (Aldredge 2019Aldredge, Jourdan 2019 “The Michael Jackson Popcorn Gif and the Controversy of ‘Leaving Neverland’.” No Film School. Available at https://​nofilmschool​.com​/Michael​-Jackson​-Popcorn​-GIF. Accessed 15 October 2020.). Michael Jackson’s action and facial expression represented in this GIF perceptually resemble the facial expression associated with enjoyment anticipating excitement. At the same time, this facial expression could also be seen as ridiculing. By echoing this particular representation, the communicator expresses their mockery as well as what this GIF perceptual resembles: their state of mind, i.e. anticipation. Hence, people upload this GIF not to contribute to the argument, but to show that they are a bystander, enjoying and being entertained by the argument, while expressing mockery.

Judge Judy’s face palm GIF involves a sequence of images from a famous US courtroom show Judge Judy, where Judy Sheindlin, acting as the judge for the ‘court’, famously face palms and shakes her head to express her dismay at petty arguments taking place in front of her. What is represented in this GIF perceptually resembles a bodily expression which is widely accepted as expressing one’s dismay or annoyance. That is, the communicator could use this GIF based on its perceptual resemblance to their own state of mind to express their emotion. However, at the same time, by echoing the way Judge Judy behaves, the use of this particular GIF conveys a mocking attitude.

Note that the addressee need not know the source of the attributed thought. They might notice that thoughts represented in the GIF should be attributed to some source other than the communicator, but this does not necessarily involve identifying that source. They might not recognise Judge Judy when they see the GIF. They may not know where the Michael Jackson GIF is taken from. Still, the use of the GIF allows for the recovery of some attitude. When one sees a person in a judge’s attire shaking their head, they would follow the relevance-theoretic comprehension procedure to recover an interpretation that satisfies their expectation of relevance. This might be based on the simple resemblance between the face palm and head shake represented in the GIF and the bodily expression generally acknowledged as that of expressing anger or dismay, without any reference to the famous programme. This tacit attributive use is consistent with previous studies reviewed in Section 2 which argued that GIFs can be interpreted out of context.

So far, I have explored how the relevance theoretic understanding of metaphor and irony enables us to account for emoji and GIF use. However, it should not be taken that a simple parallel can be drawn between the relationship between emoji and GIF and the relationship between metaphor and irony. Both GIF and emoji are a case of communication by virtue of resemblance and their functions can overlap. On the one hand, both can be used to convey emotions by virtue of resemblance between their form and facial (and/or bodily) expressions associated with communicator’s emotional state. For example, one could send the GIF of Michael Jackson’s GIF to genuinely express one’s excitement with no intention of mockery. On the other hand, both could be used to echo an attributed thought that is made accessible via the use of the emoji or GIF. For example, one could add a face-palm emoji in a message I forgot to put the bin out again, which could be taken as self-mockery. In this case, just like a case of irony, the communicator might be expressing a dissociative attitude resulting in mockery.1717.I would like to send heartfelt thanks to the anonymous reviewers of this manuscript for their comments, in particular on the parallel between emoji/GIFs and metaphor/irony. This shows that while it is true that GIFs often involve echoing thoughts available in public discourse, emoji could also be used to communicate the mocking attitude. Whether an emoji or a GIF is used does not determine whether a mocking attitude is communicated or not. What impacts on the communication of attitude is the dissociative attitude. If it involves an echoic element that results in the communication of a dissociative attitude, then it would function like irony. If there is no echoic element involved, then it does not trigger the communication of attitude.

5.Conclusion

This study is an attempt to offer an explanatory account of emoji and GIFs. There is a rich body of studies on emoji and GIFs (to a lesser extent), but most studies focus on classifying and describing their meaning or affordances. In contrast, in this study, I used ideas from relevance theory and attempted to account for the contribution emoji and GIFs make in communication. I argued that both emoji and reaction GIFs are cases of showing, used by virtue of perceptual resemblance between their form (e.g. the image of emoji/GIF) and the communicator’s state of mind via associated facial/bodily expressions. In both cases, the relation between the visually presented input and the emotional state provides quick access to a range of extremely weakly communicated assumptions about the communicator’s emotions. In addition, both could be used to echo an attributed thought and hence allow the communicator to express their (often dissociative) attitude to the attributed thought represented by the image delivered via a particular GIF or emoji.

By analysing them under the umbrella of relevance theory, this study has shown how both emoji and reaction GIFs can be used to convey expressive meaning, including attitudes and emotions which are difficult to put into words. Analysing them as involving perceptual resemblance as direct evidence for communication and a case of the showing aspect of communication allows us to explain their contribution to communication without treating them as a special case for digital communication. It is true that emoji and GIFs are the product of digital communication. However, the use of visually presented input in communication based on resemblance is a function of human cognition, and this is not limited to digital communication.

As the focus of this study is to explain how emoji and GIFs contribute to communication, a detailed analysis of each emoji or GIF use is beyond its scope. The investigation of emoji and GIFs, perhaps using a multimodal corpus, would shed further light on the nature of emoji and GIFs in communication. Furthermore, this study has stopped short of discussing the matter of visual information and coding fully. This certainly deserves fuller discussion elsewhere.