The pragmatics of text-emoji co-occurrences on Chinese social media

Xiran Yang and Meichun Liu

Abstract

This paper explores the pragmatics of emojis co-occurring with or embedded in text on Chinese social media with this central research question: what are the patterns and the communicative functions manifested by emojis in co-occurrence with Chinese text? Building on the metafunctional approach of multimodal analysis, popular online posts from Sina Weibo which contain both emoji(s) and text have been collected and analyzed to discover the representational, interactive, and compositional features manifested by emojis co-occurring with text. We have found that these emojis on Weibo appear most frequently at the end of the posts and reflect some unique Chinese cultural and linguistic features. Based on recurring pragmatic and functional patterns of text-emoji co-occurrences, it is proposed that emojis are used to perform speech acts, highlight subjective interpretations, and enhance informality, while substituting, reinforcing, and complementing the meanings conveyed by verbal language.

Keywords:
Publication history
Table of contents

1.Introduction: Research question and object of study

The growth of social networking has highlighted the quickly emerging and evolving expressive means of emojis as an essential element of multimodal literacy (SwiftKey 2015SwiftKey 2015SwiftKey Emoji Report, April 2015 (Retrieved 21-02-2020 via https://​blog​.swiftkey​.com​/americans​-love​-skulls​-brazilians​-love​-cats​-swiftkey​-emoji​-meanings​-report/); Danesi 2017Danesi, Marcel 2017The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. CrossrefGoogle Scholar; Herring and Dainas 2017Herring, Susan C., and Ashley Dainas 2017 “Nice Picture Comment!. In Graphicons in Facebook Comment Threads.” Proceedings of the 50th Hawaii International Conference on System Sciences, 2185–2194. Los Alamitos, CA: IEEE. CrossrefGoogle Scholar; Ge and Herring 2018Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) Crossref; Bai et al. 2019Bai, Qiyu, Qi Dan, Zhe Mu, and Maokun Yang 2019 “A Systematic Review of Emoji: Current Research and Future Perspectives.” Frontiers in Psychology 10 (2019): Article 2221. CrossrefGoogle Scholar; see also Kress 2003Kress, Gunther 2003Literacy in the New Media Age. London: Routledge.Google Scholar).11.The word ‘emoji’ could be singular and plural, and could also be pluralized (Danesi 2017Danesi, Marcel 2017The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. CrossrefGoogle Scholar, 2). In this paper, ‘emoji’ is used as a singular term, with ‘emojis’ as its plural form. Originating in Japan, emojis are chromatic graphic icons which are usually seen as successors to emoticons, which consist of ASCII symbols and were first used in the US (Moschini 2016Moschini, Ilaria 2016 “The ‘Face with Tears of Joy’ Emoji. A Socio-Semiotic and Multimodal Insight into a Japan-America Mash-Up.” HERMES-Journal of Language and Communication in Business 55: 11–25. CrossrefGoogle Scholar; Hakami 2017Hakami, Shatha Ali A. 2017 “The Importance of Understanding Emoji: an Investigative Study.” Research Topics in HCI (Spring 2017).Google Scholar; Aull 2019Aull, Bethany 2019 “A Study of Phatic Emoji Use in WhatsApp Communication.” Internet Pragmatics 2 (2): 206–232. CrossrefGoogle Scholar). The Romanized word emoji is a blend of the Japanese words “e ‘picture’” and “moji ‘character’” (Rodrigues et al. 2018Rodrigues, David, Marilia Prada, Rui Gaspar, Margarida V. Garrido, and Diniz Lopes 2018 “Lisbon Emoji and Emoticon Database (LEED): Norms for Emoji and Emoticons in Seven Evaluative Dimensions.” Behavior Research Methods 50: 392–405. CrossrefGoogle Scholar).

Visually adapted to various digital platforms, emojis are now widely accepted and extensively used in China (Lu et al. 2016Lu, Xuan, Wei Ai, Xuanzhe Liu, Qian Li, Ning Wang, Gang Huang, and Qiaozhu Mei 2016 “Learning from the Ubiquitous Language: An Empirical Analysis of Emoji Usage of Smartphone Users.” Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, 770–780. New York: Association for Computing Machinery. CrossrefGoogle Scholar; Jaeger and Ares 2017Jaeger, Sara R., and Gastón Ares 2017 “Dominant Meanings of Facial Emoji: Insights from Chinese Consumers and Comparison with Meanings from Internet Resources.” Food Quality and Preference 62: 275–283. CrossrefGoogle Scholar; Ge and Herring 2018Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) Crossref). Emojis as non-linguistic (or paralinguistic) elements often co-occur with linguistic expressions and form text-emoji co-occurrences, offering pragmatic assistance as well as semantic complements. Such co-occurrences are not haphazard, but appear to follow certain patterns. What, then, are the patterns and the communicative functions manifested by emojis in co-occurrence with Chinese text?

In this study, we adopt a multimodal approach to addressing this question with our data drawn from Sina Weibo (Weibo henceforth), one of the most popular and influential social media platforms in China (Chiu et al. 2012Chiu, Cindy, Chris Ip, and Ari Silverman 2012 “Understanding Social Media in China.” McKinsey Quarterly 2 (2012): 78–81.Google Scholar; Zhou and Wang 2014Zhou, Lijun, and Tao Wang 2014 “Social Media: A New Vehicle for City Marketing in China.” Cities 37: 27–32. CrossrefGoogle Scholar; Fuchs 2014Fuchs, Christian 2014Social Media: A Critical Introduction. London: Sage.Google Scholar; Rauchfleisch and Schäfer 2015Rauchfleisch, Adrian, and Mike S. Schäfer 2015 “Multiple Public Spheres of Weibo: A Typology of Forms and Potentials of Online Public Spheres in China.” Information, Communication & Society 18 (2): 139–155. CrossrefGoogle Scholar).22.The term ‘Weibo’ refers to Sina Weibo (http://​weibo​.com/) in this study. We are particularly interested in Weibo posts which consist of both text and emoji(s), here referred to as text-emoji co-occurrences, or text-emoji mode mixtures.

The main reasons for choosing Weibo as our data source include the following:

  1. It is one of the largest Chinese social networks, with millions of active daily users (Chiu et al. 2012Chiu, Cindy, Chris Ip, and Ari Silverman 2012 “Understanding Social Media in China.” McKinsey Quarterly 2 (2012): 78–81.Google Scholar; Rauchfleisch and Schäfer 2015Rauchfleisch, Adrian, and Mike S. Schäfer 2015 “Multiple Public Spheres of Weibo: A Typology of Forms and Potentials of Online Public Spheres in China.” Information, Communication & Society 18 (2): 139–155. CrossrefGoogle Scholar).

  2. It is a public platform that allows any user to follow any other user and read any post that is not marked private.

  3. It is used by news agencies, corporations, and even governmental organizations to access public opinion.

  4. While emojis on Weibo are limited in variety compared to the more culturally diversified Unicode emojis, they carry more Chinese cultural and linguistic features.

In particular, the data of this study were collected from “Hot Weibo”, which is a column that automatically pushes trending and popular Weibo posts. The criteria for labelling hot Weibo posts include the amount of reposts and comments. As Weibo does not support data extraction or emoji mining through direct site request, we manually collected hot Weibo posts of Chinese text-emoji co-occurrence from February 1 to May 20, 2017.33.We are not using existing corpora of Weibo posts mainly because they have no readily available information on the popularity or accessibility of the posts.

2.Theoretical framework and background

2.1Multimodal studies and the metafunctional approach

Building on the metafunctional approach of systemic-functional theories (see Halliday 1994Halliday, Michael A. K. 1994An Introduction to Functional Grammar (2nd edition). London: Edward Arnold.Google Scholar), multimodal analysis involves non-linguistic and paralinguistic resources for meaning-making (Kress and van Leeuwen 2001Kress, Gunther, and Theo V. van Leeuwen 2001Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.Google Scholar, 2006 2006Reading Images: The Grammar of Visual Design (2nd Edition). London: Routledge. CrossrefGoogle Scholar; Baldry and Thibault 2006Baldry, Anthony, and Paul J. Thibault 2006Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Coursebook. London: Equinox.Google Scholar). The metafunctional approach has the potential to account for all modes and forms of meaning-making, offering a top-down frame in which anatomic analyses can be conducted (O’Halloran 2004O’Halloran, Kay (ed) 2004Multimodal Discourse Analysis: Systemic Functional Perspectives. London: Continuum.Google Scholar; Kress and van Leeuwen 2006 2006Reading Images: The Grammar of Visual Design (2nd Edition). London: Routledge. CrossrefGoogle Scholar; Painter et al. 2013Painter, Clare, James Martin, and Leonard Unsworth 2013Reading Visual Narratives: Image Analysis of Children’s Picture Books. London: Equinox.Google Scholar; Jewitt et al. 2016Jewitt, Carey, Jeff Bezemer, and Kay O’Halloran 2016Introducing Multimodality. London: Routledge. CrossrefGoogle Scholar).

In language use or any form of communication, there are three metafunctions that reflect different aspects of meaning-making: the ideational metafunction construes experience of the world, either with linguistic coding or other semiotic display; the interpersonal metafunction covers social and attitudinal aspects of the discourse; the textual metafunction is concerned with coherence and cohesion in discourse (Halliday 1994Halliday, Michael A. K. 1994An Introduction to Functional Grammar (2nd edition). London: Edward Arnold.Google Scholar; Halliday and Hasan 1976; Kress and van Leeuwen 2001Kress, Gunther, and Theo V. van Leeuwen 2001Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.Google Scholar, 2006 2006Reading Images: The Grammar of Visual Design (2nd Edition). London: Routledge. CrossrefGoogle Scholar; Baldry and Thibault 2006Baldry, Anthony, and Paul J. Thibault 2006Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Coursebook. London: Equinox.Google Scholar). In the context of multimodal discourse, the ideational metafunction is also referred to as representational, the interpersonal as interactive, and the textual as compositional (Kress and van Leeuwen 2006 2006Reading Images: The Grammar of Visual Design (2nd Edition). London: Routledge. CrossrefGoogle Scholar; Baldry and Thibault 2006Baldry, Anthony, and Paul J. Thibault 2006Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Coursebook. London: Equinox.Google Scholar; Jewitt 2014Jewitt, Carey (ed) 2014The Routledge Handbook of Multimodal Analysis. (2nd edition). London: Routledge.Google Scholar; Jewitt et al. 2016Jewitt, Carey, Jeff Bezemer, and Kay O’Halloran 2016Introducing Multimodality. London: Routledge. CrossrefGoogle Scholar). Although different terms are used to better capture the expanded potential and modal variety of discourse components and realization, the essence of each metafunction remains relatively unchanged.

Text-emoji co-occurrences comply with the three metafunctions via multimodal meaning-making. They are multimodal in nature since emojis are usually seen as pictograms, providing a refreshing visual alternative to plain text. Thus the patterns and communicative functions of text-emoji mode mixtures are best studied within a multimodal framework. In multimodal analysis, mode is not strictly defined, and is usually used to refer to a set of socially and culturally shaped resources for meaning-making (MODE 2012MODE 2012Glossary of Multimodal Terms. (Retrieved 21-02-2020 via https://​multimodalityglossary​.wordpress​.com​/mode​-2/); Jewitt 2014Jewitt, Carey (ed) 2014The Routledge Handbook of Multimodal Analysis. (2nd edition). London: Routledge.Google Scholar). Emojis on their own can also be seen as multimodal since they may highlight imagistic or verbal meaning-making processes by activating iconic or symbolic semiotic channels. Emojis can thus be subcategorized into iconic emojis (e.g. smileys), symbolic emojis (e.g. horoscope/star signs), and emojis which contain both iconic and symbolic features. In this study, emojis are viewed as a multimodal component (alongside text) but are not further classified.

Past research on emojis and social media has touched upon various aspects of text-emoji mode mixing. The relations between emojis and text have been found to be mostly complementary or substitutional (Cramer et al. 2016Cramer, Henriette, Paloma de Juan, and Joel Tetreault 2016 “Sender-intended Functions of Emojis in US Messaging.” Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services: 504–509. New York: Association for Computing Machinery. CrossrefGoogle Scholar; Ai et al. 2017Ai, Wei, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei 2017 “Untangling Emoji Popularity Through Semantic Embeddings.” Proceedings of the Eleventh International Conference on Web and Social Media (ICWSM 2017), 2–11. Palo Alto: The AAAI Press. (Retrieved 21-02-2020 via https://​www​.aaai​.org​/ocs​/index​.php​/ICWSM​/ICWSM17​/paper​/view​/15705​/14788); Danesi 2017Danesi, Marcel 2017The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. CrossrefGoogle Scholar; Ge and Herring 2018Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) Crossref). Ai et al. (2017)Ai, Wei, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei 2017 “Untangling Emoji Popularity Through Semantic Embeddings.” Proceedings of the Eleventh International Conference on Web and Social Media (ICWSM 2017), 2–11. Palo Alto: The AAAI Press. (Retrieved 21-02-2020 via https://​www​.aaai​.org​/ocs​/index​.php​/ICWSM​/ICWSM17​/paper​/view​/15705​/14788) find that emojis representing entities usually act as replacement to words, whereas emojis expressing sentiments tend to co-occur with and complement text with attitudes and tones. Danesi (2017)Danesi, Marcel 2017The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. CrossrefGoogle Scholar studies text message exchanges containing emojis and groups the pragmatic functions of emojis into two main categories: adding tone and injecting a positive mood, while other functions such as salutation, punctuation, and irony are also discussed.

Regarding the pragmatic values of text-emoji strings, Herring and Dainas (2017Herring, Susan C., and Ashley Dainas 2017 “Nice Picture Comment!. In Graphicons in Facebook Comment Threads.” Proceedings of the 50th Hawaii International Conference on System Sciences, 2185–2194. Los Alamitos, CA: IEEE. CrossrefGoogle Scholar, 2187) have identified six pragmatic uses of graphicons (emojis, stickers, etc.): mention, reaction, riff, tone modification, action, and narrative sequence. While their study focuses on conversational exchanges and not the co-occurrence of text and emojis, it offers a comprehensive series of steps for investigating mixed modes. Ge and Herring (2018)Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) Crossref focus on the speech acts of emojis and the rhetorical relations between emojis and their accompanying text. Ge and Gretzel (2018)Ge, Jing, and Ulrike Gretzel 2018 “Emoji Rhetoric: A Social Media Influencer Perspective.” Journal of Marketing Management 34 (15–16): 1272–1295. CrossrefGoogle Scholar have examined the emoji-based communicative strategies of Sina Weibo influencers.

In text-emoji co-occurrences, emojis play a critical role in creating and modifying meaning. Different emojis may shade the same verbal message with different sentiments and attitudes; different verbal messages may activate different senses of the same emoji (Wijeratne et al. 2016Wijeratne, Sanjaya, Lakshika Balasuriya, Amit Sheth, and Derek Doran 2016 “Emojinet: Building a Machine Readable Sense Inventory for Emoji.” In SocInfo 2016 Social Informatics, ed. by Emma Spiro, and Yong-Yeol Ahn, 527–541. N.p.: Springer, Cham.Google Scholar; Hakami 2017Hakami, Shatha Ali A. 2017 “The Importance of Understanding Emoji: an Investigative Study.” Research Topics in HCI (Spring 2017).Google Scholar). Such usages of language in combination with emojis point to an integrated stratum of meaning which combines verbal text and graphical emojis. This stratum is most effectively studied from a metafunctional framework which explores representational, interactive, and compositional aspects of communication.

2.2Special features of Weibo emojis

Unlike emoticons which consist of ASCII symbols and usually require readers to tip their heads from one side (e.g. :D) to the other (e.g. < 3) to grasp the meaning of the horizontal display, emojis are easier to view with their upright and colorful design. Previous studies have compared the two alongside other graphicons such as stickers and custom images (Herring and Dainas 2017Herring, Susan C., and Ashley Dainas 2017 “Nice Picture Comment!. In Graphicons in Facebook Comment Threads.” Proceedings of the 50th Hawaii International Conference on System Sciences, 2185–2194. Los Alamitos, CA: IEEE. CrossrefGoogle Scholar; Rodrigues et al. 2018Rodrigues, David, Marilia Prada, Rui Gaspar, Margarida V. Garrido, and Diniz Lopes 2018 “Lisbon Emoji and Emoticon Database (LEED): Norms for Emoji and Emoticons in Seven Evaluative Dimensions.” Behavior Research Methods 50: 392–405. CrossrefGoogle Scholar; Tang and Hew 2018Tang, Ying, and Khe Foon Hew 2018 “Emoticon, Emoji, and Sticker Use in Computer-Mediated Communications: Understanding Its Communicative Function, Impact, User Behavior, and Motive.” In New Media for Educational Change, ed. by Liping Deng, Will W. K. Ma, and Cheuk Wai Rose Fong, 191–201. Singapore: Springer. CrossrefGoogle Scholar; de Seta 2018de Seta, Gabriele 2018 “Biaoqing: The Circulation of Emoticons, Emoji, Stickers, and Custom Images on Chinese Digital Media Platforms.” First Monday 23 (9). (Retrieved 21-02-2020 via http://​firstmonday​.org​/article​/view​/9391​/7566) Crossref). Because emojis reflect users’ emotions and attitudes in digital communication, they have been considered as multimodal tokens in sentiment analysis and opinion mining (Wolf 2000Wolf, Alecia 2000 “Emotional Expression Online: Gender Differences in Emoticon Use.” CyberPsychology & Behavior 3 (5): 827–833. CrossrefGoogle Scholar; Krohn 2004Krohn, Franklin B. 2004 “A Generational Approach to Using Emoticons as Non-verbal Communication.” Journal of Technical Writing and Communication 43: 321–328. CrossrefGoogle Scholar; Churches et al. 2009Churches, Owen, Simon Baron-Cohen, and Howard Ring 2009 “Seeing Face-like Objects: An Event-related Potential Study.” Neuroreport 20 (14): 1290–1294. CrossrefGoogle Scholar; Zhao et al. 2012Zhao, Jichang, Li Dong, Junjie Wu, and Ke Xu 2012 “Moodlens: An Emoticon-based Sentiment Analysis System for Chinese Tweets.” Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, 1528–1531. New York: Association for Computing Machinery. CrossrefGoogle Scholar; Schnoebelen 2012Schnoebelen, Tyler 2012 “Do You Smile with Your Nose? Stylistic Variation in Twitter Smoticons.” University of Pennsylvania Working Papers in Linguistics 18 (2): Article 14.Google Scholar; Novak et al. 2015Novak, Petra Kralj, Jasmina Smailović, Borut Sluban, and Igor Mozetič 2015 “Sentiment of Emojis.” PloS one 10 (12): e0144296.Google Scholar).

While some emojis can be understood across languages and cultures, using emoji as part of the communicative language remains culturally and linguistically dependent. Speakers from different regions or of different languages may prefer different emojis or usages (Lotherington and Xu 2004Lotherington, Heather, and Yejun Xu 2004 “How to Chat in English and Chinese: Emerging Digital Language Conventions.” ReCALL 16 (2): 308–329. CrossrefGoogle Scholar; Ljubešic and Fišer 2016Ljubešic, Nikola, and Darja Fišer 2016 “A Global Analysis of Emoji Usage.” Proceedings of the 10th Web as Corpus Workshop, 82–89. Association for Computational Linguistics. (Retrieved 18-02-2020 via https://​www​.aclweb​.org​/anthology​/W16​-2610​.pdf) Crossref; Lu et al. 2016Lu, Xuan, Wei Ai, Xuanzhe Liu, Qian Li, Ning Wang, Gang Huang, and Qiaozhu Mei 2016 “Learning from the Ubiquitous Language: An Empirical Analysis of Emoji Usage of Smartphone Users.” Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, 770–780. New York: Association for Computing Machinery. CrossrefGoogle Scholar; Sampietro 2019 2019 “Emoji and Rapport Management in Spanish WhatsApp Chats.” Journal of Pragmatics 143: 109–120. CrossrefGoogle Scholar). The interpretation of emojis as part of multimodal discourse, especially alongside co-presented text, is influenced by the text, the linguistic features, and the context (see Halliday and Hasan 1989 1989Language, Context, and Text: Aspects of Language in a Social-semiotic Perspective. Oxford: Oxford University Press.Google Scholar; Stöckl 2004Stöckl, Hartmut 2004 “In between Modes: Language and Image in Printed Media.” In Perspectives on multimodality, ed. by Eija Ventola, Classily Charles, and Martin Kaltenbacher, 9–30. Amsterdam/Philadelphia: John Benjamins Publishing Company. CrossrefGoogle Scholar; Coulson 2006Coulson, Seana 2006 “Constructing Meaning.” Metaphor and Symbol 21(4): 245–266. CrossrefGoogle Scholar).

According to the Unicode Consortium, the Unicode of each emoji is composed of a Unicode string that identifies it across platforms, and a linguistic description of the emoji’s graphic representation (Miller et al. 2017Miller, Hannah, Daniel Kluver, Jacob Thebault-Spieker, Loren Terveen, and Brent Hecht 2017 “Understanding Emoji Ambiguity in Context: The Role of Text in Emoji-Related Miscommunication.” Proceedings of the Eleventh International Conference on Web and Social Media (ICWSM 2017), 152–161. Palo Alto: The AAAI Press. (Retrieved 21-02-2020 via https://​aaai​.org​/ocs​/index​.php​/ICWSM​/ICWSM17​/paper​/view​/15703​/14804); Hakami 2017Hakami, Shatha Ali A. 2017 “The Importance of Understanding Emoji: an Investigative Study.” Research Topics in HCI (Spring 2017).Google Scholar; see also Davis and Edberg 2019Davis, Mark, and Peter Edberg 2019Unicode Emoji-unicode Technical Report #51. Technical Report 51 (Version 12.0). (Retrieved 21-02-2020 via http://​unicode​.org​/reports​/tr51/)). Since pictorial appearance is not standardized by the Unicode, the same emoji would have different graphic renderings when displayed on different digital platforms (See Emojipedia).44. http://​emojipedia​.org/

The emojis available on Weibo are different from emojis that are registered on Emojipedia. While Unicode emojis could be posted on Weibo through smartphones or other devices that have various input systems, Weibo has a set of idiosyncratic emojis specifically designed for use on Weibo. A Weibo emoji, similar to a Unicode emoji, has a pictorial appearance and a verbal description. Unlike other Unicode emojis, however, some particular aspects of Weibo emojis include the following (see Section 4.1. for examples):

  1. The verbal description is usually in Chinese.

  2. Certain emojis have strong Chinese cultural or linguistic features.

  3. Certain emojis are animated with wiggling/flickering effects when displayed on Weibo.55.In this study moving emojis are still viewed as emojis because (1) the moving effect of emojis on Weibo is simplistic and serves to highlight the emojis, and (2) the moving feature is not held across platforms and once reproduced outside Weibo they will mostly become static (like in this paper).

Some Weibo emojis are integrated with the Chinese language and culture, and particular to the communication of users from the online Chinese community. Despite their distinctive features and extensive use, Weibo emojis (and their interaction with text) have been comparatively less studied than Unicode emojis.

3.Methodology

3.1Data collection

Our data set was collected over a period of over three months (February 1 to May 20, 2017) via the column of Hot Weibo.66.Some posts in our data set might no longer be accessible through the search engine on Weibo now because some accounts have been set so they only show posts in the past 6–12 months, and some posts may have been removed by the users. We manually collected Hot Weibo posts that contain both text and emoji(s). The posts were mainly from verified accounts of news agencies, websites, celebrities or other popular media accounts which have large numbers of followers. While online influencers may have contributed extensively to the pool of popular posts, different from other studies which have targeted online influencers’ accounts (Ge and Gretzel 2018Ge, Jing, and Ulrike Gretzel 2018 “Emoji Rhetoric: A Social Media Influencer Perspective.” Journal of Marketing Management 34 (15–16): 1272–1295. CrossrefGoogle Scholar), our approach to data collection relied on the popularity of the post as the main criterion.

As we primarily focus on text-emoji co-occurrences, we confine the data and analyses to the two types of mode/component: Chinese text and emojis. Posts in which text and emojis co-occur, but whose meaning cannot be understood without referring to other co-presented modes, are excluded to maintain semantic completeness of text-emoji co-occurrences. Altogether we obtained a total of 408 Chinese text-emoji mode mixing posts or strings as our data set.77.Since the posts were publicly accessible when we collected them, the ones containing names have not been excluded. In the examples reproduced in this paper, the names mentioned in the posts are replaced with initial letters for ethical concerns. Each text-emoji co-occurrence is seen as an independent unit in the analysis. While not large, still we consider this data set to be representative as the posts, which were popular among users, were collected over a period of over three months.

Some of the Weibo posts may contain a title/headline, which is usually bracketed (‘【】’), ahead of its main body. Emojis may appear in the title and/or main body of a post. Since the title and the main body have a hierarchical relation, they are collected separately.88.When there are emoji(s) in both title and text of the same post, the post will be divided and counted as two units of text-emoji co-occurrence. About 3.8% of the posts originally collected are of this type. Within the title or main body, the strings are not further segmented to provide a more holistic picture of emojis used in each post. The title or main body of a Weibo post that contains both text and emoji(s), therefore, is a unit of analysis.

In this study, while emoji-only strings are not discussed at length, it should be noted that emojis have great potential for making meaning, even without assistance from verbal text (see Xu 2012Xu, Bing 2012Di Shu [The book of earth]. Guangxi: Guangxi Normal University Press.Google Scholar; Mrowiec 2016Mrowiec, Anna 2016How to Speak Emoji Love. London: Ebury Press.Google Scholar; Herring and Dainas 2017Herring, Susan C., and Ashley Dainas 2017 “Nice Picture Comment!. In Graphicons in Facebook Comment Threads.” Proceedings of the 50th Hawaii International Conference on System Sciences, 2185–2194. Los Alamitos, CA: IEEE. CrossrefGoogle Scholar).

3.2Framework of analysis

We study the patterns and communicative functions manifested by emojis mainly by focusing on the representational, interactive, and compositional features of emojis in text-emoji co-occurrences. Viewing text and emojis as two components, we propose a group of componential patterns to capture some recurring pragmatic and functional structures in text-emoji co-occurrences.99.Emojis could be regarded as a mode for its visual consistency in display even though the semiotic nature may range from predominantly iconic to predominantly symbolic in different items.

The representational features of emojis are analyzed by first looking at visual-semantic representations of single emoji items on Weibo, and second by grouping and systematizing the patterns in which text and emojis co-occur based on the number, type, and positioning of emojis. The number of units is taken as the frequency of different patterns.

The interactive features of emojis when co-occurring with text are approached from their emotion-arousing implications and their potential for facilitating social exchanges (see Churches et al. 2009Churches, Owen, Simon Baron-Cohen, and Howard Ring 2009 “Seeing Face-like Objects: An Event-related Potential Study.” Neuroreport 20 (14): 1290–1294. CrossrefGoogle Scholar, 2014Churches, Owen, Mike Nicholls, Myra Thiessen, Mark Kohler, and Hannah Keage 2014 “Emoticons in Mind: An Event-related Potential Study.” Social Neuroscience 9 (2): 196–202. CrossrefGoogle Scholar; Jaeger and Ares 2017Jaeger, Sara R., and Gastón Ares 2017 “Dominant Meanings of Facial Emoji: Insights from Chinese Consumers and Comparison with Meanings from Internet Resources.” Food Quality and Preference 62: 275–283. CrossrefGoogle Scholar; Ge and Gretzel 2018Ge, Jing, and Ulrike Gretzel 2018 “Emoji Rhetoric: A Social Media Influencer Perspective.” Journal of Marketing Management 34 (15–16): 1272–1295. CrossrefGoogle Scholar). Like other forms of communication, emojis could be used to convey attitudes and realize interpersonal exchanges (Gibson et al. 2018Gibson, Will, Pingping Huang, and Qianyun Yu 2018 “Emoji and Communicative Action: The Semiotics, Sequence and Gestural Actions of ‘Face Covering Hand’.” Discourse, Context & Media 26: 91–99. CrossrefGoogle Scholar; Aull 2019Aull, Bethany 2019 “A Study of Phatic Emoji Use in WhatsApp Communication.” Internet Pragmatics 2 (2): 206–232. CrossrefGoogle Scholar; Bai et al. 2019Bai, Qiyu, Qi Dan, Zhe Mu, and Maokun Yang 2019 “A Systematic Review of Emoji: Current Research and Future Perspectives.” Frontiers in Psychology 10 (2019): Article 2221. CrossrefGoogle Scholar). We draw upon the speech act theory in highlighting the illocutionary forces, i.e. the forces to perform acts, of emojis when they are co-presented with text (see Austin 1962Austin, John Langshaw 1962How to Do Things with Words. London: Oxford University Press.Google Scholar; Searle 1976 1976 “A Classification of Illocutionary Acts.” Language in Society 5 (1): 1–23. CrossrefGoogle Scholar; Bach and Harnish 1979Bach, Kent, and Robert M. Harnish 1979Linguistic Communication and Speech Acts. Cambridge: The MIT Press.Google Scholar; Yule 1996Yule, George 1996Pragmatics. Oxford: Oxford University Press.Google Scholar; Skovholt et al. 2014Skovholt, Karianne, Anette Grønning, and Anne Kankaanranta 2014 “The Communicative Functions of Emoticons in Workplace E-Mails::-).” Journal of Computer-Mediated Communication 19 (4): 780–797. CrossrefGoogle Scholar).

The compositional features of text-emoji co-occurrences are based on the visual coherence and reflected by the cohesive patterns in which text and emojis jointly realize the componential integration (see Halliday and Hasan 1985/1976Halliday, Michael A. K., and Ruquiya Hasan 1985/1976Cohesion in English. London: Longman.Google Scholar; Kress and van Leeuwen 2006 2006Reading Images: The Grammar of Visual Design (2nd Edition). London: Routledge. CrossrefGoogle Scholar). The componential patterns displayed in the data reflect the compositional structures of text-emoji co-occurrences and showcase the visual interface of pragmatic functions and practices on Weibo.

Overall, the metafunctional framework of multimodal analysis is top-down as it outlines the three strata in which the actual meaning-building and pragmatic functions unfurl. Within this ternary structure, theories of meaning construction, speech acts, and textual cohesion are employed to interpret the multi-layered pragmatics of emojis.

4.Emojis in representation

4.1Visual-semantic features of Weibo emojis

Compared with emoji sets in iOS and on Twitter, the most conspicuously special Weibo emojis are those that have been integrated with the symbolic meaning of the Chinese characters, such as ‘ ’ (神马 shén mǎ, homonymous to 什么 shén me ‘what’) and ‘ ’ (V5 V-wǔ, homonymous to 威武 wēi wǔ ‘awesome’). Other unique Weibo emojis include those that represent Chinese traditions and concepts, such as ‘ ’ (红包 hóng bāo ‘red packet’), and those that reflect online memes and trending words/expressions, such as ‘ ’ (二哈 èr hā ‘silly dog husky’) and ‘ ’ (互粉 hù fěn ‘to follow each other/become each other’s online friend on Weibo’).1010.Red packets are red envelopes containing money that are given to children during the Chinese New Year. Husky dogs are believed to be troublemakers because they could be overly energetic, especially when kept indoors. See Zhang (2017)Zhang, Yi 2017 “The Semiotic Multifunctionality of Arabic Numerals in Chinese Online Discourse.” Language@Internet 14: Article 2.Google Scholar for more discussions on the integration of Arabic numerals and Chinese on the internet, such as the V5 emoji. Furthermore, some seemingly common emojis have acquired new senses on Weibo, such as ‘ ’ (再见 zài jiàn ‘bye’). Instead of bidding farewell, this emoji could be used to express discontentment or cold shouldering on Weibo. For example, in “别人家的男朋友 bié rén jiā de nán péng yǒu ‘(Look at) Other people’s boyfriends’ ”, the emoji is attached to express the will to part ways with the unsatisfactory boyfriend (probably) of the one posting because others’ boyfriends have done better. Note that although there are some Unicode emojis that are from built-in input systems of various terminals, most of the emojis found in our data come from the emoji set on Weibo.

4.2Visual patterns of text-emoji co-occurrences

Emojis are typically found to be either following a sentence or substituted into the sentence (Cramer et al. 2016Cramer, Henriette, Paloma de Juan, and Joel Tetreault 2016 “Sender-intended Functions of Emojis in US Messaging.” Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services: 504–509. New York: Association for Computing Machinery. CrossrefGoogle Scholar; Ai et al. 2017Ai, Wei, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei 2017 “Untangling Emoji Popularity Through Semantic Embeddings.” Proceedings of the Eleventh International Conference on Web and Social Media (ICWSM 2017), 2–11. Palo Alto: The AAAI Press. (Retrieved 21-02-2020 via https://​www​.aaai​.org​/ocs​/index​.php​/ICWSM​/ICWSM17​/paper​/view​/15705​/14788)). We have found that in a text-emoji co-occurrence, the text is interspersed with emoji(s). Table 1 shows some examples of posts with varied text-emoji representational patterns.1111.The examples are shown in Chinese and pinyin, with word-by-word (sans person, number, tense, and some particles) and overall English translations.

Table 1.Examples of different text-emoji co-occurrences
No. Text-emoji co-occurrences Category
(1)
【永未
yǒng wèi
 never
落地的
luò dì de
landed
MH370
MH370
MH370
航班
háng bān
Flight

‘The Flight MH370 that never landed

Final-single
(2)
【开国少将
kāi guó shào jiàng
 founding general
再陨
zài yǔn
again
一员
yī yuán
die one
jǐn
only
shèng
remain
zhè
these
最 后
zuì hòu
last
30
30
30
人了】
rén le
people

‘Another of the founding generals passed away only 30 are still alive’

Inset-replicated
(3)
Y 入驻
Y rù zhù
Y enter
上海
shàng hǎi
Shanghai
杜莎夫人蜡像馆,
dù shā fū rén là xiàng guǎn,
Madame Tussauds
Y家
Y jiā
the Ys
兄弟,
xiōng dì,
brothers
相似度
xiāng sì dù
likeness
tài
so
高了!
gāo le
high

‘Y enters Madame Tussauds Shanghai, brothers Y, so much alike!

Multiple- (single +mixed)

To classify the representational patterns of text-emoji co-presentation in posts, we have examined the number and type of emojis when they appear together with text. In a text-emoji co-occurrence, wherever emojis appear, there might be a single emoji (single), multiple identical emojis (replicated), or multiple different emojis (mixed), forming an emoji grouplet (which consists of a single emoji or multiple emojis).1212.The word “grouplet” is used as an umbrella term to refer to any instance of emoji(s) that is preceded and/or followed by text within a unit. A grouplet does not contain text. Meanwhile, the positions of emoji grouplet(s) in a co-occurrence unit include onset (one grouplet at the beginning of the text), inset (one grouplet in the middle of the text), final (one grouplet at the end of the text), and multiple emoji grouplets embedded in different positions of the text. The labels “single”, “replicated”, and “mixed” are used to indicate the type of an emoji grouplet; the labels “onset”, “inset”, “final”, and “multiple” are used to categorize text-emoji co-occurrence units based on the positioning of emoji grouplet(s).

Here we propose a diagram (Diagram 1) to categorize text-emoji co-occurrences along four representational dimensions: (1) number of emoji grouplets in one co-occurrence unit, (2) positioning of emoji grouplets in one co-occurrence unit, (3) number of emojis in one grouplet, and (4) type of emojis in one grouplet. This categorization reflects the visual patterns of text-emoji co-occurrences.

Diagram 1.Representational patterns of text-emoji co-occurrences
Diagram 1.

The distribution of different representational categories is shown in Table 2. Overall, approximately 77.5% of text-emoji co-occurrences only have emoji(s) at the end (the “Final” category), of which about 80% only have a single emoji. Within the “Multiple (occurrences)” category, 86.4% of the posts contain final-positioned emoji(s).

Table 2.Percentage of different categories of text-emoji co-occurrences
(%) Text-emoji representational category
Onset Inset Final Multiple Subtotal
Subtotal 1.7% 4.6% 77.5% 16.2% 100%

A contingency table (Table 3) is given to summarize the detailed frequencies of different emoji grouplets and their positions in different categories of text-emoji co-occurrence. In the “Multiple (occurrences)” category, one unit contains multiple emoji grouplets; other categories only contain one emoji grouplet. Altogether, there are 486 emoji grouplets in the data.

Table 3.Number of emoji grouplets of different types and in different positions
Type of emoji grouplets Position of emoji grouplets
Onset Inset Final Multiple Subtotal
Single 6 15 256 110 387 (79.6%)
Replicated 0  4  45  28  77 (15.9%)
Mixed 1  0  15   6  22 (4.5%)
Subtotal 7 19 316 144 (in 66 units) 486 grouplets

In general, the distribution is skewed towards the final positioning of emoji(s) and the single emoji occurrence. When inserting multiple emojis as one grouplet, the users prefer repeating the same emoji over mixing different emojis. We have observed a common representational pattern of “text followed by emoji(s)” in the co-occurrence units, which suggests that text is the principal method of coding information in these posts, and that emoji(s) contribute additional meaning to the preceding text. This might be in line with earlier studies in which emojis have been found to carry complementary or supplementary meaning to accompanying words (see Danesi 2017Danesi, Marcel 2017The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. CrossrefGoogle Scholar; Ai et al. 2017Ai, Wei, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei 2017 “Untangling Emoji Popularity Through Semantic Embeddings.” Proceedings of the Eleventh International Conference on Web and Social Media (ICWSM 2017), 2–11. Palo Alto: The AAAI Press. (Retrieved 21-02-2020 via https://​www​.aaai​.org​/ocs​/index​.php​/ICWSM​/ICWSM17​/paper​/view​/15705​/14788); Ge and Herring 2018Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) Crossref).

Based on the metafunctional approach of multimodal analysis and the cognitive meaning-constructing model raised by Coulson (2006)Coulson, Seana 2006 “Constructing Meaning.” Metaphor and Symbol 21(4): 245–266. CrossrefGoogle Scholar, the readers rely on the imagistic feature of emojis to decide the meaningful and structural linkages between the emojis and the accompanying text. In real communication, a single emoji is found to be understood in different senses (Wijeratne et al. 2016Wijeratne, Sanjaya, Lakshika Balasuriya, Amit Sheth, and Derek Doran 2016 “Emojinet: Building a Machine Readable Sense Inventory for Emoji.” In SocInfo 2016 Social Informatics, ed. by Emma Spiro, and Yong-Yeol Ahn, 527–541. N.p.: Springer, Cham.Google Scholar). However, different from what has been suggested in some previous studies, we do not consider this potential for ambiguity to be an obstacle in analyzing emoji-embedded text. Rather, we view the multimodal meaning-making of text-emoji as a compositionally based interactive process that allows for flexibility in terms of representational expression. In other words, the multimodal meaning of text-emoji co-occurrence is not only represented by its visual form, but is also generated by functional relations between text and emoji(s). The communicative functions of text-emoji co-occurrences are further examined from interactive and compositional perspectives.

5.Emojis in social interaction

Emoticons were first used to complement the lack of access to emotional contagion or empathy in digital communication (see Hatfield et al. 1993Hatfield, Elaine, John T. Cacioppo, and Richard L. Rapson 1993 “Emotional Contagion.” Current Directions in Psychological Sciences 2: 96–99. CrossrefGoogle Scholar; Churches et al. 2009Churches, Owen, Simon Baron-Cohen, and Howard Ring 2009 “Seeing Face-like Objects: An Event-related Potential Study.” Neuroreport 20 (14): 1290–1294. CrossrefGoogle Scholar, 2014Churches, Owen, Mike Nicholls, Myra Thiessen, Mark Kohler, and Hannah Keage 2014 “Emoticons in Mind: An Event-related Potential Study.” Social Neuroscience 9 (2): 196–202. CrossrefGoogle Scholar; Yuasa et al. 2011Yuasa, Masahide, Keiichi Saito, and Naoki Mukawa 2011 “Brain Activity When Reading Sentences and Emoticons: an fMRI Study of Verbal and Nonverbal Communication.” Electronics and Communications in Japan 94 (5): 17–24. CrossrefGoogle Scholar; Skovholt et al. 2014Skovholt, Karianne, Anette Grønning, and Anne Kankaanranta 2014 “The Communicative Functions of Emoticons in Workplace E-Mails::-).” Journal of Computer-Mediated Communication 19 (4): 780–797. CrossrefGoogle Scholar), and emojis have further extended the options and convenience of such practice (Tauch and Kanjo 2016Tauch, Channary, and Eiman Kanjo 2016 “The Roles of Emojis in Mobile Phone Notifications.” Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunt, 1560–1565. New York: Association for Computing Machinery. CrossrefGoogle Scholar; Vidal et al. 2016Vidal, Leticia, Gastón Ares, and Sara R. Jaeger 2016 “Use of Emoticon and Emoji in Tweets for Food Related Emotional Expression.” Food Quality and Preference 49: 119–128. CrossrefGoogle Scholar; Jaeger and Ares 2017Jaeger, Sara R., and Gastón Ares 2017 “Dominant Meanings of Facial Emoji: Insights from Chinese Consumers and Comparison with Meanings from Internet Resources.” Food Quality and Preference 62: 275–283. CrossrefGoogle Scholar; Aull 2019Aull, Bethany 2019 “A Study of Phatic Emoji Use in WhatsApp Communication.” Internet Pragmatics 2 (2): 206–232. CrossrefGoogle Scholar; Smith and Rose 2019Smith, Leah Warfield, and Randall L. Rose 2019 “Service with a Smiley Face: Emojional Contagion in Digitally Mediated Relationships.” International Journal of Research in Marketing (In press; available online).Google Scholar). Emojis could be seen as a part of social exchange that is visually materialized and has become almost indispensable with the rise of information technology (Wolf 2000Wolf, Alecia 2000 “Emotional Expression Online: Gender Differences in Emoticon Use.” CyberPsychology & Behavior 3 (5): 827–833. CrossrefGoogle Scholar; Crystal 2006Crystal, David 2006Language and the Internet. Cambridge: Cambridge University Press. CrossrefGoogle Scholar; Danesi 2017Danesi, Marcel 2017The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. CrossrefGoogle Scholar; Bai et al. 2019Bai, Qiyu, Qi Dan, Zhe Mu, and Maokun Yang 2019 “A Systematic Review of Emoji: Current Research and Future Perspectives.” Frontiers in Psychology 10 (2019): Article 2221. CrossrefGoogle Scholar).

For a particular text-emoji string, there are interactive meanings conveyed and realized with the assistance of emojis. The most common interactive functions of emoji(s) observed in this study include performing speech acts, highlighting subjective interpretations, and enhancing informality.

5.1Speech acts of emojis

Any utterance in actual use carries three layers of force to indicate the implied act: locutionary, illocutionary, and perlocutionary (Austin 1962Austin, John Langshaw 1962How to Do Things with Words. London: Oxford University Press.Google Scholar; Searle 1969/2011Searle, John R. 1969/2011Speech Acts: An Essay in the Philosophy of Language. Cambridge: Cambridge University Press. CrossrefGoogle Scholar; Yule 1996Yule, George 1996Pragmatics. Oxford: Oxford University Press.Google Scholar). Like words, emojis and emoticons can be used to perform speech acts and facilitate communication (Dresner and Herring 2010Dresner, Eli, and Susan C. Herring 2010 “Functions of the Nonverbal in CMC: Emoticons and Illocutionary Force.” Communication Theory 20 (3): 249–268. CrossrefGoogle Scholar, 2014 2014 “Emoticons and Illocutionary Force.” In Perspectives on Theory of Controversies and the Ethics of Communication, ed. by Dana Riesenfeld, and Giovanni Scarafile, 59–70. Springer Netherlands. CrossrefGoogle Scholar; Vandergriff 2013Vandergriff, Ilona 2013 “Emotive Communication Online: a Contextual Analysis of Computer-mediated Communication (CMC) cues.” Journal of Pragmatics 51: 1–12. CrossrefGoogle Scholar; Ge and Herring 2018Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) Crossref; dos Reis et al. 2018dos Reis, Julio C., Rodrigo Bonacin, Heiko H. Hornung, and M. Cecília C. Baranauskas 2018 “Intenticons: Participatory Selection of Emoticons for Communication of Intentions.” Computers in Human Behavior 85: 146–162. CrossrefGoogle Scholar). Ge and Herring (2018)Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) Crossref build on the speech act taxonomy in computer-mediated discourse (see Herring et al. 2005Herring, Susan C., Anupam Das, and Shashikant Penumarthy 2005CMC Act Taxonomy. (Retrieved 21-02-2020 via http://​ella​.slis​.indiana​.edu​/~herring​/cmc​.acts​.html)) to categorize the illocutionary acts of emoji sequences on Sina Weibo, discovering that claim, desire, and explain are among the most frequently used acts.

In this study, we highlight the illocutionary force of emojis, which is usually activated by semantic content of the co-occurring text. The referent of the illocutionary act could be objects or notions mentioned in co-occurring text, or textual meaning in general. The acting power of emojis is examined in the context of text-emoji co-occurrence because when an act is being executed by the emoji(s), it is part of the message that is delivered by the text-emoji co-occurrence. We draw upon the general classifications of illocutionary acts and discussions of perlocutionary intentions proposed by Bach and Harnish (1979)Bach, Kent, and Robert M. Harnish 1979Linguistic Communication and Speech Acts. Cambridge: The MIT Press.Google Scholar and Searle (1976) 1976 “A Classification of Illocutionary Acts.” Language in Society 5 (1): 1–23. CrossrefGoogle Scholar to label the acts exemplified here (Table 4).

Table 4.Illustrating speech-act analysis of emojis in co-occurrence units
No. Co-occurrence units
(4)
【突发!
tū fā!
 sudden
#南宁
#nán níng
Nanning
mǒu
a
职业学院
zhí yè xué yuàn
vocational school
学生
xué shēng
student
tiào
jump
楼#
lóu#
building

‘Breaking news! #A student of a vocational school in Nanning jumped off the roof#

Locutionary act (representational meanings of the emoji): a lit candle stick

Illocutionary act: acknowledgement/expressive – expressing condolences and mourning towards the roof-jumping event reported in the preceding text

Perlocutionary act (intention): to make the readers believe that the poster is mourning

(5)
lùn
discuss
选择的
xuǎn zé de
choice’s
重要性
zhòng yào xìng
importance
不要
bú yào
do not
相信
xiāng xìn
trust
低价
dī jià
cheap
护肤品
hù fū pǐn
skin care product

‘Choice matters Do not trust cheap skin care products

Locutionary act (representational meanings of the emoji): waving goodbye/leaving/parting ways

Illocutionary act: directive – refusing or dismissing the low-priced cosmetics mentioned in the preceding text

Perlocutionary act (intention): to make the readers believe that they need to refuse cheap cosmetics

On Weibo, one distinctive emoji that has been used to conduct speech acts is the lit candle emoji ‘ ’, which expresses condolences or R.I.P. in posts that contain messages of casualty, mourning, or obituary. In such cases the lit candle emoji carries the illocutionary force of sympathizing or expressing condolences towards the deceased that is often understood from the text or context. On the receiving end, the emoji could effectuate the perlocutionary force of making the readers believe that whoever posts the emoji is mourning and thus appear to be compassionate.

5.2Highlighted subjective interpretation

Re-posting is frequently seen on social media. We have noticed that when re-posting news briefings, those re-posting, either agencies or individual users, might change emojis in the re-posted news while maintaining the verbal text, reflecting the posters’ subjective interpretations of the re-post. Examples are given in Table 5.1313.Example (6) was accessed from the Hot Weibo column. The posts in Table 5 are not in chronological order.

Table 5.Same (or similar) news posts with different emojis
No. News posts (by different posters) Emoji descriptors
(6)
【在
zài
on
高速上
gāo sù shàng
highway
倒车逆行,
dào chē nì xíng,
back a car
司机:
sī jī:
driver
I
以为
yǐ wéi
think
可以
kě yǐ
allowed
Throwing hands up/dismissive
(7)
【司机
sī jī
driver
zài
on
高速上
gāo sù shàng
highway
倒车逆行:
dào chē nì xíng:
back a car
I
以为
yǐ wéi
think
可以
kě yǐ
allowed
Sweating/ashamed
(8)
【高速上
gāo sù shàng
(on) highway
倒车逆行
dào chē nì xíng
back a car
司机:
sī jī:
driver
I
以为
yǐ wéi
think
可以
kě yǐ
allowed
Suffering/pathetic
Eng. ‘Backing the car on highway. The driver: I thought it was allowed (Emoji)’

It can be observed from Table 5 that the reported speech of the rule-breaking driver “我以为可以 (wǒ yǐ wéi kě yǐ) ‘I thought it was allowed’” has maintained both its verbal content and its relative position to emoji component, while the rest of the verbal information has undergone some minor changes. In general, the information conveyed by text has been left undisturbed in all the re-posts. The emojis included, however, are drastically different.

In Example (6) the emoji depicts a dismissive character that does not seem to take the incident seriously, thereby implying the referent of the emoji is the driver, since the poster of the news is unlikely to be indifferent to the violation of traffic law. In Example (7) an emoji of ashamed face is used after the verbal text, indicating that the driver was mortified of the incident. In Example (8) the emoji features a shocked/overwhelmed face that, instead of a troublemaker, looks like a victim.

All three emojis in Table 5 have complemented the verbal information. However, the different emojis have exerted considerable influence over the original post, supplying different complementary information and judgement about the driver’s reaction towards the incident. Simply attaching different emojis can alter how the same incident is interpreted. While emojis generally highlight subjective stance and attitude, those emojis in different re-posts more obviously reveal different attitudes.

5.3Enhanced informality

Another important and probably more general interactive and pragmatic function of emojis is their power in promoting informality and accessibility. In digital communication emojis are usually associated with informality and a casual tone, and they are often assumed to be used more frequently among peers, friends, and relatives (Danesi 2017Danesi, Marcel 2017The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. CrossrefGoogle Scholar). On Weibo, emojis are not only used by ordinary users but have also been incorporated into news briefs and short reports that are publicized by officially acknowledged accounts/users (see Zhou and Pan 2016Zhou, Huiquan, and Quanxiao Pan 2016 “Information, Community, and Action on Sina-Weibo: How Chinese Philnatropic NGOs Use Social Media.” VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations 27 (5): 2433–2457. CrossrefGoogle Scholar). Once emojis are embedded, the over-all tone of the message becomes less formal and sometimes more approachable, especially with serious genres such as news brief (see Table 4 and 5).

While news reports are supposed to appear authoritative and formal, online platforms such as Weibo have to some extent mollified the rigid impression, partly by inserting emojis to make the news publishers appear more approachable to their audience. The use of text-emoji co-occurrences on Weibo has encouraged and created more informal and casual exchanges and contexts. The inclusion of emojis also helps make the news posts stand out and compete for readers’ attention along with commercial and entertaining posts.

6.Compositional and componential patterning of text-emoji co-occurrences

6.1Compositional relations

The textual or compositional meaning is manifested primarily through the coherence and cohesion of the discourse (Halliday and Hasan 1985/1976Halliday, Michael A. K., and Ruquiya Hasan 1985/1976Cohesion in English. London: Longman.Google Scholar; Kress and van Leeuwen 2006 2006Reading Images: The Grammar of Visual Design (2nd Edition). London: Routledge. CrossrefGoogle Scholar). On the platform of Weibo, text-emoji co-presentation is but part of the entire discourse, which contains other multi-semiotic resources (see Baldry and Thibault 2006Baldry, Anthony, and Paul J. Thibault 2006Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Coursebook. London: Equinox.Google Scholar; Herring and Dainas 2017Herring, Susan C., and Ashley Dainas 2017 “Nice Picture Comment!. In Graphicons in Facebook Comment Threads.” Proceedings of the 50th Hawaii International Conference on System Sciences, 2185–2194. Los Alamitos, CA: IEEE. CrossrefGoogle Scholar). Here we focus on the interaction between Chinese text and accompanying emoji(s), observing how their co-dependence contribute to coherence and cohesion. This part leads to the establishment of the componential patterning of text-emoji co-occurrences by specifying compositional relations between text and emojis, and identifying the functions of emojis in the co-occurrences.

Coherence and cohesion refer to structural and semantic ties that are found among different components of the text (see Halliday and Hasan 1985/1976Halliday, Michael A. K., and Ruquiya Hasan 1985/1976Cohesion in English. London: Longman.Google Scholar; Halliday 1994Halliday, Michael A. K. 1994An Introduction to Functional Grammar (2nd edition). London: Edward Arnold.Google Scholar; Kress and van Leeuwen 2001Kress, Gunther, and Theo V. van Leeuwen 2001Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.Google Scholar, 2006 2006Reading Images: The Grammar of Visual Design (2nd Edition). London: Routledge. CrossrefGoogle Scholar; Baldry and Thibault 2006Baldry, Anthony, and Paul J. Thibault 2006Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Coursebook. London: Equinox.Google Scholar). The expansion of componential modality to include emojis means that the realization of coherence and cohesion is also expanded. The compositional relations are based on representational and interpersonal meanings of text-emoji co-occurrences, engaging visual attention and constructing coherence and logical relations.

Cohesive devices commonly found in textual discourse include reference, substitution, and lexical cohesion (Halliday and Hasan 1985/1976Halliday, Michael A. K., and Ruquiya Hasan 1985/1976Cohesion in English. London: Longman.Google Scholar, 29). Taking into consideration the multimodal features and meaning-making potential of emojis, we have observed mainly four types of compositional relations between text and emojis in the data: substitution (where emojis are used to substitute lexical or phrasal items in the text, e.g. ‘ ’ for ‘flight’), co-reference (where emojis and verbal elements refer to the same meaning or semantically related concepts, e.g. ‘ ’ and ‘cake/birthday’), complement (where emojis are used to code additional tones or other information to verbal text), and others (where the patterning of text and emojis cannot be identified by the previous three types).

These patterns are basically in line with the findings from other emoji studies that used English or Chinese-based examples, namely that emojis could be used to complement, reinforce, and replace words, and that emojis could be used on their own to tell stories (Danesi 2017Danesi, Marcel 2017The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. CrossrefGoogle Scholar; Herring and Dainas 2017Herring, Susan C., and Ashley Dainas 2017 “Nice Picture Comment!. In Graphicons in Facebook Comment Threads.” Proceedings of the 50th Hawaii International Conference on System Sciences, 2185–2194. Los Alamitos, CA: IEEE. CrossrefGoogle Scholar; Ai et al. 2017Ai, Wei, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei 2017 “Untangling Emoji Popularity Through Semantic Embeddings.” Proceedings of the Eleventh International Conference on Web and Social Media (ICWSM 2017), 2–11. Palo Alto: The AAAI Press. (Retrieved 21-02-2020 via https://​www​.aaai​.org​/ocs​/index​.php​/ICWSM​/ICWSM17​/paper​/view​/15705​/14788); Ge and Herring 2018Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) Crossref). However, unlike previous emoji-related analyses, we use a componential approach to sketch the interactive patterns between text and emojis, and distinguish between different uses of complementary emojis – external tonal, internal tonal, and non-tonal.

6.2Componential patterning

We view text and emoji as two types of component in making meaning on Weibo. Combining the visual patterns and pragmatic functions revealed so far, we propose a set of quasi-compositional rules that would incorporate the communicative features of text-emoji co-occurrences. Since the compositional relations are built between text and emojis, they are termed as componential patterns, indicating that the two different components are seen as interdependent chunks in forming collective meaning.

The basic componential patterning of text-emoji co-occurrence takes shape according to the aforementioned compositional relations, and is manifested in five basic types: componential substitution, inter-modal reinforcement, tonal complement (external and internal), non-tonal complement, and creative illustration. Here the categories are divided and re-named to reflect not only the cohesive ties, but also the forms and functions of text-emoji co-occurrences, as well as the relevant findings in previous studies (see Danesi 2017Danesi, Marcel 2017The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. CrossrefGoogle Scholar; Ai et al. 2017Ai, Wei, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei 2017 “Untangling Emoji Popularity Through Semantic Embeddings.” Proceedings of the Eleventh International Conference on Web and Social Media (ICWSM 2017), 2–11. Palo Alto: The AAAI Press. (Retrieved 21-02-2020 via https://​www​.aaai​.org​/ocs​/index​.php​/ICWSM​/ICWSM17​/paper​/view​/15705​/14788)). The five basic structures assign different roles to text and emojis to build a multimodal layer of meaning. Within the basic structures, text and emojis have the capacity to formulate various representations and perform interactive functions.

Before elaborating on the basic patterns, key terms of componential patterning are defined and explained as follows:

  1. (Componential) Meaning: the meaning of text and the meaning of emojis. The meanings examined in componential patterns are usually labeled in generic and categorical manner, leading to logical and semantic relations between different components.

  2. (Componential) Agreement: the compatibility between the meaning of text and the meaning of emojis. Agreement can be found along a continuum from entirely identical meaning between text and emoji (which may never be achieved), to related meanings or concepts, to logical matches. Note that obvious discrepancies between text and emojis in their co-occurrence would probably render rhetorical meanings such as sarcasm.

  3. (Componential) Functions: the contextualized pragmatic functions that reflect the representational, interactive, and compositional aspects of text-emoji co-occurrences.

  4. Projection: the reference to, reflection of, or quotation of a certain (part of) component. For example, an emoji could be used to project the information mentioned in the accompanying text, or to project the role of the poster.

In the following analyses, the subscripts of angle brackets (‘< >componential meaning’) describe the componential meaning of the component, and the subscripts of square brackets (‘[ ]componential agreement’) describe the agreements between text and emojis (as two components) that are pinpointed in each case.1414.Most of the examples in this section contain a single emoji to reflect the predominant representational pattern in text-emoji co-occurrences on Weibo. Multiple identical emojis in one text-emoji co-occurrence could be interpreted as holding visual emphasis; multiple different emojis would be analyzed separately (see Ge and Herring 2018Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) Crossref).

In componential substitution (Table 6), the meaning of the whole message is usually not complete without including emoji(s) as part of the verbal text. The role of emojis compensates for the obligatory part of the text in line with the grammatical patterns of Chinese. If emojis appear after a noun, they are likely to assume an attributive or modifying function. If an emoji co-occurs with a verb, it likely fills in an adverbial gap.

Table 6.Componential substitution
Componential Patterning
Example
外交
wài jiāo
foreign affairs
ministry

‘Ministry of Foreign Affairs

Meaning [<Ministry of Foreign Affairs> state institution]topic/modified[< > awesome]modifier
Agreement A logical match found between the name of an institution ‘Ministry of Foreign Affairs’ and the emoji that indicates positive properties ‘ ’ (awesome)
Functions To substitute Chinese characters with more visually salient components; to enhance the informality of the message that mentions an authority
Pattern Componential substitution: the emoji ‘ ’ substitutes a verbal modifier (e.g. awesome) for the preceding text

In inter-modal reinforcement (Table 7), the emojis usually repeat or reinforce the notion/object mentioned in the text. Namely, the text or part of the text and the emojis refer to the same or similar concepts. Since different people may associate the same emoji with different meanings, the exact pairing of inter-modal reinforcement might be debatable sometimes. It is, however, mostly straightforward when part of the information conveyed by text is repeated or reinforced by co-referring emojis.

Table 7.Inter-modal reinforcement
Componential Patterning
Example
父子俩
fù zǐ liǎng
father and son
对话,
duì huà,
conversation
笑死人啊!
xiào sǐ rén ā!
laugh so hard

‘A conversation between father and son, lots of laugh!

Meaning [A conversation between father and son, lots of <laugh> laugh!]related meaning [< >laughing/hilarious]related meaning
Agreement A pair of related/similar meanings found between the word ‘laugh’ and the emoji of laughing expression ‘ ’ (laughing/hilarious)
Functions To reinforce verbal message; to visually highlight/emphasize the funny property of the message
Pattern Inter-modal reinforcement: the emoji ‘ ’ reinforces the preceding text/word ‘laugh’

In tonal complement and non-tonal complement, emojis encode meanings beyond the co-presented text, and provide tonal/sentimental/attitudinal or other types of information alongside the text, which then usually becomes (partly) projected. Here the text and emojis do not refer to the same concept but complement each other to formulate a larger meaningful unit. Functions such as tone-adding/modifying are included in the tonal category (see Danesi 2017Danesi, Marcel 2017The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. CrossrefGoogle Scholar; Herring and Dainas 2017Herring, Susan C., and Ashley Dainas 2017 “Nice Picture Comment!. In Graphicons in Facebook Comment Threads.” Proceedings of the 50th Hawaii International Conference on System Sciences, 2185–2194. Los Alamitos, CA: IEEE. CrossrefGoogle Scholar). Functions such as adding contextual or other non-tonal/non-sentimental information are included in the non-tonal category.

Based on our data, the category of tonal complement could be further divided into two sub-categories. External tonal complement (Table 8) refers to the cases in which emojis are used to express the post writer’s personal attitude, while internal tonal complement (Table 9) indicates the cases where emojis depict the attitudes or tonal features of the characters or concepts mentioned in the co-occurring text. Either way, it should be noted that the use of emojis is mostly a subjective choice.

Table 8.External tonal complement
Componential Patterning
Example
车水马龙
chē shuǐ mǎ lóng de
busy
的街道上,
jiē dào shàng,
(on) a street
司机们
sī jī men
drivers
为一
wèi yī
for
tiáo
a
liú
stray
浪的
làng de
dog
汪让路......
wāng ràng lù......
make way

‘Drivers make way for a stray dog on a busy street......

Meaning [<Drivers make way for a stray dog on a busy street......>warm/adorable scene/action]projected message [< >heart/love/fondness]external comment/tone
Agreement A logical match is found between the words that describe a warm and adorable scene/action and the emoji of positive feelings ‘ ’ (heart/love/fondness)
Functions To complemnt verbal message; to visually present the posotive attitude of the one posting (toward the warm and adorable scene/action)
Pattern External tonal complement: the emoji ‘ ’ serves as tonal information of the post writer to comment on the preceding text
Table 9.Internal tonal complement
Componential Patterning
Example
【在
zài
on
高速上
gāo sù shàng
highway
倒车逆行,
dào chē nì xíng,
back a car
司机:
sī jī:
driver
I
以为
yǐ wéi
think
可以
kě yǐ
allowed

‘Backing the car on highway, the driver: I thought it was allowed

Meaning [<Backing the car on highway>traffic violation<The driver>character: I thought it was allowed]partly projected message [< >dismissive attitude]internal comment/tone
Agreement A logical match is found between the word ‘driver’ (the rule-breaking character) and the emoji indicating a dismissive attitude ‘ ’ (throwing hands up/dismissive)
Functions To complement verbal message; to visually present the questionable attitude of the character in question
Pattern Internal tonal complement: the emoji ‘ ’ serves as descriptive information of the character to partly project the text

Complementary information other than tones or attitudes, such as contextual and other circumstantial information, is covered by the category of non-tonal complement (Table 10).

Table 10.Non-tonal complement
Componential Patterning
Example
中国
zhōng guó
China
zuì
most
měi
beautiful
十大
shí dà
top 10
古镇,
gǔ zhèn,
ancient town
you
去过
qù guò
been to
几个?
jǐ gè?
how many

‘Top 10 most beautiful ancient towns in China, how many have you been to?

Meaning [<Top 10 most beautiful ancient towns in China>scenic attractions, <how many have you been to?>engaging question]projected and contextualized message[< >crowds/tourists]contextual information
Agreement A logical match is found between an engaging question on scenic spots/attractions and the emoji indicating the context of the question, namely crowds and tourism ‘ ’ (crowds/people)
Functions To complement verbal message; to visually present the contextual and circumstantial information of the message
Pattern Non-tonal complement: the emoji ‘ ’ serves as non-sentimental information to help contextualize and specify the text

The last category, i.e. creative illustration (Table 11), includes idiosyncratic usage and creative expression, which are made possible by online platforms where the conventional use of language is constantly being challenged. Users are free to combine text and emojis however they like with whatever logical rules they prefer. We avoid using terms that imply sequencing or linearity because even though the established patterns mostly follow linear order, it is possible to formulate non-linear mixtures of text and emojis. As diversified as this structure may be, this pattern is rare in our data mainly because the posts we have examined are popular ones that are composed for the general public on Weibo to read and are therefore less challenging in intelligibility.

Table 11.Creative illustration
Componential Patterning
Example
 
…脑补
nǎo bǔ
make up
故事
gù shì
story

…making up a story’

Meaning [<