2.1Multimodal studies and the metafunctional approach

Building on the metafunctional approach of systemic-functional theories (see Halliday 1994Halliday, Michael A. K. 1994 An Introduction to Functional Grammar (2nd edition). London: Edward Arnold.), multimodal analysis involves non-linguistic and paralinguistic resources for meaning-making (Kress and van Leeuwen 2001Kress, Gunther, and Theo V. van Leeuwen 2001 Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold., 2006Kress, Gunther, and Theo V. van Leeuwen 2006 Reading Images: The Grammar of Visual Design (2nd Edition). London: Routledge. ; Baldry and Thibault 2006Baldry, Anthony, and Paul J. Thibault 2006 Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Coursebook. London: Equinox.). The metafunctional approach has the potential to account for all modes and forms of meaning-making, offering a top-down frame in which anatomic analyses can be conducted (O’Halloran 2004O’Halloran, Kay (ed) 2004 Multimodal Discourse Analysis: Systemic Functional Perspectives. London: Continuum.; Kress and van Leeuwen 2006Kress, Gunther, and Theo V. van Leeuwen 2006 Reading Images: The Grammar of Visual Design (2nd Edition). London: Routledge. ; Painter et al. 2013Painter, Clare, James Martin, and Leonard Unsworth 2013 Reading Visual Narratives: Image Analysis of Children’s Picture Books. London: Equinox.; Jewitt et al. 2016Jewitt, Carey, Jeff Bezemer, and Kay O’Halloran 2016 Introducing Multimodality. London: Routledge. ).

In language use or any form of communication, there are three metafunctions that reflect different aspects of meaning-making: the ideational metafunction construes experience of the world, either with linguistic coding or other semiotic display; the interpersonal metafunction covers social and attitudinal aspects of the discourse; the textual metafunction is concerned with coherence and cohesion in discourse (Halliday 1994Halliday, Michael A. K. 1994 An Introduction to Functional Grammar (2nd edition). London: Edward Arnold.; Halliday and Hasan 1976; Kress and van Leeuwen 2001Kress, Gunther, and Theo V. van Leeuwen 2001 Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold., 2006Kress, Gunther, and Theo V. van Leeuwen 2006 Reading Images: The Grammar of Visual Design (2nd Edition). London: Routledge. ; Baldry and Thibault 2006Baldry, Anthony, and Paul J. Thibault 2006 Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Coursebook. London: Equinox.). In the context of multimodal discourse, the ideational metafunction is also referred to as representational, the interpersonal as interactive, and the textual as compositional (Kress and van Leeuwen 2006Kress, Gunther, and Theo V. van Leeuwen 2006 Reading Images: The Grammar of Visual Design (2nd Edition). London: Routledge. ; Baldry and Thibault 2006Baldry, Anthony, and Paul J. Thibault 2006 Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Coursebook. London: Equinox.; Jewitt 2014Jewitt, Carey (ed) 2014 The Routledge Handbook of Multimodal Analysis. (2nd edition). London: Routledge.; Jewitt et al. 2016Jewitt, Carey, Jeff Bezemer, and Kay O’Halloran 2016 Introducing Multimodality. London: Routledge. ). Although different terms are used to better capture the expanded potential and modal variety of discourse components and realization, the essence of each metafunction remains relatively unchanged.

Text-emoji co-occurrences comply with the three metafunctions via multimodal meaning-making. They are multimodal in nature since emojis are usually seen as pictograms, providing a refreshing visual alternative to plain text. Thus the patterns and communicative functions of text-emoji mode mixtures are best studied within a multimodal framework. In multimodal analysis, mode is not strictly defined, and is usually used to refer to a set of socially and culturally shaped resources for meaning-making (MODE 2012MODE 2012 Glossary of Multimodal Terms. (Retrieved 21-02-2020 via https://​multimodalityglossary​.wordpress​.com​/mode​-2/); Jewitt 2014Jewitt, Carey (ed) 2014 The Routledge Handbook of Multimodal Analysis. (2nd edition). London: Routledge.). Emojis on their own can also be seen as multimodal since they may highlight imagistic or verbal meaning-making processes by activating iconic or symbolic semiotic channels. Emojis can thus be subcategorized into iconic emojis (e.g. smileys), symbolic emojis (e.g. horoscope/star signs), and emojis which contain both iconic and symbolic features. In this study, emojis are viewed as a multimodal component (alongside text) but are not further classified.

Past research on emojis and social media has touched upon various aspects of text-emoji mode mixing. The relations between emojis and text have been found to be mostly complementary or substitutional (Cramer et al. 2016Cramer, Henriette, Paloma de Juan, and Joel Tetreault 2016 “Sender-intended Functions of Emojis in US Messaging.” Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services: 504–509. New York: Association for Computing Machinery. ; Ai et al. 2017Ai, Wei, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei 2017 “Untangling Emoji Popularity Through Semantic Embeddings.” Proceedings of the Eleventh International Conference on Web and Social Media (ICWSM 2017), 2–11. Palo Alto: The AAAI Press. (Retrieved 21-02-2020 via https://​www​.aaai​.org​/ocs​/index​.php​/ICWSM​/ICWSM17​/paper​/view​/15705​/14788); Danesi 2017Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. ; Ge and Herring 2018Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) ). Ai et al. (2017)Ai, Wei, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei 2017 “Untangling Emoji Popularity Through Semantic Embeddings.” Proceedings of the Eleventh International Conference on Web and Social Media (ICWSM 2017), 2–11. Palo Alto: The AAAI Press. (Retrieved 21-02-2020 via https://​www​.aaai​.org​/ocs​/index​.php​/ICWSM​/ICWSM17​/paper​/view​/15705​/14788) find that emojis representing entities usually act as replacement to words, whereas emojis expressing sentiments tend to co-occur with and complement text with attitudes and tones. Danesi (2017)Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. studies text message exchanges containing emojis and groups the pragmatic functions of emojis into two main categories: adding tone and injecting a positive mood, while other functions such as salutation, punctuation, and irony are also discussed.

Regarding the pragmatic values of text-emoji strings, Herring and Dainas (2017Herring, Susan C., and Ashley Dainas 2017 “Nice Picture Comment!. In Graphicons in Facebook Comment Threads.” Proceedings of the 50th Hawaii International Conference on System Sciences, 2185–2194. Los Alamitos, CA: IEEE. , 2187) have identified six pragmatic uses of graphicons (emojis, stickers, etc.): mention, reaction, riff, tone modification, action, and narrative sequence. While their study focuses on conversational exchanges and not the co-occurrence of text and emojis, it offers a comprehensive series of steps for investigating mixed modes. Ge and Herring (2018)Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) focus on the speech acts of emojis and the rhetorical relations between emojis and their accompanying text. Ge and Gretzel (2018)Ge, Jing, and Ulrike Gretzel 2018 “Emoji Rhetoric: A Social Media Influencer Perspective.” Journal of Marketing Management 34 (15–16): 1272–1295. have examined the emoji-based communicative strategies of Sina Weibo influencers.

In text-emoji co-occurrences, emojis play a critical role in creating and modifying meaning. Different emojis may shade the same verbal message with different sentiments and attitudes; different verbal messages may activate different senses of the same emoji (Wijeratne et al. 2016Wijeratne, Sanjaya, Lakshika Balasuriya, Amit Sheth, and Derek Doran 2016 “Emojinet: Building a Machine Readable Sense Inventory for Emoji.” In SocInfo 2016 Social Informatics, ed. by Emma Spiro, and Yong-Yeol Ahn, 527–541. N.p.: Springer, Cham.; Hakami 2017Hakami, Shatha Ali A. 2017 “The Importance of Understanding Emoji: an Investigative Study.” Research Topics in HCI (Spring 2017).). Such usages of language in combination with emojis point to an integrated stratum of meaning which combines verbal text and graphical emojis. This stratum is most effectively studied from a metafunctional framework which explores representational, interactive, and compositional aspects of communication.

2.2Special features of Weibo emojis

Unlike emoticons which consist of ASCII symbols and usually require readers to tip their heads from one side (e.g. :D) to the other (e.g. < 3) to grasp the meaning of the horizontal display, emojis are easier to view with their upright and colorful design. Previous studies have compared the two alongside other graphicons such as stickers and custom images (Herring and Dainas 2017Herring, Susan C., and Ashley Dainas 2017 “Nice Picture Comment!. In Graphicons in Facebook Comment Threads.” Proceedings of the 50th Hawaii International Conference on System Sciences, 2185–2194. Los Alamitos, CA: IEEE. ; Rodrigues et al. 2018Rodrigues, David, Marilia Prada, Rui Gaspar, Margarida V. Garrido, and Diniz Lopes 2018 “Lisbon Emoji and Emoticon Database (LEED): Norms for Emoji and Emoticons in Seven Evaluative Dimensions.” Behavior Research Methods 50: 392–405. ; Tang and Hew 2018Tang, Ying, and Khe Foon Hew 2018 “Emoticon, Emoji, and Sticker Use in Computer-Mediated Communications: Understanding Its Communicative Function, Impact, User Behavior, and Motive.” In New Media for Educational Change, ed. by Liping Deng, Will W. K. Ma, and Cheuk Wai Rose Fong, 191–201. Singapore: Springer. ; de Seta 2018de Seta, Gabriele 2018 “Biaoqing: The Circulation of Emoticons, Emoji, Stickers, and Custom Images on Chinese Digital Media Platforms.” First Monday 23 (9). (Retrieved 21-02-2020 via http://​firstmonday​.org​/article​/view​/9391​/7566) ). Because emojis reflect users’ emotions and attitudes in digital communication, they have been considered as multimodal tokens in sentiment analysis and opinion mining (Wolf 2000Wolf, Alecia 2000 “Emotional Expression Online: Gender Differences in Emoticon Use.” CyberPsychology & Behavior 3 (5): 827–833. ; Krohn 2004Krohn, Franklin B. 2004 “A Generational Approach to Using Emoticons as Non-verbal Communication.” Journal of Technical Writing and Communication 43: 321–328. ; Churches et al. 2009Churches, Owen, Simon Baron-Cohen, and Howard Ring 2009 “Seeing Face-like Objects: An Event-related Potential Study.” Neuroreport 20 (14): 1290–1294. ; Zhao et al. 2012Zhao, Jichang, Li Dong, Junjie Wu, and Ke Xu 2012 “Moodlens: An Emoticon-based Sentiment Analysis System for Chinese Tweets.” Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, 1528–1531. New York: Association for Computing Machinery. ; Schnoebelen 2012Schnoebelen, Tyler 2012 “Do You Smile with Your Nose? Stylistic Variation in Twitter Smoticons.” University of Pennsylvania Working Papers in Linguistics 18 (2): Article 14.; Novak et al. 2015Novak, Petra Kralj, Jasmina Smailović, Borut Sluban, and Igor Mozetič 2015 “Sentiment of Emojis.” PloS one 10 (12): e0144296.).

While some emojis can be understood across languages and cultures, using emoji as part of the communicative language remains culturally and linguistically dependent. Speakers from different regions or of different languages may prefer different emojis or usages (Lotherington and Xu 2004Lotherington, Heather, and Yejun Xu 2004 “How to Chat in English and Chinese: Emerging Digital Language Conventions.” ReCALL 16 (2): 308–329. ; Ljubešic and Fišer 2016Ljubešic, Nikola, and Darja Fišer 2016 “A Global Analysis of Emoji Usage.” Proceedings of the 10th Web as Corpus Workshop, 82–89. Association for Computational Linguistics. (Retrieved 18-02-2020 via https://​www​.aclweb​.org​/anthology​/W16​-2610​.pdf) ; Lu et al. 2016Lu, Xuan, Wei Ai, Xuanzhe Liu, Qian Li, Ning Wang, Gang Huang, and Qiaozhu Mei 2016 “Learning from the Ubiquitous Language: An Empirical Analysis of Emoji Usage of Smartphone Users.” Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, 770–780. New York: Association for Computing Machinery. ; Sampietro 2019Sampietro, Agnese 2019 “Emoji and Rapport Management in Spanish WhatsApp Chats.” Journal of Pragmatics 143: 109–120. ). The interpretation of emojis as part of multimodal discourse, especially alongside co-presented text, is influenced by the text, the linguistic features, and the context (see Halliday and Hasan 1989Halliday, Michael A. K., and Ruquiya Hasan 1989 Language, Context, and Text: Aspects of Language in a Social-semiotic Perspective. Oxford: Oxford University Press.; Stöckl 2004Stöckl, Hartmut 2004 “In between Modes: Language and Image in Printed Media.” In Perspectives on multimodality, ed. by Eija Ventola, Classily Charles, and Martin Kaltenbacher, 9–30. Amsterdam/Philadelphia: John Benjamins Publishing Company. ; Coulson 2006Coulson, Seana 2006 “Constructing Meaning.” Metaphor and Symbol 21(4): 245–266. ).

According to the Unicode Consortium, the Unicode of each emoji is composed of a Unicode string that identifies it across platforms, and a linguistic description of the emoji’s graphic representation (Miller et al. 2017Miller, Hannah, Daniel Kluver, Jacob Thebault-Spieker, Loren Terveen, and Brent Hecht 2017 “Understanding Emoji Ambiguity in Context: The Role of Text in Emoji-Related Miscommunication.” Proceedings of the Eleventh International Conference on Web and Social Media (ICWSM 2017), 152–161. Palo Alto: The AAAI Press. (Retrieved 21-02-2020 via https://​aaai​.org​/ocs​/index​.php​/ICWSM​/ICWSM17​/paper​/view​/15703​/14804); Hakami 2017Hakami, Shatha Ali A. 2017 “The Importance of Understanding Emoji: an Investigative Study.” Research Topics in HCI (Spring 2017).; see also Davis and Edberg 2019Davis, Mark, and Peter Edberg 2019 Unicode Emoji-unicode Technical Report #51. Technical Report 51 (Version 12.0). (Retrieved 21-02-2020 via http://​unicode​.org​/reports​/tr51/)). Since pictorial appearance is not standardized by the Unicode, the same emoji would have different graphic renderings when displayed on different digital platforms (See Emojipedia).44. http://​emojipedia​.org/

The emojis available on Weibo are different from emojis that are registered on Emojipedia. While Unicode emojis could be posted on Weibo through smartphones or other devices that have various input systems, Weibo has a set of idiosyncratic emojis specifically designed for use on Weibo. A Weibo emoji, similar to a Unicode emoji, has a pictorial appearance and a verbal description. Unlike other Unicode emojis, however, some particular aspects of Weibo emojis include the following (see Section 4.1. for examples):

1. The verbal description is usually in Chinese.

2. Certain emojis have strong Chinese cultural or linguistic features.

3. Certain emojis are animated with wiggling/flickering effects when displayed on Weibo.55.In this study moving emojis are still viewed as emojis because (1) the moving effect of emojis on Weibo is simplistic and serves to highlight the emojis, and (2) the moving feature is not held across platforms and once reproduced outside Weibo they will mostly become static (like in this paper).

Some Weibo emojis are integrated with the Chinese language and culture, and particular to the communication of users from the online Chinese community. Despite their distinctive features and extensive use, Weibo emojis (and their interaction with text) have been comparatively less studied than Unicode emojis.

3.1Data collection

Our data set was collected over a period of over three months (February 1 to May 20, 2017) via the column of Hot Weibo.66.Some posts in our data set might no longer be accessible through the search engine on Weibo now because some accounts have been set so they only show posts in the past 6–12 months, and some posts may have been removed by the users. We manually collected Hot Weibo posts that contain both text and emoji(s). The posts were mainly from verified accounts of news agencies, websites, celebrities or other popular media accounts which have large numbers of followers. While online influencers may have contributed extensively to the pool of popular posts, different from other studies which have targeted online influencers’ accounts (Ge and Gretzel 2018Ge, Jing, and Ulrike Gretzel 2018 “Emoji Rhetoric: A Social Media Influencer Perspective.” Journal of Marketing Management 34 (15–16): 1272–1295. ), our approach to data collection relied on the popularity of the post as the main criterion.

As we primarily focus on text-emoji co-occurrences, we confine the data and analyses to the two types of mode/component: Chinese text and emojis. Posts in which text and emojis co-occur, but whose meaning cannot be understood without referring to other co-presented modes, are excluded to maintain semantic completeness of text-emoji co-occurrences. Altogether we obtained a total of 408 Chinese text-emoji mode mixing posts or strings as our data set.77.Since the posts were publicly accessible when we collected them, the ones containing names have not been excluded. In the examples reproduced in this paper, the names mentioned in the posts are replaced with initial letters for ethical concerns. Each text-emoji co-occurrence is seen as an independent unit in the analysis. While not large, still we consider this data set to be representative as the posts, which were popular among users, were collected over a period of over three months.

Some of the Weibo posts may contain a title/headline, which is usually bracketed (‘【】’), ahead of its main body. Emojis may appear in the title and/or main body of a post. Since the title and the main body have a hierarchical relation, they are collected separately.88.When there are emoji(s) in both title and text of the same post, the post will be divided and counted as two units of text-emoji co-occurrence. About 3.8% of the posts originally collected are of this type. Within the title or main body, the strings are not further segmented to provide a more holistic picture of emojis used in each post. The title or main body of a Weibo post that contains both text and emoji(s), therefore, is a unit of analysis.

In this study, while emoji-only strings are not discussed at length, it should be noted that emojis have great potential for making meaning, even without assistance from verbal text (see Xu 2012Xu, Bing 2012 Di Shu [The book of earth]. Guangxi: Guangxi Normal University Press.; Mrowiec 2016Mrowiec, Anna 2016 How to Speak Emoji Love. London: Ebury Press.; Herring and Dainas 2017Herring, Susan C., and Ashley Dainas 2017 “Nice Picture Comment!. In Graphicons in Facebook Comment Threads.” Proceedings of the 50th Hawaii International Conference on System Sciences, 2185–2194. Los Alamitos, CA: IEEE. ).

3.2Framework of analysis

We study the patterns and communicative functions manifested by emojis mainly by focusing on the representational, interactive, and compositional features of emojis in text-emoji co-occurrences. Viewing text and emojis as two components, we propose a group of componential patterns to capture some recurring pragmatic and functional structures in text-emoji co-occurrences.99.Emojis could be regarded as a mode for its visual consistency in display even though the semiotic nature may range from predominantly iconic to predominantly symbolic in different items.

The representational features of emojis are analyzed by first looking at visual-semantic representations of single emoji items on Weibo, and second by grouping and systematizing the patterns in which text and emojis co-occur based on the number, type, and positioning of emojis. The number of units is taken as the frequency of different patterns.

The interactive features of emojis when co-occurring with text are approached from their emotion-arousing implications and their potential for facilitating social exchanges (see Churches et al. 2009Churches, Owen, Simon Baron-Cohen, and Howard Ring 2009 “Seeing Face-like Objects: An Event-related Potential Study.” Neuroreport 20 (14): 1290–1294. , 2014Churches, Owen, Mike Nicholls, Myra Thiessen, Mark Kohler, and Hannah Keage 2014 “Emoticons in Mind: An Event-related Potential Study.” Social Neuroscience 9 (2): 196–202. ; Jaeger and Ares 2017Jaeger, Sara R., and Gastón Ares 2017 “Dominant Meanings of Facial Emoji: Insights from Chinese Consumers and Comparison with Meanings from Internet Resources.” Food Quality and Preference 62: 275–283. ; Ge and Gretzel 2018Ge, Jing, and Ulrike Gretzel 2018 “Emoji Rhetoric: A Social Media Influencer Perspective.” Journal of Marketing Management 34 (15–16): 1272–1295. ). Like other forms of communication, emojis could be used to convey attitudes and realize interpersonal exchanges (Gibson et al. 2018Gibson, Will, Pingping Huang, and Qianyun Yu 2018 “Emoji and Communicative Action: The Semiotics, Sequence and Gestural Actions of ‘Face Covering Hand’.” Discourse, Context & Media 26: 91–99. ; Aull 2019Aull, Bethany 2019 “A Study of Phatic Emoji Use in WhatsApp Communication.” Internet Pragmatics 2 (2): 206–232. ; Bai et al. 2019Bai, Qiyu, Qi Dan, Zhe Mu, and Maokun Yang 2019 “A Systematic Review of Emoji: Current Research and Future Perspectives.” Frontiers in Psychology 10 (2019): Article 2221. ). We draw upon the speech act theory in highlighting the illocutionary forces, i.e. the forces to perform acts, of emojis when they are co-presented with text (see Austin 1962Austin, John Langshaw 1962 How to Do Things with Words. London: Oxford University Press.; Searle 1976Searle, John R. 1976 “A Classification of Illocutionary Acts.” Language in Society 5 (1): 1–23. ; Bach and Harnish 1979Bach, Kent, and Robert M. Harnish 1979 Linguistic Communication and Speech Acts. Cambridge: The MIT Press.; Yule 1996Yule, George 1996 Pragmatics. Oxford: Oxford University Press.; Skovholt et al. 2014Skovholt, Karianne, Anette Grønning, and Anne Kankaanranta 2014 “The Communicative Functions of Emoticons in Workplace E-Mails::-).” Journal of Computer-Mediated Communication 19 (4): 780–797. ).

The compositional features of text-emoji co-occurrences are based on the visual coherence and reflected by the cohesive patterns in which text and emojis jointly realize the componential integration (see Halliday and Hasan 1985/1976Halliday, Michael A. K., and Ruquiya Hasan 1985/1976 Cohesion in English. London: Longman.; Kress and van Leeuwen 2006Kress, Gunther, and Theo V. van Leeuwen 2006 Reading Images: The Grammar of Visual Design (2nd Edition). London: Routledge. ). The componential patterns displayed in the data reflect the compositional structures of text-emoji co-occurrences and showcase the visual interface of pragmatic functions and practices on Weibo.

Overall, the metafunctional framework of multimodal analysis is top-down as it outlines the three strata in which the actual meaning-building and pragmatic functions unfurl. Within this ternary structure, theories of meaning construction, speech acts, and textual cohesion are employed to interpret the multi-layered pragmatics of emojis.

4.1Visual-semantic features of Weibo emojis

Compared with emoji sets in iOS and on Twitter, the most conspicuously special Weibo emojis are those that have been integrated with the symbolic meaning of the Chinese characters, such as ‘ ’ (神马 shén mǎ, homonymous to 什么 shén me ‘what’) and ‘ ’ (V5 V-wǔ, homonymous to 威武 wēi wǔ ‘awesome’). Other unique Weibo emojis include those that represent Chinese traditions and concepts, such as ‘ ’ (红包 hóng bāo ‘red packet’), and those that reflect online memes and trending words/expressions, such as ‘ ’ (二哈 èr hā ‘silly dog husky’) and ‘ ’ (互粉 hù fěn ‘to follow each other/become each other’s online friend on Weibo’).1010.Red packets are red envelopes containing money that are given to children during the Chinese New Year. Husky dogs are believed to be troublemakers because they could be overly energetic, especially when kept indoors. See Zhang (2017)Zhang, Yi 2017 “The Semiotic Multifunctionality of Arabic Numerals in Chinese Online Discourse.” Language@Internet 14: Article 2. for more discussions on the integration of Arabic numerals and Chinese on the internet, such as the V5 emoji. Furthermore, some seemingly common emojis have acquired new senses on Weibo, such as ‘ ’ (再见 zài jiàn ‘bye’). Instead of bidding farewell, this emoji could be used to express discontentment or cold shouldering on Weibo. For example, in “别人家的男朋友 bié rén jiā de nán péng yǒu ‘(Look at) Other people’s boyfriends’ ”, the emoji is attached to express the will to part ways with the unsatisfactory boyfriend (probably) of the one posting because others’ boyfriends have done better. Note that although there are some Unicode emojis that are from built-in input systems of various terminals, most of the emojis found in our data come from the emoji set on Weibo.

4.2Visual patterns of text-emoji co-occurrences

Emojis are typically found to be either following a sentence or substituted into the sentence (Cramer et al. 2016Cramer, Henriette, Paloma de Juan, and Joel Tetreault 2016 “Sender-intended Functions of Emojis in US Messaging.” Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services: 504–509. New York: Association for Computing Machinery. ; Ai et al. 2017Ai, Wei, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei 2017 “Untangling Emoji Popularity Through Semantic Embeddings.” Proceedings of the Eleventh International Conference on Web and Social Media (ICWSM 2017), 2–11. Palo Alto: The AAAI Press. (Retrieved 21-02-2020 via https://​www​.aaai​.org​/ocs​/index​.php​/ICWSM​/ICWSM17​/paper​/view​/15705​/14788)). We have found that in a text-emoji co-occurrence, the text is interspersed with emoji(s). Table 1 shows some examples of posts with varied text-emoji representational patterns.1111.The examples are shown in Chinese and pinyin, with word-by-word (sans person, number, tense, and some particles) and overall English translations.

To classify the representational patterns of text-emoji co-presentation in posts, we have examined the number and type of emojis when they appear together with text. In a text-emoji co-occurrence, wherever emojis appear, there might be a single emoji (single), multiple identical emojis (replicated), or multiple different emojis (mixed), forming an emoji grouplet (which consists of a single emoji or multiple emojis).1212.The word “grouplet” is used as an umbrella term to refer to any instance of emoji(s) that is preceded and/or followed by text within a unit. A grouplet does not contain text. Meanwhile, the positions of emoji grouplet(s) in a co-occurrence unit include onset (one grouplet at the beginning of the text), inset (one grouplet in the middle of the text), final (one grouplet at the end of the text), and multiple emoji grouplets embedded in different positions of the text. The labels “single”, “replicated”, and “mixed” are used to indicate the type of an emoji grouplet; the labels “onset”, “inset”, “final”, and “multiple” are used to categorize text-emoji co-occurrence units based on the positioning of emoji grouplet(s).

Here we propose a diagram (Diagram 1) to categorize text-emoji co-occurrences along four representational dimensions: (1) number of emoji grouplets in one co-occurrence unit, (2) positioning of emoji grouplets in one co-occurrence unit, (3) number of emojis in one grouplet, and (4) type of emojis in one grouplet. This categorization reflects the visual patterns of text-emoji co-occurrences.

The distribution of different representational categories is shown in Table 2. Overall, approximately 77.5% of text-emoji co-occurrences only have emoji(s) at the end (the “Final” category), of which about 80% only have a single emoji. Within the “Multiple (occurrences)” category, 86.4% of the posts contain final-positioned emoji(s).

A contingency table (Table 3) is given to summarize the detailed frequencies of different emoji grouplets and their positions in different categories of text-emoji co-occurrence. In the “Multiple (occurrences)” category, one unit contains multiple emoji grouplets; other categories only contain one emoji grouplet. Altogether, there are 486 emoji grouplets in the data.

In general, the distribution is skewed towards the final positioning of emoji(s) and the single emoji occurrence. When inserting multiple emojis as one grouplet, the users prefer repeating the same emoji over mixing different emojis. We have observed a common representational pattern of “text followed by emoji(s)” in the co-occurrence units, which suggests that text is the principal method of coding information in these posts, and that emoji(s) contribute additional meaning to the preceding text. This might be in line with earlier studies in which emojis have been found to carry complementary or supplementary meaning to accompanying words (see Danesi 2017Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. ; Ai et al. 2017Ai, Wei, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei 2017 “Untangling Emoji Popularity Through Semantic Embeddings.” Proceedings of the Eleventh International Conference on Web and Social Media (ICWSM 2017), 2–11. Palo Alto: The AAAI Press. (Retrieved 21-02-2020 via https://​www​.aaai​.org​/ocs​/index​.php​/ICWSM​/ICWSM17​/paper​/view​/15705​/14788); Ge and Herring 2018Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) ).

Based on the metafunctional approach of multimodal analysis and the cognitive meaning-constructing model raised by Coulson (2006)Coulson, Seana 2006 “Constructing Meaning.” Metaphor and Symbol 21(4): 245–266. , the readers rely on the imagistic feature of emojis to decide the meaningful and structural linkages between the emojis and the accompanying text. In real communication, a single emoji is found to be understood in different senses (Wijeratne et al. 2016Wijeratne, Sanjaya, Lakshika Balasuriya, Amit Sheth, and Derek Doran 2016 “Emojinet: Building a Machine Readable Sense Inventory for Emoji.” In SocInfo 2016 Social Informatics, ed. by Emma Spiro, and Yong-Yeol Ahn, 527–541. N.p.: Springer, Cham.). However, different from what has been suggested in some previous studies, we do not consider this potential for ambiguity to be an obstacle in analyzing emoji-embedded text. Rather, we view the multimodal meaning-making of text-emoji as a compositionally based interactive process that allows for flexibility in terms of representational expression. In other words, the multimodal meaning of text-emoji co-occurrence is not only represented by its visual form, but is also generated by functional relations between text and emoji(s). The communicative functions of text-emoji co-occurrences are further examined from interactive and compositional perspectives.

5.1Speech acts of emojis

Any utterance in actual use carries three layers of force to indicate the implied act: locutionary, illocutionary, and perlocutionary (Austin 1962Austin, John Langshaw 1962 How to Do Things with Words. London: Oxford University Press.; Searle 1969/2011Searle, John R. 1969/2011 Speech Acts: An Essay in the Philosophy of Language. Cambridge: Cambridge University Press. ; Yule 1996Yule, George 1996 Pragmatics. Oxford: Oxford University Press.). Like words, emojis and emoticons can be used to perform speech acts and facilitate communication (Dresner and Herring 2010Dresner, Eli, and Susan C. Herring 2010 “Functions of the Nonverbal in CMC: Emoticons and Illocutionary Force.” Communication Theory 20 (3): 249–268. , 2014Dresner, Eli, and Susan C. Herring 2014 “Emoticons and Illocutionary Force.” In Perspectives on Theory of Controversies and the Ethics of Communication, ed. by Dana Riesenfeld, and Giovanni Scarafile, 59–70. Springer Netherlands. ; Vandergriff 2013Vandergriff, Ilona 2013 “Emotive Communication Online: a Contextual Analysis of Computer-mediated Communication (CMC) cues.” Journal of Pragmatics 51: 1–12. ; Ge and Herring 2018Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) ; dos Reis et al. 2018dos Reis, Julio C., Rodrigo Bonacin, Heiko H. Hornung, and M. Cecília C. Baranauskas 2018 “Intenticons: Participatory Selection of Emoticons for Communication of Intentions.” Computers in Human Behavior 85: 146–162. ). Ge and Herring (2018)Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) build on the speech act taxonomy in computer-mediated discourse (see Herring et al. 2005Herring, Susan C., Anupam Das, and Shashikant Penumarthy 2005 CMC Act Taxonomy. (Retrieved 21-02-2020 via http://​ella​.slis​.indiana​.edu​/~herring​/cmc​.acts​.html)) to categorize the illocutionary acts of emoji sequences on Sina Weibo, discovering that claim, desire, and explain are among the most frequently used acts.

In this study, we highlight the illocutionary force of emojis, which is usually activated by semantic content of the co-occurring text. The referent of the illocutionary act could be objects or notions mentioned in co-occurring text, or textual meaning in general. The acting power of emojis is examined in the context of text-emoji co-occurrence because when an act is being executed by the emoji(s), it is part of the message that is delivered by the text-emoji co-occurrence. We draw upon the general classifications of illocutionary acts and discussions of perlocutionary intentions proposed by Bach and Harnish (1979)Bach, Kent, and Robert M. Harnish 1979 Linguistic Communication and Speech Acts. Cambridge: The MIT Press. and Searle (1976)Searle, John R. 1976 “A Classification of Illocutionary Acts.” Language in Society 5 (1): 1–23. to label the acts exemplified here (Table 4).

On Weibo, one distinctive emoji that has been used to conduct speech acts is the lit candle emoji ‘ ’, which expresses condolences or R.I.P. in posts that contain messages of casualty, mourning, or obituary. In such cases the lit candle emoji carries the illocutionary force of sympathizing or expressing condolences towards the deceased that is often understood from the text or context. On the receiving end, the emoji could effectuate the perlocutionary force of making the readers believe that whoever posts the emoji is mourning and thus appear to be compassionate.

5.2Highlighted subjective interpretation

Re-posting is frequently seen on social media. We have noticed that when re-posting news briefings, those re-posting, either agencies or individual users, might change emojis in the re-posted news while maintaining the verbal text, reflecting the posters’ subjective interpretations of the re-post. Examples are given in Table 5.1313.Example (6) was accessed from the Hot Weibo column. The posts in Table 5 are not in chronological order.

It can be observed from Table 5 that the reported speech of the rule-breaking driver “我以为可以 (wǒ yǐ wéi kě yǐ) ‘I thought it was allowed’” has maintained both its verbal content and its relative position to emoji component, while the rest of the verbal information has undergone some minor changes. In general, the information conveyed by text has been left undisturbed in all the re-posts. The emojis included, however, are drastically different.

In Example (6) the emoji depicts a dismissive character that does not seem to take the incident seriously, thereby implying the referent of the emoji is the driver, since the poster of the news is unlikely to be indifferent to the violation of traffic law. In Example (7) an emoji of ashamed face is used after the verbal text, indicating that the driver was mortified of the incident. In Example (8) the emoji features a shocked/overwhelmed face that, instead of a troublemaker, looks like a victim.

All three emojis in Table 5 have complemented the verbal information. However, the different emojis have exerted considerable influence over the original post, supplying different complementary information and judgement about the driver’s reaction towards the incident. Simply attaching different emojis can alter how the same incident is interpreted. While emojis generally highlight subjective stance and attitude, those emojis in different re-posts more obviously reveal different attitudes.

5.3Enhanced informality

Another important and probably more general interactive and pragmatic function of emojis is their power in promoting informality and accessibility. In digital communication emojis are usually associated with informality and a casual tone, and they are often assumed to be used more frequently among peers, friends, and relatives (Danesi 2017Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. ). On Weibo, emojis are not only used by ordinary users but have also been incorporated into news briefs and short reports that are publicized by officially acknowledged accounts/users (see Zhou and Pan 2016Zhou, Huiquan, and Quanxiao Pan 2016 “Information, Community, and Action on Sina-Weibo: How Chinese Philnatropic NGOs Use Social Media.” VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations 27 (5): 2433–2457. ). Once emojis are embedded, the over-all tone of the message becomes less formal and sometimes more approachable, especially with serious genres such as news brief (see Table 4 and 5).

While news reports are supposed to appear authoritative and formal, online platforms such as Weibo have to some extent mollified the rigid impression, partly by inserting emojis to make the news publishers appear more approachable to their audience. The use of text-emoji co-occurrences on Weibo has encouraged and created more informal and casual exchanges and contexts. The inclusion of emojis also helps make the news posts stand out and compete for readers’ attention along with commercial and entertaining posts.

6.1Compositional relations

The textual or compositional meaning is manifested primarily through the coherence and cohesion of the discourse (Halliday and Hasan 1985/1976Halliday, Michael A. K., and Ruquiya Hasan 1985/1976 Cohesion in English. London: Longman.; Kress and van Leeuwen 2006Kress, Gunther, and Theo V. van Leeuwen 2006 Reading Images: The Grammar of Visual Design (2nd Edition). London: Routledge. ). On the platform of Weibo, text-emoji co-presentation is but part of the entire discourse, which contains other multi-semiotic resources (see Baldry and Thibault 2006Baldry, Anthony, and Paul J. Thibault 2006 Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Coursebook. London: Equinox.; Herring and Dainas 2017Herring, Susan C., and Ashley Dainas 2017 “Nice Picture Comment!. In Graphicons in Facebook Comment Threads.” Proceedings of the 50th Hawaii International Conference on System Sciences, 2185–2194. Los Alamitos, CA: IEEE. ). Here we focus on the interaction between Chinese text and accompanying emoji(s), observing how their co-dependence contribute to coherence and cohesion. This part leads to the establishment of the componential patterning of text-emoji co-occurrences by specifying compositional relations between text and emojis, and identifying the functions of emojis in the co-occurrences.

Coherence and cohesion refer to structural and semantic ties that are found among different components of the text (see Halliday and Hasan 1985/1976Halliday, Michael A. K., and Ruquiya Hasan 1985/1976 Cohesion in English. London: Longman.; Halliday 1994Halliday, Michael A. K. 1994 An Introduction to Functional Grammar (2nd edition). London: Edward Arnold.; Kress and van Leeuwen 2001Kress, Gunther, and Theo V. van Leeuwen 2001 Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold., 2006Kress, Gunther, and Theo V. van Leeuwen 2006 Reading Images: The Grammar of Visual Design (2nd Edition). London: Routledge. ; Baldry and Thibault 2006Baldry, Anthony, and Paul J. Thibault 2006 Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Coursebook. London: Equinox.). The expansion of componential modality to include emojis means that the realization of coherence and cohesion is also expanded. The compositional relations are based on representational and interpersonal meanings of text-emoji co-occurrences, engaging visual attention and constructing coherence and logical relations.

Cohesive devices commonly found in textual discourse include reference, substitution, and lexical cohesion (Halliday and Hasan 1985/1976Halliday, Michael A. K., and Ruquiya Hasan 1985/1976 Cohesion in English. London: Longman., 29). Taking into consideration the multimodal features and meaning-making potential of emojis, we have observed mainly four types of compositional relations between text and emojis in the data: substitution (where emojis are used to substitute lexical or phrasal items in the text, e.g. ‘ ’ for ‘flight’), co-reference (where emojis and verbal elements refer to the same meaning or semantically related concepts, e.g. ‘ ’ and ‘cake/birthday’), complement (where emojis are used to code additional tones or other information to verbal text), and others (where the patterning of text and emojis cannot be identified by the previous three types).

These patterns are basically in line with the findings from other emoji studies that used English or Chinese-based examples, namely that emojis could be used to complement, reinforce, and replace words, and that emojis could be used on their own to tell stories (Danesi 2017Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. ; Herring and Dainas 2017Herring, Susan C., and Ashley Dainas 2017 “Nice Picture Comment!. In Graphicons in Facebook Comment Threads.” Proceedings of the 50th Hawaii International Conference on System Sciences, 2185–2194. Los Alamitos, CA: IEEE. ; Ai et al. 2017Ai, Wei, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei 2017 “Untangling Emoji Popularity Through Semantic Embeddings.” Proceedings of the Eleventh International Conference on Web and Social Media (ICWSM 2017), 2–11. Palo Alto: The AAAI Press. (Retrieved 21-02-2020 via https://​www​.aaai​.org​/ocs​/index​.php​/ICWSM​/ICWSM17​/paper​/view​/15705​/14788); Ge and Herring 2018Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) ). However, unlike previous emoji-related analyses, we use a componential approach to sketch the interactive patterns between text and emojis, and distinguish between different uses of complementary emojis – external tonal, internal tonal, and non-tonal.

6.2Componential patterning

We view text and emoji as two types of component in making meaning on Weibo. Combining the visual patterns and pragmatic functions revealed so far, we propose a set of quasi-compositional rules that would incorporate the communicative features of text-emoji co-occurrences. Since the compositional relations are built between text and emojis, they are termed as componential patterns, indicating that the two different components are seen as interdependent chunks in forming collective meaning.

The basic componential patterning of text-emoji co-occurrence takes shape according to the aforementioned compositional relations, and is manifested in five basic types: componential substitution, inter-modal reinforcement, tonal complement (external and internal), non-tonal complement, and creative illustration. Here the categories are divided and re-named to reflect not only the cohesive ties, but also the forms and functions of text-emoji co-occurrences, as well as the relevant findings in previous studies (see Danesi 2017Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. ; Ai et al. 2017Ai, Wei, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei 2017 “Untangling Emoji Popularity Through Semantic Embeddings.” Proceedings of the Eleventh International Conference on Web and Social Media (ICWSM 2017), 2–11. Palo Alto: The AAAI Press. (Retrieved 21-02-2020 via https://​www​.aaai​.org​/ocs​/index​.php​/ICWSM​/ICWSM17​/paper​/view​/15705​/14788)). The five basic structures assign different roles to text and emojis to build a multimodal layer of meaning. Within the basic structures, text and emojis have the capacity to formulate various representations and perform interactive functions.

Before elaborating on the basic patterns, key terms of componential patterning are defined and explained as follows:

1. (Componential) Meaning: the meaning of text and the meaning of emojis. The meanings examined in componential patterns are usually labeled in generic and categorical manner, leading to logical and semantic relations between different components.

2. (Componential) Agreement: the compatibility between the meaning of text and the meaning of emojis. Agreement can be found along a continuum from entirely identical meaning between text and emoji (which may never be achieved), to related meanings or concepts, to logical matches. Note that obvious discrepancies between text and emojis in their co-occurrence would probably render rhetorical meanings such as sarcasm.

3. (Componential) Functions: the contextualized pragmatic functions that reflect the representational, interactive, and compositional aspects of text-emoji co-occurrences.

4. Projection: the reference to, reflection of, or quotation of a certain (part of) component. For example, an emoji could be used to project the information mentioned in the accompanying text, or to project the role of the poster.

In the following analyses, the subscripts of angle brackets (‘< >_{componential meaning}’) describe the componential meaning of the component, and the subscripts of square brackets (‘[ ]_{componential agreement}’) describe the agreements between text and emojis (as two components) that are pinpointed in each case.1414.Most of the examples in this section contain a single emoji to reflect the predominant representational pattern in text-emoji co-occurrences on Weibo. Multiple identical emojis in one text-emoji co-occurrence could be interpreted as holding visual emphasis; multiple different emojis would be analyzed separately (see Ge and Herring 2018Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) ).

In componential substitution (Table 6), the meaning of the whole message is usually not complete without including emoji(s) as part of the verbal text. The role of emojis compensates for the obligatory part of the text in line with the grammatical patterns of Chinese. If emojis appear after a noun, they are likely to assume an attributive or modifying function. If an emoji co-occurs with a verb, it likely fills in an adverbial gap.

In inter-modal reinforcement (Table 7), the emojis usually repeat or reinforce the notion/object mentioned in the text. Namely, the text or part of the text and the emojis refer to the same or similar concepts. Since different people may associate the same emoji with different meanings, the exact pairing of inter-modal reinforcement might be debatable sometimes. It is, however, mostly straightforward when part of the information conveyed by text is repeated or reinforced by co-referring emojis.

In tonal complement and non-tonal complement, emojis encode meanings beyond the co-presented text, and provide tonal/sentimental/attitudinal or other types of information alongside the text, which then usually becomes (partly) projected. Here the text and emojis do not refer to the same concept but complement each other to formulate a larger meaningful unit. Functions such as tone-adding/modifying are included in the tonal category (see Danesi 2017Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. ; Herring and Dainas 2017Herring, Susan C., and Ashley Dainas 2017 “Nice Picture Comment!. In Graphicons in Facebook Comment Threads.” Proceedings of the 50th Hawaii International Conference on System Sciences, 2185–2194. Los Alamitos, CA: IEEE. ). Functions such as adding contextual or other non-tonal/non-sentimental information are included in the non-tonal category.

Based on our data, the category of tonal complement could be further divided into two sub-categories. External tonal complement (Table 8) refers to the cases in which emojis are used to express the post writer’s personal attitude, while internal tonal complement (Table 9) indicates the cases where emojis depict the attitudes or tonal features of the characters or concepts mentioned in the co-occurring text. Either way, it should be noted that the use of emojis is mostly a subjective choice.

Complementary information other than tones or attitudes, such as contextual and other circumstantial information, is covered by the category of non-tonal complement (Table 10).

The last category, i.e. creative illustration (Table 11), includes idiosyncratic usage and creative expression, which are made possible by online platforms where the conventional use of language is constantly being challenged. Users are free to combine text and emojis however they like with whatever logical rules they prefer. We avoid using terms that imply sequencing or linearity because even though the established patterns mostly follow linear order, it is possible to formulate non-linear mixtures of text and emojis. As diversified as this structure may be, this pattern is rare in our data mainly because the posts we have examined are popular ones that are composed for the general public on Weibo to read and are therefore less challenging in intelligibility.

While most of the componential links could be categorized based on the relations of the emojis and the text which have clear pragmatically functioning patterns or tend to recur in the data set, there are also creative links established by individual preferences. Although the general componential structures should apply to make the co-occurrences intelligible to others, the co-occurring patterns may change and new templates or structures might be able to emerge and stabilize through creative associations.

Of all the co-occurrence units we have examined, nearly 90% of them contain only one type of componential patterns. For the cases with one type of componential function, the distribution of the patterns is summarized in Diagram 2.

External tonal complement (71.2%) appears to be the most popular pattern of text-emoji co-occurrence on Weibo. Another commonly seen pattern is inter-modal reinforcement (23.6%), and the other three patterns only play minor roles in the larger picture (internal tonal complement 1.6%; non-tonal complement 1.6%; componential substitution 1.4%; creative illustration 0.6%).

While both supplementary (replacement/substitution) and complementary (co-presentation) uses of emojis (see Ai et al. 2017Ai, Wei, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei 2017 “Untangling Emoji Popularity Through Semantic Embeddings.” Proceedings of the Eleventh International Conference on Web and Social Media (ICWSM 2017), 2–11. Palo Alto: The AAAI Press. (Retrieved 21-02-2020 via https://​www​.aaai​.org​/ocs​/index​.php​/ICWSM​/ICWSM17​/paper​/view​/15705​/14788)) are seen in our data, our findings show that the supplementary use is less popular on Weibo, and the complementary use could be divided into different patterns depending on the nature and target of modification. Adding emojis as complementary message to convey the poster’s attitude or opinion seems to be the common reason to use emojis in our data set.

Similar to Ge and Herring (2018)Ge, Jing, and Susan C. Herring 2018 “Communicative Functions of Emoji Sequences on Sina Weibo.” First Monday 23 (11). (Retrieved 21-02-2020 via https://​firstmonday​.org​/ojs​/index​.php​/fm​/article​/view​/9413​/7610) ’s category of “evaluation (using emojis to give subjective comments)”, our notion of tonal complement represents one of the prominent usages of emojis alongside text on Weibo. However, the category that takes up the highest percentage in Ge and Herring’s study, “restatement”, is not comparable to any of our categories since most of our inter-modal reinforcement cases would likely be interpreted as “partial repetition” which holds no rhetorical significance in their framework. The discrepancy might have been due to the different purposes of study and criteria used for data collection.

Function-wise, the category of inter-modal reinforcement in our study may correspond partly to Herring and Dainas (2017)Herring, Susan C., and Ashley Dainas 2017 “Nice Picture Comment!. In Graphicons in Facebook Comment Threads.” Proceedings of the 50th Hawaii International Conference on System Sciences, 2185–2194. Los Alamitos, CA: IEEE. ’s category of “mention” based on data from Facebook, although their study is not confined to the co-occurrences of text and emojis. Also, their category of “tone” resembles our proposal of tonal complement. Our findings have verified the popularity of these two types of pragmatic functions also observed in their study. It seems that the patterns may hold to some extent for both Chinese and English text with emojis.

In general, emojis are mainly used to complement or reinforce textual information, with intentions to highlight certain points, perform acts, and build more informal and usually less authoritative voices and exchanges. Meanwhile, emojis also carry other pragmatic functions such as punctuating and attention seeking that could also be attributed to the interpersonal and interactive aspect of text-emoji co-occurrence (see Sampietro 2016Sampietro, Agnese 2016 “Exploring the Punctuating Effect of Emoji in Spanish WhatsApp Chats.” Lenguas Modernas 47: 91–113.; Danesi 2017Danesi, Marcel 2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London and New York: Bloomsbury Publishing. ).