Despite a growing awareness of methodological issues, the literature on appraisal has not so far provided adequate answers
to some of the key challenges involved in reliably identifying and classifying evaluative language expressions. This article
presents a stepwise method for the manual annotation of appraisal in text that is designed to optimize reliability,
replicability and transparency. The procedure consists of seven steps, from the creation of a context-specific annotation manual
to the statistical analysis of the quantitative data derived from the manually-performed annotations. By presenting this method,
the article pursues the twofold purpose of (i) providing a practical tool that can facilitate more reliable, replicable and
transparent analyses, and (ii) fostering a discussion of the best practices that should be observed when manually annotating
appraisal.
Article outline
1.Introduction
2.Challenges in annotating appraisal
2.1Challenges in identifying appraisal
2.2Challenges in classifying appraisal
2.3Reliability, replicability, transparency
3.The stepwise annotation method
3.1General principles
Principle 1.All choices should be accounted for
Principle 2.The annotation guidelines should be tested and refined until maximum reliability is achieved
Principle 3.Reliability should always be assessed, and reliability scores reported and discussed
3.2An outline of the steps
3.2.1Step 1: Define the scope of the project
3.2.2Step 2: Select and configure an annotation tool
3.2.3Step 3: Draft a context-specific annotation manual
Artstein, Ron & Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics 34(4). 555–596.
Baayen, R. Harald. 2008. Analyzing linguistic data: A practical introduction to Statistics using R. Cambridge: CUP.
Bartalesi Lenzi, Valentina, Giovanni Moretti & Rachele Sprugnoli. 2012. CAT: the CELCT Annotation Tool. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk & Stelios Piperidis (eds.), Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC12), Istanbul: European Language Resources Association (ELRA). Available online at: [URL] [Accessed 22 September 2015]
Bednarek, Monika. 2006. Evaluation in media discourse: Analysis of a newspaper corpus. London: Continuum.
Bednarek, Monika. 2008. Emotion talk across corpora. Basingstoke: Palgrave.
Bednarek, Monika. 2014. An astonishing season of destiny! Evaluation in blurbs used for advertising TV series. In Geoff Thompson & Laura Alba-Juez (eds.), 197–220.
Ben-Aaron, Diana. 2005. Given and news: Evaluation in newspaper stories about national anniversaries. Text & Talk 25(5). 691–718.
Carretero, Marta & Maite Taboada. 2014. Graduation within the scope of Attitude in English and Spanish consumer reviews of books and movies. In Geoff Thompson & Laura Alba-Juez (eds.), 221–239.
Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1). 37–46.
Cozby, Paul & Scott Bates. 2011. Methods in behavioral research, 11th edn. New York, NY: McGraw-Hill.
Dice, Lee R.1945. Measures of the amount of ecologic association between species. Ecology 26(3). 297–302.
Divjak, Dagmar. 2006. Ways of intending: Delineating and structuring near synonyms. In Stefan Th. Gries & Anatol Stefanowitsch (eds.), Corpora in cognitive linguistics: Corpus-based approaches to syntax and lexis, 19–56. Berlin: Mouton.
Don, Alexanne. 2007. An approach to the analysis of textual identity through profiles of evaluative disposition. In Proceedings of the Australian Systemic Functional Linguistics Association 2007 Conference. Available at: [URL]. [Accessed 22 September 2015]
Fuoli, Matteo. 2012. Assessing social responsibility: A quantitative analysis of Appraisal in BP’s and IKEA’s social reports. Discourse & Communication 6(1). 55–81.
Fuoli, Matteo & Dylan Glynn. 2013. Computer-assisted manual annotation of evaluative language expressions: Bridging discourse and corpus approaches. Paper presented at the Evaluative Language and Corpus Linguistics Workshop
, Lancaster University, 22 July 2013.
Fuoli, Matteo & Charlotte Hommerberg. 2015. Optimizing transparency, reliability and replicability: Annotation principles and inter-coder agreement in the quantification of evaluative expressions. Corpora 10(3). 315–349.
Fuoli, Matteo & Carita Paradis. 2014. A model of trust-repair discourse. Journal of Pragmatics 741. 52–69.
Geeraerts, Dirk, Stefan Grondelaers & Peter Bakema. 1994. The structure of lexical variation. Berlin: Mouton.
Hood, Susan. 2004. Appraising research: Taking a stance in academic writing. Sydney: University of Technology PhD thesis.
Hunston, Susan. 2004. Counting the uncountable: Problems of identifying evaluation in a text and in a corpus. In Alan Partington, John Morley & Louann Haarman (eds.), Corpora and Discourse, 157–188. Bern: Peter Lang.
Hunston, Susan. 2011. Corpus approaches to evaluation: Phraseology and evaluative language. New York, NY: Taylor & Francis.
Kirk, Jerome & Marc L. Miller. 1986. Reliability and validity in qualitative research. Beverly Hills, CA: Sage.
Krippendorf, Klaus. 2004. Content Analysis: An introduction to its methodology. Thousand Oaks, CA: Sage.
Lipovsky, Caroline. 2008. Constructing affiliation and solidarity in job interviews. Discourse & Communication 2(4). 411–432.
Lipovsky, Caroline. 2011. It’s really a great presentation!: Appraising candidates in job interviews. Linguistics and the Human Sciences 4(2). 161–185.
Lipovsky, Caroline. 2013. Negotiating ones expertise through appraisal in CVs. Linguistics and the Human Sciences 8(3). 307–333.
Mackay, James & Jean Parkinson. 2009. My very own mission impossible: An APPRAISAL analysis of student teacher reflections on a design and technology project. Text & Talk 29(6). 729–753.
Macken-Horarik, Mary & Anne Isaac. 2014. Appraising Appraisal. In Geoff Thompson & Laura Alba-Juez (eds.), 67–92.
Marshall, Christine, Ralph Adendorff & Vivian de Klerk. 2010. The role of APPRAISAL in the NRF Rating System: An analysis of Judgement and Appreciation in peer reviewers’ reports. Southern African Linguistics and Applied Language Studies 27(4). 391–412.
Martin, J. R.2000. Beyond exchange: Appraisal systems in English. In Susan Hunston & Geoff Thompson (eds.), Evaluation in text: Authorial stance and the construction of discourse, 142–175. Oxford: OUP.
Martin, J. R. & David Rose. 2003. Working with discourse: Meaning beyond the clause. London: Continuum.
Martin, J. R. & Peter R. R. White. 2005. The language of evaluation: Appraisal in English. London: Palgrave Macmillan.
Mauranen, Anna & Marina Bondi. 2003. Evaluative language use in academic discourse. Journal of English for Academic Purposes 2(4). 269–271.
O’Donnell, Mick. 2008. Demonstration of the UAM CorpusTool for text and image annotation. Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Demo Session. 13–16.
O’Donnell, Mick. 2012. Appraisal analysis and the computer. Revista Canaria de Estudios Ingleses 651. 115–130.
O’Donnell, Mick. 2014. Exploring identity through Appraisal Analysis: A corpus annotation methodology. Linguistics and the Human Sciences 9(1). 95–116.
Page, Ruth E.2003. An analysis of APPRAISAL in childbirth narratives with special consideration of gender and storytelling style. Text & Talk 23(2). 211–237.
Paradis, Carita. 2001. Adjectives and boundedness. Cognitive Linguistics 12(1). 47–64.
Paradis, Carita, Joost van de Weijer, Caroline Willners & Magnus Lindgren. 2012. Evaluative polarity of antonyms. Lingue e linguaggio 11(2). 199–214.
Pounds, Gabrina. 2010. Attitude and subjectivity in Italian and British hardnews reporting: The construction of a culture-specific ‘reporter’ voice. Discourse Studies 12(1). 106–137.
Pounds, Gabrina. 2011. This property offers much character and charm: Evaluation in the discourse of online property advertising. Text & Talk 31(2). 195–220.
Pustejovsky, James & Amber Stubbs. 2012. Natural language annotation for machine learning. Sebastopol, CA: O’Reilly Media.
Read, Jonathon & John Carroll. 2012. Annotating expressions of Appraisal in English. Language Resources and Evaluation 46(3). 421–447.
Ryshina-Pankova, Marianna. 2014. Exploring argumentation in course-related blogs through ENGAGEMENT. In Geoff Thompson & Laura Alba-Juez (eds.), 281–302.
Santamaría-García, Carmen. 2014. Evaluative discourse and politeness in university students’ communication through social networking sites. In Geoff Thompson & Laura Alba-Juez (eds.), 387–411.
Taboada, Maite & Marta Carretero. 2012. Contrastive analyses of evaluation in text: Key issues in the design of an annotation system for ATTITUDE applicable to consumer reviews in English and Spanish. Linguistics and the Human Sciences 6(1–3). 275–295.
Thompson, Geoff. 2014. AFFECT and emotion, target-value mismatches, and Russian dolls: Refining the APPRAISAL model. In Geoff Thompson & Laura Alba-Juez (eds.), 47–66.
Thompson, Geoff & Laura Alba-Juez (eds.). 2014. Evaluation in context. Amsterdam: Benjamins.
Thompson, Geoff & Susan Hunston. 2000. Evaluation: An introduction. In Susan Hunston & Geoff Thompson (eds.), Evaluation in text: Authorial stance and the construction of discourse, 1–27. Oxford: OUP.
2024. Functions of idioms in English as lingua franca: An appraisal system account. International Journal of Applied Linguistics
Geng, Yifan, Gong Chen & Ming Li
2024. Analyzing engagement strategies in argument chain: A comparison between high- and low-scoring EFL undergraduate argumentative essays. Journal of English for Academic Purposes 71 ► pp. 101428 ff.
Hidalgo-Downing, Laura, Paula Pérez-Sobrino, Laura Filardo-Llamas, Carmen Maíz-Arévalo, Begoña Núñez-Perucha, Alfonso Sánchez-Moya & Julia T. Williams Camus
2024. Audience design and pragmatic conceptions of moves and upvotes during advice-giving on Reddit. Journal of Pragmatics 219 ► pp. 30 ff.
Sun, Shuyi Amelia & Feng (Kevin) Jiang
2024. “The results might not fully represent…”: Negation in the limitations sections of doctoral theses by Chinese and American students. Text & Talk
Sun, Shuyi Amelia, Feng Kevin Jiang & Yanhua Liu
2024. “Maybe, but probably not”: A cross-disciplinary study of negation in Three Minute Thesis presentations. English for Specific Purposes 74 ► pp. 117 ff.
Trnavac, Radoslava & Nele Põldvere
2024. Investigating Appraisal and the Language of Evaluation in Fake News Corpora. Corpus Pragmatics 8:2 ► pp. 107 ff.
Xu, Qingxin
2024. Identifying disputants’ attitudinal variations in family mediations: A data mining approach. Discourse Studies
Alwohaibi, Hala Abdulrahman & Hesham Suleiman Alyousef
2023. An Investigation of Engagement Resources in the World Cup 2022 Newspaper Articles in the Arab World Media. Journal of Contemporary Language Research 2:2 ► pp. 118 ff.
Carretero, Marta
2023. ‘Noticeably Better / Visibly Shaken’: The Use of Four Adverbs of High Perceivability with Meanings of Attitude. <i>WORD</i> 69:2 ► pp. 117 ff.
Cheng, Shi
2023. A review of interpersonal metafunction studies in systemic functional linguistics (2012–2022). Journal of World Languages
2023. Evaluation Mechanism of Political Discourse: A Holistic Approach. Journal of Psycholinguistic Research 52:6 ► pp. 2143 ff.
Trnavac, Radoslava & Maite Taboada
2023. Engagement and constructiveness in online news comments in English and Russian. Text & Talk 43:2 ► pp. 235 ff.
Watanabe, Hideo
2023. The discursive construction of a conflict: a case of disputed islands in the East China Sea. Text & Talk 43:3 ► pp. 333 ff.
Yu, Wei
2023. Negotiation of justice: the discursive construction of attitudinal positioning in bilingual legal judgments of HKSAR v KWAN WAN KI
. International Journal of Legal Discourse 8:2 ► pp. 299 ff.
2022. Discursive Strategies of Blaming: The Language of Judgment and Political Protest Online. Social Media + Society 8:4 ► pp. 205630512211387 ff.
Hidalgo-Downing, Laura & Paula Pérez-Sobrino
2022. Developing an annotation protocol for evaluative stance and metaphor in discourse: theoretical and methodological considerations. Text & Talk 0:0
2022. The US-China battle over Coronavirus in the news media: Metaphor transfer as a representation of stance mediation. Discourse & Society 33:4 ► pp. 456 ff.
Shrestha, Prithvi N.
2022. Examining evaluative language used in assessment feedback on business students’ academic writing. Assessing Writing 54 ► pp. 100664 ff.
Simaki, Vasiliki, Eleni Seitanidi & Carita Paradis
2022. Evaluating stance annotation of Twitter data. Research in Corpus Linguistics 11:1 ► pp. 53 ff.
Sun, Shuyi & Peter Crosthwaite
2022. “Establish a niche” vianegation: A corpus-based study ofnegationwithin the Move 2 sections of PhD thesis introductions. Open Linguistics 8:1 ► pp. 189 ff.
Sun, Shuyi Amelia & Peter Crosthwaite
2022. “The findings might not be generalizable”: Investigating negation in the limitations sections of PhD theses across disciplines. Journal of English for Academic Purposes 59 ► pp. 101155 ff.
2021. The Language of Evaluation in the Narratives by the Magdalene Laundries Survivors: The Discourse of Female Victimhood. Applied Linguistics 42:2 ► pp. 315 ff.
Mavis Ho, Nga-Ki
2021. Transcreation in marketing: a corpus-based study of persuasion in optional shifts from English to Chinese. Perspectives 29:3 ► pp. 426 ff.
2020. The SFU Opinion and Comments Corpus: A Corpus for the Analysis of Online News Comments. Corpus Pragmatics 4:2 ► pp. 155 ff.
Ross, Andrew S. & David Caldwell
2020. ‘Going negative’: An APPRAISAL analysis of the rhetoric of Donald Trump on Twitter. Language & Communication 70 ► pp. 13 ff.
Van Herck, Rebecca, Sofie Decock & Bernard De Clerck
2020. “Can you send us a PM please?” Service recovery interactions on social media from the perspective of organizational legitimacy. Discourse, Context & Media 38 ► pp. 100445 ff.
Fusari, Sabrina
2019. “Logically, We Quite Agree with the IARC”: Negotiating Interpersonal Meaning in a Corpus of Scientific Texts. In Specialized Discourses and Their Readerships [The M.A.K. Halliday Library Functional Linguistics Series, ], ► pp. 25 ff.
Geng, Yifan & Sue Wharton
2019. How do thesis writers evaluate their own and others’ findings? An appraisal analysis and a pedagogical intervention. English for Specific Purposes 56 ► pp. 3 ff.
2018. A cross-linguistic analysis of the ‘homework’ metaphor in German and English political discourse. Discourse & Society 29:6 ► pp. 609 ff.
This list is based on CrossRef data as of 7 september 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.