A stepwise method for annotating appraisal

Fuoli, Matteo

doi:10.1075/fol.15016.fuo

Article published In:

Functions of Language
Vol. 25:2 (2018) ► pp.229–258

A stepwise method for annotating appraisal

Matteo Fuoli | University of Birmingham

Despite a growing awareness of methodological issues, the literature on appraisal has not so far provided adequate answers to some of the key challenges involved in reliably identifying and classifying evaluative language expressions. This article presents a stepwise method for the manual annotation of appraisal in text that is designed to optimize reliability, replicability and transparency. The procedure consists of seven steps, from the creation of a context-specific annotation manual to the statistical analysis of the quantitative data derived from the manually-performed annotations. By presenting this method, the article pursues the twofold purpose of (i) providing a practical tool that can facilitate more reliable, replicable and transparent analyses, and (ii) fostering a discussion of the best practices that should be observed when manually annotating appraisal.

Article outline

1.Introduction
2.Challenges in annotating appraisal
- 2.1Challenges in identifying appraisal
- 2.2Challenges in classifying appraisal
- 2.3Reliability, replicability, transparency
3.The stepwise annotation method
- 3.1General principles
  - Principle 1.All choices should be accounted for
  - Principle 2.The annotation guidelines should be tested and refined until maximum reliability is achieved
  - Principle 3.Reliability should always be assessed, and reliability scores reported and discussed
- 3.2An outline of the steps
  - 3.2.1Step 1: Define the scope of the project
  - 3.2.2Step 2: Select and configure an annotation tool
  - 3.2.3Step 3: Draft a context-specific annotation manual
  - 3.2.4Step 4: Assess reliability
  - 3.2.5Step 5: Refine the annotation manual
  - 3.2.6Step 6: Annotate the corpus
  - 3.2.7Step 7: Analyze the results
4.Conclusion
Acknowledgements
Notes
References

Published online: 19 October 2018

https://doi.org/10.1075/fol.15016.fuo

References (55)

References

Artstein, Ron & Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics 34(4). 555–596.

Baayen, R. Harald. 2008. Analyzing linguistic data: A practical introduction to Statistics using R. Cambridge: CUP.

Bartalesi Lenzi, Valentina, Giovanni Moretti & Rachele Sprugnoli. 2012. CAT: the CELCT Annotation Tool. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk & Stelios Piperidis (eds.), Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC12), Istanbul: European Language Resources Association (ELRA). Available online at: [URL] [Accessed 22 September 2015]

Bednarek, Monika. 2006. Evaluation in media discourse: Analysis of a newspaper corpus. London: Continuum.

. 2008. Emotion talk across corpora. Basingstoke: Palgrave.

. 2009. Language patterns and ATTITUDE. Functions of Language 16(2). 165–192.

. 2014. An astonishing season of destiny! Evaluation in blurbs used for advertising TV series. In Geoff Thompson & Laura Alba-Juez (eds.), 197–220.

Ben-Aaron, Diana. 2005. Given and news: Evaluation in newspaper stories about national anniversaries. Text & Talk 25(5). 691–718.

Carretero, Marta & Maite Taboada. 2014. Graduation within the scope of Attitude in English and Spanish consumer reviews of books and movies. In Geoff Thompson & Laura Alba-Juez (eds.), 221–239.

Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1). 37–46.

Cozby, Paul & Scott Bates. 2011. Methods in behavioral research, 11th edn. New York, NY: McGraw-Hill.

Dice, Lee R. 1945. Measures of the amount of ecologic association between species. Ecology 26(3). 297–302.

Divjak, Dagmar. 2006. Ways of intending: Delineating and structuring near synonyms. In Stefan Th. Gries & Anatol Stefanowitsch (eds.), Corpora in cognitive linguistics: Corpus-based approaches to syntax and lexis, 19–56. Berlin: Mouton.

Don, Alexanne. 2007. An approach to the analysis of textual identity through profiles of evaluative disposition. In Proceedings of the Australian Systemic Functional Linguistics Association 2007 Conference. Available at: [URL]. [Accessed 22 September 2015]

Fuoli, Matteo. 2012. Assessing social responsibility: A quantitative analysis of Appraisal in BP’s and IKEA’s social reports. Discourse & Communication 6(1). 55–81.

Fuoli, Matteo & Dylan Glynn. 2013. Computer-assisted manual annotation of evaluative language expressions: Bridging discourse and corpus approaches. Paper presented at the Evaluative Language and Corpus Linguistics Workshop , Lancaster University, 22 July 2013.

Fuoli, Matteo & Charlotte Hommerberg. 2015. Optimizing transparency, reliability and replicability: Annotation principles and inter-coder agreement in the quantification of evaluative expressions. Corpora 10(3). 315–349.

Fuoli, Matteo & Carita Paradis. 2014. A model of trust-repair discourse. Journal of Pragmatics 741. 52–69.

Geeraerts, Dirk, Stefan Grondelaers & Peter Bakema. 1994. The structure of lexical variation. Berlin: Mouton.

Glynn, Dylan. 2009. Polysemy, syntax, and variation: A usage-based method for Cognitive Semantics. In Vyvyan Evans & Stéphanie Pourcel (eds.), New Directions in Cognitive Linguistics, 77–104. Amsterdam: Benjamins.

Gries, Stefan Th. 1999. Particle movement: A cognitive and functional approach. Cognitive Linguistics 10(2). 105–146.

2013. Statistics for linguistics with R: A practical introduction. Berlin: Mouton.

Hommerberg, Charlotte & Alexanne Don. 2015. Appraisal and the language of wine appreciation: A critical discussion of the potential of the Appraisal framework as a tool to analyse specialised genres. Functions of Language 22(2). 161–191.

Hood, Susan. 2004. Appraising research: Taking a stance in academic writing. Sydney: University of Technology PhD thesis.

Hunston, Susan. 2004. Counting the uncountable: Problems of identifying evaluation in a text and in a corpus. In Alan Partington, John Morley & Louann Haarman (eds.), Corpora and Discourse, 157–188. Bern: Peter Lang.

. 2011. Corpus approaches to evaluation: Phraseology and evaluative language. New York, NY: Taylor & Francis.

Kirk, Jerome & Marc L. Miller. 1986. Reliability and validity in qualitative research. Beverly Hills, CA: Sage.

Krippendorf, Klaus. 2004. Content Analysis: An introduction to its methodology. Thousand Oaks, CA: Sage.

Lipovsky, Caroline. 2008. Constructing affiliation and solidarity in job interviews. Discourse & Communication 2(4). 411–432.

. 2011. It’s really a great presentation!: Appraising candidates in job interviews. Linguistics and the Human Sciences 4(2). 161–185.

. 2013. Negotiating ones expertise through appraisal in CVs. Linguistics and the Human Sciences 8(3). 307–333.

Mackay, James & Jean Parkinson. 2009. My very own mission impossible: An APPRAISAL analysis of student teacher reﬂections on a design and technology project. Text & Talk 29(6). 729–753.

Macken-Horarik, Mary & Anne Isaac. 2014. Appraising Appraisal. In Geoff Thompson & Laura Alba-Juez (eds.), 67–92.

Marshall, Christine, Ralph Adendorff & Vivian de Klerk. 2010. The role of APPRAISAL in the NRF Rating System: An analysis of Judgement and Appreciation in peer reviewers’ reports. Southern African Linguistics and Applied Language Studies 27(4). 391–412.

Martin, J. R. 2000. Beyond exchange: Appraisal systems in English. In Susan Hunston & Geoff Thompson (eds.), Evaluation in text: Authorial stance and the construction of discourse, 142–175. Oxford: OUP.

Martin, J. R. & David Rose. 2003. Working with discourse: Meaning beyond the clause. London: Continuum.

Martin, J. R. & Peter R. R. White. 2005. The language of evaluation: Appraisal in English. London: Palgrave Macmillan.

Mauranen, Anna & Marina Bondi. 2003. Evaluative language use in academic discourse. Journal of English for Academic Purposes 2(4). 269–271.

O’Donnell, Mick. 2008. Demonstration of the UAM CorpusTool for text and image annotation. Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Demo Session. 13–16.

. 2012. Appraisal analysis and the computer. Revista Canaria de Estudios Ingleses 651. 115–130.

. 2014. Exploring identity through Appraisal Analysis: A corpus annotation methodology. Linguistics and the Human Sciences 9(1). 95–116.

Page, Ruth E. 2003. An analysis of APPRAISAL in childbirth narratives with special consideration of gender and storytelling style. Text & Talk 23(2). 211–237.

Paradis, Carita. 2001. Adjectives and boundedness. Cognitive Linguistics 12(1). 47–64.

Paradis, Carita, Joost van de Weijer, Caroline Willners & Magnus Lindgren. 2012. Evaluative polarity of antonyms. Lingue e linguaggio 11(2). 199–214.

Pounds, Gabrina. 2010. Attitude and subjectivity in Italian and British hardnews reporting: The construction of a culture-specific ‘reporter’ voice. Discourse Studies 12(1). 106–137.

. 2011. This property offers much character and charm: Evaluation in the discourse of online property advertising. Text & Talk 31(2). 195–220.

Pustejovsky, James & Amber Stubbs. 2012. Natural language annotation for machine learning. Sebastopol, CA: O’Reilly Media.

Read, Jonathon & John Carroll. 2012. Annotating expressions of Appraisal in English. Language Resources and Evaluation 46(3). 421–447.

Ryshina-Pankova, Marianna. 2014. Exploring argumentation in course-related blogs through ENGAGEMENT. In Geoff Thompson & Laura Alba-Juez (eds.), 281–302.

Santamaría-García, Carmen. 2014. Evaluative discourse and politeness in university students’ communication through social networking sites. In Geoff Thompson & Laura Alba-Juez (eds.), 387–411.

Taboada, Maite & Marta Carretero. 2012. Contrastive analyses of evaluation in text: Key issues in the design of an annotation system for ATTITUDE applicable to consumer reviews in English and Spanish. Linguistics and the Human Sciences 6(1–3). 275–295.

Taboada, Maite, Marta Carretero & Jennifer Hinnell. 2014. Loving and hating the movies in English, German and Spanish. Languages in Contrast 14(1). 127–161.

Thompson, Geoff. 2014. AFFECT and emotion, target-value mismatches, and Russian dolls: Refining the APPRAISAL model. In Geoff Thompson & Laura Alba-Juez (eds.), 47–66.

Thompson, Geoff & Laura Alba-Juez (eds.). 2014. Evaluation in context. Amsterdam: Benjamins.

Thompson, Geoff & Susan Hunston. 2000. Evaluation: An introduction. In Susan Hunston & Geoff Thompson (eds.), Evaluation in text: Authorial stance and the construction of discourse, 1–27. Oxford: OUP.

Cited by (50)

Cited by 50 other publications

Order by:

Allami, Hamid, Monica Karlsson & Hamid Reza Shahroosvand

2024. Functions of idioms in English as lingua franca: An appraisal system account. International Journal of Applied Linguistics 34:4 ► pp. 1709 ff.

Cheng, Shi

2024. A review of interpersonal metafunction studies in systemic functional linguistics (2012–2022). Journal of World Languages 10:3 ► pp. 623 ff.

Eguchi, Masaki & Kristopher Kyle

2024. Building custom NLP tools to annotate discourse-functional features for second language writing research: A tutorial. Research Methods in Applied Linguistics 3:3 ► pp. 100153 ff.

Geng, Yifan, Gong Chen & Ming Li

2024. Analyzing engagement strategies in argument chain: A comparison between high- and low-scoring EFL undergraduate argumentative essays. Journal of English for Academic Purposes 71 ► pp. 101428 ff.

Hidalgo-Downing, Laura & Paula Pérez-Sobrino

2024. Developing an annotation protocol for evaluative stance and metaphor in discourse: theoretical and methodological considerations. Text & Talk 44:2 ► pp. 197 ff.

Hidalgo-Downing, Laura, Paula Pérez-Sobrino, Laura Filardo-Llamas, Carmen Maíz-Arévalo, Begoña Núñez-Perucha, Alfonso Sánchez-Moya & Julia T. Williams Camus

2024. A protocol for the annotation of evaluative stance and metaphor across four discourse genres. Revista Española de Lingüística Aplicada/Spanish Journal of Applied Linguistics 37:2 ► pp. 486 ff.

Hsu, Chan-Chia, Yu-Yun Chang & Yun Biao

2024. Evaluative language in Chinese online food discourse: overall distribution, local patterning, and readers’ involvement. Text & Talk

Jung, Yejin, Dana Gablasova, Vaclav Brezina & Hanna Schmück

2024. Developing a coding scheme for annotating opinion statements in L2 interactive spoken English with application for language teaching and assessment. Research in Corpus Linguistics 12:2 ► pp. 146 ff.

Li, Long, Shoshana Dreyfus & Alexanne Don

2024. An appraisal analysis of judgement values on a Chinese political web forum. Text & Talk

Lu, Rickey

2024. Audience design and pragmatic conceptions of moves and upvotes during advice-giving on Reddit. Journal of Pragmatics 219 ► pp. 30 ff.

Parviz, Muhammed, Alireza Jalilifar & Alexanne Don

2024. Unpacking Evaluative Meaning-Making Resources in Verbal Movie Descriptions: A Comparative Study of Native and Non-Native English Speakers. Sage Open 14:4

Song, Jiannan

2024. Representing state identity with journalistic attitudes: a corpus-assisted linguistic analysis of CGTN’s trade disputes coverage. Círculo de Lingüística Aplicada a la Comunicación 100 ► pp. 181 ff.

Sun, Shuyi Amelia & Feng (Kevin) Jiang

2024. “The results might not fully represent…”: Negation in the limitations sections of doctoral theses by Chinese and American students. Text & Talk

Sun, Shuyi Amelia, Feng Kevin Jiang & Yanhua Liu

2024. “Maybe, but probably not”: A cross-disciplinary study of negation in Three Minute Thesis presentations. English for Specific Purposes 74 ► pp. 117 ff.

Trnavac, Radoslava & Nele Põldvere

2024. Investigating Appraisal and the Language of Evaluation in Fake News Corpora. Corpus Pragmatics 8:2 ► pp. 107 ff.

Xu, Qingxin

2024. Identifying disputants’ attitudinal variations in family mediations: A data mining approach. Discourse Studies 26:5 ► pp. 693 ff.

Alwohaibi, Hala Abdulrahman & Hesham Suleiman Alyousef

2023. An Investigation of Engagement Resources in the World Cup 2022 Newspaper Articles in the Arab World Media. Journal of Contemporary Language Research 2:2 ► pp. 118 ff.

Carretero, Marta

2023. ‘Noticeably Better / Visibly Shaken’: The Use of Four Adverbs of High Perceivability with Meanings of Attitude. <i>WORD</i> 69:2 ► pp. 117 ff.

Cibulskienė, Jurga

2023. Scepticism voiced through extended metaphors. Metaphor and the Social World 13:2 ► pp. 197 ff.

Domingo, Cecilia

2023. INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION, ► pp. 731 ff.

Jing, Yi & Qingxin Xu

2023. Danmu video as the resemiotised: An attitudinal analysis of Home Visit. Media, Culture & Society 45:4 ► pp. 753 ff.

Makki, Mohammad & Andrew S. Ross

2023. “We were cocked & loaded to retaliate”. Journal of Language Aggression and Conflict 11:1 ► pp. 1 ff.

Sun, Jing & Zhenqian Liu

2023. Evaluation Mechanism of Political Discourse: A Holistic Approach. Journal of Psycholinguistic Research 52:6 ► pp. 2143 ff.

Trnavac, Radoslava & Maite Taboada

2023. Engagement and constructiveness in online news comments in English and Russian. Text & Talk 43:2 ► pp. 235 ff.

Watanabe, Hideo

2023. The discursive construction of a conflict: a case of disputed islands in the East China Sea. Text & Talk 43:3 ► pp. 333 ff.

Yu, Wei

2023. Negotiation of justice: the discursive construction of attitudinal positioning in bilingual legal judgments of HKSAR v KWAN WAN KI . International Journal of Legal Discourse 8:2 ► pp. 299 ff.

Andersson, Marta

2022. ‘So many “virologists” in this thread!’. Pragmatics. Quarterly Publication of the International Pragmatics Association (IPrA) 32:4 ► pp. 489 ff.

Benítez-Castro, Miguel-Ángel & Encarnación Hidalgo-Tenorio

2022. “I Am Proud to Be a Traitor”. Pragmatics and Society 13:3 ► pp. 501 ff.

Forceville, Charles & Natalia Sánchez-Querubín

2022. Relevance theory perspectives on web-mediated communication. In Dynamism in Metaphor and Beyond [Metaphor in Language, Cognition, and Communication, 9], ► pp. 325 ff.

FUOLI, MATTEO, JEANNETTE LITTLEMORE & SARAH TURNER

2022. Sunken ships and screaming banshees: metaphor and evaluation in film reviews. English Language and Linguistics 26:1 ► pp. 75 ff.

Glynn, Dylan

2022. Emergent categories. In Analogy and Contrast in Language [Human Cognitive Processing, 73], ► pp. 245 ff.

Hansson, Sten, Ruth Page & Matteo Fuoli

2022. Discursive Strategies of Blaming: The Language of Judgment and Political Protest Online. Social Media + Society 8:4

Lamb, David A. & Thora Tenbrink

2022. Evaluating Jesus and other “heroes”. Language, Context and Text. The Social Semiotics Forum 4:2 ► pp. 227 ff.

Liu, Yufeng & Dechao Li

2022. The US-China battle over Coronavirus in the news media: Metaphor transfer as a representation of stance mediation. Discourse & Society 33:4 ► pp. 456 ff.

Shrestha, Prithvi N.

2022. Examining evaluative language used in assessment feedback on business students’ academic writing. Assessing Writing 54 ► pp. 100664 ff.

Simaki, Vasiliki, Eleni Seitanidi & Carita Paradis

2022. Evaluating stance annotation of Twitter data. Research in Corpus Linguistics 11:1 ► pp. 53 ff.

Sun, Shuyi & Peter Crosthwaite

2022. “Establish a niche” vianegation: A corpus-based study ofnegationwithin the Move 2 sections of PhD thesis introductions. Open Linguistics 8:1 ► pp. 189 ff.

Sun, Shuyi Amelia & Peter Crosthwaite

2022. “The findings might not be generalizable”: Investigating negation in the limitations sections of PhD theses across disciplines. Journal of English for Academic Purposes 59 ► pp. 101155 ff.

Vilar Lluch, Sara

2022. Redefining attitude for studying explicit and indirect evaluations of human behaviour. Functions of Language 29:2 ► pp. 199 ff.

HIDALGO-TENORIO, ENCARNACIÓN & MIGUEL-ÁNGEL BENÍTEZ-CASTRO

2021. The Language of Evaluation in the Narratives by the Magdalene Laundries Survivors: The Discourse of Female Victimhood. Applied Linguistics 42:2 ► pp. 315 ff.

Mavis Ho, Nga-Ki

2021. Transcreation in marketing: a corpus-based study of persuasion in optional shifts from English to Chinese. Perspectives 29:3 ► pp. 426 ff.

van Driel, Martine

2021. Evaluating news events. In Style and Reader Response [Linguistic Approaches to Literature, 36], ► pp. 143 ff.

Kolhatkar, Varada, Hanhan Wu, Luca Cavasso, Emilie Francis, Kavan Shukla & Maite Taboada

2020. The SFU Opinion and Comments Corpus: A Corpus for the Analysis of Online News Comments. Corpus Pragmatics 4:2 ► pp. 155 ff.

Ross, Andrew S. & David Caldwell

2020. ‘Going negative’: An APPRAISAL analysis of the rhetoric of Donald Trump on Twitter. Language & Communication 70 ► pp. 13 ff.

Van Herck, Rebecca, Sofie Decock & Bernard De Clerck

2020. “Can you send us a PM please?” Service recovery interactions on social media from the perspective of organizational legitimacy. Discourse, Context & Media 38 ► pp. 100445 ff.

Fusari, Sabrina

2019. “Logically, We Quite Agree with the IARC”: Negotiating Interpersonal Meaning in a Corpus of Scientific Texts. In Specialized Discourses and Their Readerships [The M.A.K. Halliday Library Functional Linguistics Series, ], ► pp. 25 ff.

Geng, Yifan & Sue Wharton

2019. How do thesis writers evaluate their own and others’ findings? An appraisal analysis and a pedagogical intervention. English for Specific Purposes 56 ► pp. 3 ff.

Page, Ruth

2019. Self-denigration and the mixed messages of ‘ugly’ selfies in Instagram. Internet Pragmatics 2:2 ► pp. 173 ff.

Paternoster, Annick

2019. Politeness and evaluative adjectives in Italian turn-of-the-century etiquette books (1877–1914). In Politeness in Nineteenth-Century Europe [Pragmatics & Beyond New Series, 299], ► pp. 107 ff.

Bisiada, Mario

2018. A cross-linguistic analysis of the ‘homework’ metaphor in German and English political discourse. Discourse & Society 29:6 ► pp. 609 ff.

This list is based on CrossRef data as of 4 january 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.