Article published in:
Functions of Language
Vol. 25:2 (2018) ► pp. 229–258

A stepwise method for annotating appraisal

| University of Birmingham
Despite a growing awareness of methodological issues, the literature on appraisal has not so far provided adequate answers to some of the key challenges involved in reliably identifying and classifying evaluative language expressions. This article presents a stepwise method for the manual annotation of appraisal in text that is designed to optimize reliability, replicability and transparency. The procedure consists of seven steps, from the creation of a context-specific annotation manual to the statistical analysis of the quantitative data derived from the manually-performed annotations. By presenting this method, the article pursues the twofold purpose of (i) providing a practical tool that can facilitate more reliable, replicable and transparent analyses, and (ii) fostering a discussion of the best practices that should be observed when manually annotating appraisal.
Article outline
  • 1.Introduction
  • 2.Challenges in annotating appraisal
    • 2.1Challenges in identifying appraisal
    • 2.2Challenges in classifying appraisal
    • 2.3Reliability, replicability, transparency
  • 3.The stepwise annotation method
    • 3.1General principles
      • Principle 1.All choices should be accounted for
      • Principle 2.The annotation guidelines should be tested and refined until maximum reliability is achieved
      • Principle 3.Reliability should always be assessed, and reliability scores reported and discussed
    • 3.2An outline of the steps
      • 3.2.1Step 1: Define the scope of the project
      • 3.2.2Step 2: Select and configure an annotation tool
      • 3.2.3Step 3: Draft a context-specific annotation manual
      • 3.2.4Step 4: Assess reliability
      • 3.2.5Step 5: Refine the annotation manual
      • 3.2.6Step 6: Annotate the corpus
      • 3.2.7Step 7: Analyze the results
  • 4.Conclusion
  • Acknowledgements
  • Notes
  • References
Published online: 19 October 2018
https://doi.org/10.1075/fol.15016.fuo
References

References

Artstein, Ron & Massimo Poesio
2008 Inter-coder agreement for computational linguistics. Computational Linguistics 34(4). 555–596. CrossrefGoogle Scholar
Baayen, R. Harald
2008 Analyzing linguistic data: A practical introduction to Statistics using R. Cambridge: CUP. CrossrefGoogle Scholar
Bartalesi Lenzi, Valentina, Giovanni Moretti & Rachele Sprugnoli
2012 CAT: the CELCT Annotation Tool. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk & Stelios Piperidis (eds.), Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC12), Istanbul: European Language Resources Association (ELRA). Available online at: http://​www​.lrec​-conf​.org​/proceedings​/lrec2012​/pdf​/216​_Paper​.pdf [Accessed 22 September 2015]
Bednarek, Monika
2006 Evaluation in media discourse: Analysis of a newspaper corpus. London: Continuum.Google Scholar
2008 Emotion talk across corpora. Basingstoke: Palgrave. CrossrefGoogle Scholar
2009 Language patterns and ATTITUDE. Functions of Language 16(2). 165–192. CrossrefGoogle Scholar
2014 An astonishing season of destiny! Evaluation in blurbs used for advertising TV series. In Geoff Thompson & Laura Alba-Juez (eds.), 197–220.Google Scholar
Ben-Aaron, Diana
2005 Given and news: Evaluation in newspaper stories about national anniversaries. Text & Talk 25(5). 691–718. CrossrefGoogle Scholar
Carretero, Marta & Maite Taboada
2014 Graduation within the scope of Attitude in English and Spanish consumer reviews of books and movies. In Geoff Thompson & Laura Alba-Juez (eds.), 221–239.Google Scholar
Cohen, Jacob
1960 A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1). 37–46. CrossrefGoogle Scholar
Cozby, Paul & Scott Bates
2011 Methods in behavioral research, 11th edn. New York, NY: McGraw-Hill.Google Scholar
Dice, Lee R.
1945 Measures of the amount of ecologic association between species. Ecology 26(3). 297–302. CrossrefGoogle Scholar
Divjak, Dagmar
2006 Ways of intending: Delineating and structuring near synonyms. In Stefan Th. Gries & Anatol Stefanowitsch (eds.), Corpora in cognitive linguistics: Corpus-based approaches to syntax and lexis, 19–56. Berlin: Mouton.Google Scholar
Don, Alexanne
2007 An approach to the analysis of textual identity through profiles of evaluative disposition. In Proceedings of the Australian Systemic Functional Linguistics Association 2007 Conference. Available at: http://​www​.asfla​.org​.au​/category​/asfla2007/. [Accessed 22 September 2015]
Fuoli, Matteo
2012 Assessing social responsibility: A quantitative analysis of Appraisal in BP’s and IKEA’s social reports. Discourse & Communication 6(1). 55–81. CrossrefGoogle Scholar
Fuoli, Matteo & Dylan Glynn
2013 Computer-assisted manual annotation of evaluative language expressions: Bridging discourse and corpus approaches. Paper presented at the Evaluative Language and Corpus Linguistics Workshop , Lancaster University, 22 July 2013.
Fuoli, Matteo & Charlotte Hommerberg
2015 Optimizing transparency, reliability and replicability: Annotation principles and inter-coder agreement in the quantification of evaluative expressions. Corpora 10(3). 315–349. CrossrefGoogle Scholar
Fuoli, Matteo & Carita Paradis
2014 A model of trust-repair discourse. Journal of Pragmatics 741. 52–69. CrossrefGoogle Scholar
Geeraerts, Dirk, Stefan Grondelaers & Peter Bakema
1994 The structure of lexical variation. Berlin: Mouton. CrossrefGoogle Scholar
Glynn, Dylan
2009 Polysemy, syntax, and variation: A usage-based method for Cognitive Semantics. In Vyvyan Evans & Stéphanie Pourcel (eds.), New Directions in Cognitive Linguistics, 77–104. Amsterdam: Benjamins. CrossrefGoogle Scholar
Gries, Stefan Th.
1999 Particle movement: A cognitive and functional approach. Cognitive Linguistics 10(2). 105–146. CrossrefGoogle Scholar
2013 Statistics for linguistics with R: A practical introduction. Berlin: Mouton. CrossrefGoogle Scholar
Hommerberg, Charlotte & Alexanne Don
2015 Appraisal and the language of wine appreciation: A critical discussion of the potential of the Appraisal framework as a tool to analyse specialised genres. Functions of Language 22(2). 161–191. CrossrefGoogle Scholar
Hood, Susan
2004 Appraising research: Taking a stance in academic writing. Sydney: University of Technology PhD thesis.Google Scholar
Hunston, Susan
2004 Counting the uncountable: Problems of identifying evaluation in a text and in a corpus. In Alan Partington, John Morley & Louann Haarman (eds.), Corpora and Discourse, 157–188. Bern: Peter Lang.Google Scholar
2011 Corpus approaches to evaluation: Phraseology and evaluative language. New York, NY: Taylor & Francis.Google Scholar
Kirk, Jerome & Marc L. Miller
1986 Reliability and validity in qualitative research. Beverly Hills, CA: Sage. CrossrefGoogle Scholar
Krippendorf, Klaus
2004 Content Analysis: An introduction to its methodology. Thousand Oaks, CA: Sage.Google Scholar
Lipovsky, Caroline
2008 Constructing affiliation and solidarity in job interviews. Discourse & Communication 2(4). 411–432. CrossrefGoogle Scholar
2011  It’s really a great presentation!: Appraising candidates in job interviews. Linguistics and the Human Sciences 4(2). 161–185. CrossrefGoogle Scholar
2013 Negotiating ones expertise through appraisal in CVs. Linguistics and the Human Sciences 8(3). 307–333. CrossrefGoogle Scholar
Mackay, James & Jean Parkinson
2009  My very own mission impossible: An APPRAISAL analysis of student teacher reflections on a design and technology project. Text & Talk 29(6). 729–753. CrossrefGoogle Scholar
Macken-Horarik, Mary & Anne Isaac
2014 Appraising Appraisal. In Geoff Thompson & Laura Alba-Juez (eds.), 67–92.Google Scholar
Marshall, Christine, Ralph Adendorff & Vivian de Klerk
2010 The role of APPRAISAL in the NRF Rating System: An analysis of Judgement and Appreciation in peer reviewers’ reports. Southern African Linguistics and Applied Language Studies 27(4). 391–412. CrossrefGoogle Scholar
Martin, J. R.
2000 Beyond exchange: Appraisal systems in English. In Susan Hunston & Geoff Thompson (eds.), Evaluation in text: Authorial stance and the construction of discourse, 142–175. Oxford: OUP.Google Scholar
Martin, J. R. & David Rose
2003 Working with discourse: Meaning beyond the clause. London: Continuum.Google Scholar
Martin, J. R. & Peter R. R. White
2005 The language of evaluation: Appraisal in English. London: Palgrave Macmillan. CrossrefGoogle Scholar
Mauranen, Anna & Marina Bondi
2003 Evaluative language use in academic discourse. Journal of English for Academic Purposes 2(4). 269–271. CrossrefGoogle Scholar
O’Donnell, Mick
2008 Demonstration of the UAM CorpusTool for text and image annotation. Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Demo Session. 13–16. CrossrefGoogle Scholar
2012 Appraisal analysis and the computer. Revista Canaria de Estudios Ingleses 651. 115–130.Google Scholar
2014 Exploring identity through Appraisal Analysis: A corpus annotation methodology. Linguistics and the Human Sciences 9(1). 95–116.Google Scholar
Page, Ruth E.
2003 An analysis of APPRAISAL in childbirth narratives with special consideration of gender and storytelling style. Text & Talk 23(2). 211–237.Google Scholar
Paradis, Carita
2001 Adjectives and boundedness. Cognitive Linguistics 12(1). 47–64. CrossrefGoogle Scholar
Paradis, Carita, Joost van de Weijer, Caroline Willners & Magnus Lindgren
2012 Evaluative polarity of antonyms. Lingue e linguaggio 11(2). 199–214.Google Scholar
Pounds, Gabrina
2010 Attitude and subjectivity in Italian and British hardnews reporting: The construction of a culture-specific ‘reporter’ voice. Discourse Studies 12(1). 106–137. CrossrefGoogle Scholar
2011  This property offers much character and charm: Evaluation in the discourse of online property advertising. Text & Talk 31(2). 195–220. CrossrefGoogle Scholar
Pustejovsky, James & Amber Stubbs
2012 Natural language annotation for machine learning. Sebastopol, CA: O’Reilly Media.Google Scholar
Read, Jonathon & John Carroll
2012 Annotating expressions of Appraisal in English. Language Resources and Evaluation 46(3). 421–447. CrossrefGoogle Scholar
Ryshina-Pankova, Marianna
2014 Exploring argumentation in course-related blogs through ENGAGEMENT. In Geoff Thompson & Laura Alba-Juez (eds.), 281–302.Google Scholar
Santamaría-García, Carmen
2014 Evaluative discourse and politeness in university students’ communication through social networking sites. In Geoff Thompson & Laura Alba-Juez (eds.), 387–411.Google Scholar
Taboada, Maite & Marta Carretero
2012 Contrastive analyses of evaluation in text: Key issues in the design of an annotation system for ATTITUDE applicable to consumer reviews in English and Spanish. Linguistics and the Human Sciences 6(1–3). 275–295.Google Scholar
Taboada, Maite, Marta Carretero & Jennifer Hinnell
2014 Loving and hating the movies in English, German and Spanish. Languages in Contrast 14(1). 127–161. CrossrefGoogle Scholar
Thompson, Geoff
2014 AFFECT and emotion, target-value mismatches, and Russian dolls: Refining the APPRAISAL model. In Geoff Thompson & Laura Alba-Juez (eds.), 47–66.Google Scholar
Thompson, Geoff & Laura Alba-Juez
(eds.) 2014 Evaluation in context. Amsterdam: Benjamins. CrossrefGoogle Scholar
Thompson, Geoff & Susan Hunston
2000 Evaluation: An introduction. In Susan Hunston & Geoff Thompson (eds.), Evaluation in text: Authorial stance and the construction of discourse, 1–27. Oxford: OUP.Google Scholar
Cited by

Cited by 16 other publications

Andersson, Marta
2022. ‘So many “virologists” in this thread!’. Pragmatics. Quarterly Publication of the International Pragmatics Association (IPrA) Crossref logo
Bisiada, Mario
2018. A cross-linguistic analysis of the ‘homework’ metaphor in German and English political discourse. Discourse & Society 29:6 ► pp. 609 ff. Crossref logo
FUOLI, MATTEO, JEANNETTE LITTLEMORE & SARAH TURNER
2022. Sunken ships and screaming banshees: metaphor and evaluation in film reviews. English Language and Linguistics 26:1 ► pp. 75 ff. Crossref logo
Fusari, Sabrina
2019.  In Specialized Discourses and Their Readerships [The M.A.K. Halliday Library Functional Linguistics Series, ], ► pp. 25 ff. Crossref logo
Geng, Yifan & Sue Wharton
2019. How do thesis writers evaluate their own and others’ findings? An appraisal analysis and a pedagogical intervention. English for Specific Purposes 56 ► pp. 3 ff. Crossref logo
HIDALGO-TENORIO, ENCARNACIÓN & MIGUEL-ÁNGEL BENÍTEZ-CASTRO
2021. The Language of Evaluation in the Narratives by the Magdalene Laundries Survivors: The Discourse of Female Victimhood. Applied Linguistics 42:2 ► pp. 315 ff. Crossref logo
Kolhatkar, Varada, Hanhan Wu, Luca Cavasso, Emilie Francis, Kavan Shukla & Maite Taboada
2020. The SFU Opinion and Comments Corpus: A Corpus for the Analysis of Online News Comments. Corpus Pragmatics 4:2 ► pp. 155 ff. Crossref logo
Makki, Mohammad & Andrew S. Ross
2021. “We were cocked & loaded to retaliate”. Journal of Language Aggression and Conflict Crossref logo
Mavis Ho, Nga-Ki
2021. Transcreation in marketing: a corpus-based study of persuasion in optional shifts from English to Chinese. Perspectives 29:3 ► pp. 426 ff. Crossref logo
Page, Ruth
2019. Self-denigration and the mixed messages of ‘ugly’ selfies in Instagram. Internet Pragmatics 2:2 ► pp. 173 ff. Crossref logo
Paternoster, Annick
2019.  In Politeness in Nineteenth-Century Europe [Pragmatics & Beyond New Series, 299], ► pp. 107 ff. Crossref logo
Ross, Andrew S. & David Caldwell
2020. ‘Going negative’: An APPRAISAL analysis of the rhetoric of Donald Trump on Twitter. Language & Communication 70 ► pp. 13 ff. Crossref logo
Trnavac, Radoslava & Maite Taboada
2021. Engagement and constructiveness in online news comments in English and Russian. Text & Talk 0:0 Crossref logo
van Driel, Martine
2021.  In Style and Reader Response [Linguistic Approaches to Literature, 36], ► pp. 143 ff. Crossref logo
Van Herck, Rebecca, Sofie Decock & Bernard De Clerck
2020. “Can you send us a PM please?” Service recovery interactions on social media from the perspective of organizational legitimacy. Discourse, Context & Media 38 ► pp. 100445 ff. Crossref logo
Vilar Lluch, Sara
2022. Redefining attitude for studying explicit and indirect evaluations of human behaviour. Functions of Language Crossref logo

This list is based on CrossRef data as of 09 april 2022. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.