Cover not available
Article published In:
Pragmatics and Society
Vol. 13:2 (2022) ► pp.193–223

Clean room, uncomfortable bed

A corpus analysis of evaluation devices in hotel reviews

| HSE University | University of Vigo

Online customer reviews, being an essential factor that determines success or failure in business, in particular in the tourism industry, demand close attention since the investigation of this type of discourse might bear some implications both for language specialists and for hotel managers. This paper is a quantitative and qualitative study of evaluation devices occurring in the corpus of hotel reviews from Booking​.com. The analysis is based on methods associated with corpus linguistics and Systemic Functional Linguistics (Halliday 1994; Eggins 2004), specifically, the Appraisal framework (Martin & White 2005), which is applied here to a new type of discourse, that of hotel ratings. The study focuses on the categories of Attitude, Graduation and Engagement suggested by Martin and White and investigates inscribed and invoked evaluation. In order to adapt the Attitude dimensions to the corpus under study, new subcategories were created as a contribution to evaluation theory. The practical result of the study is a list of evaluation devices employed in hotel reviews; the list may be useful for further corpus analyses and for designing systems for automated analysis of customer evaluation. Since online customer reviews are commonly used by hotels as a benchmark of their guests’ satisfaction and a valuable information source of features and services that need to be improved, the present study is also likely to make an important contribution to the industry.

Keywords: hotel reviews, evaluation theory, Appraisal framework, evaluation devices, corpus analysis
Article outline
  • 1.Introduction
  • 2.Literature review
    • 2.1Evaluation
    • 2.2Corpus studies of online reviews
  • 3.Data and methods
  • 4.Application of the appraisal framework to hotel reviews
    • 4.1Attitude
      • 4.1.1Appreciation
      • 4.1.2Judgement
      • 4.1.3Affect
    • 4.2Graduation and engagement
  • 5.Results
    • 5.1Attitude
    • 5.2Graduation
    • 5.3Engagement
    • 5.4Targets of evaluation
    • 5.5Invoked evaluation
  • 6.Discussion and conclusion
  • Acknowledgements
  • Notes
  • References
Published online: 23 June 2022
DOI logo
https://doi.org/10.1075/ps.19046.smi
References
Amin, Muslim, Zatilaziya Yahya, Wan F. A. Ismayatim, Siti Z. Nasharuddin, and Emma Kassim
2013 “Service quality dimension and customer satisfaction: An empirical study in the Malaysian hotel industry.” Services Marketing Quarterly 34 (2): 115–125. DOI logoGoogle Scholar
Antony, Laurence
2014 AntConc (Version 3.4.4) [Computer Software]. Tokyo: Waseda University. Available from [URL]
Banerjee, Snehasish, & Alton Y. K. Chua
2016 “In search of patterns among travellers’ hotel ratings in TripAdvisor”. Tourism Management 531: 125–131. DOI logoGoogle Scholar
Banerjee, Snehasish, and Alton Y. K. Chua
2017 “Theorizing the textual differences between authentic and fictitious reviews: Validation across positive, negative and moderate polarities.” Internet Research 27 (2): 321–337. DOI logoGoogle Scholar
Ben-Aaron, Diana
2005 “Given and news: Evaluation in newspaper stories about national anniversaries.” Text – Interdisciplinary Journal for the Study of Discourse 25 (5): 691–718. DOI logoGoogle Scholar
Biber, Douglas, and Edward Finegan
1988 “Adverbial stance types in English.” Discourse Processes 11 (1): 1–34. DOI logoGoogle Scholar
1989 “Styles of stance in English: lexical and grammatical marking of evidentiality and affect.” Text 9 (1): 93–124.Google Scholar
Biber, Douglas
2006 “Stance in spoken and written university registers.” Journal of English for Academic Purposes 5 (2): 97–116. DOI logoGoogle Scholar
Booking.com
URL: [URL] (Accessed April 8, 2018). DOI logo
Boucher, Jerry, and Charles E. Osgood
1969 “The Pollyanna hypothesis.” Journal of Verbal Learning and Verbal Behavior 8 (1): 1–8. DOI logoGoogle Scholar
Brookes, Gavin, and Paul Baker
2017 “What does patient feedback reveal about the NHS? A mixed methods study of comments posted to the NHS Choices online service.” BMJ Open 7 (4). DOI logoGoogle Scholar
Buhalis, Dimitrios, and Rob Law
2008 “Progress in information technology and tourism management: 20 years on and 10 years after the Internet – The state of eTourism research.” Tourism Management 29 (4): 609–623. DOI logoGoogle Scholar
Camiciottoli, Belinda C.
2013 Rhetoric in Financial Discourse: A Linguistic Analysis of ICT-mediated Disclosure Genres. Amsterdam & New York: Rodopi. DOI logoGoogle Scholar
Cedefop’s European skills and jobs survey
2014 URL: [URL] (Accessed February 11, 2019).
Cenni, Irene, and Patrick Goethals
2017 “Negative hotel reviews on TripAdvisor: A cross-linguistic analysis.” Discourse, Context & Media 161: 22–30. DOI logoGoogle Scholar
Chipkin, Harvey
2012, July 31. “Consumer trends 2012: review websites.” Travel Weekly. URL: [URL] Accessed: April 4, 2019.
Coffin, Caroline, and Kieran O’Halloran
2006 “The role of appraisal and corpora in detecting covert evaluation.” Functions of Language 13 (1): 77–110. DOI logoGoogle Scholar
Conrad, Susan, and Douglas Biber
2000 “Adverbial marking of stance in speech and writing.” In Evaluation in Text: Authorial Stance and the Construction of Discourse, ed. by Susan Hunston and Geoff Thompson, 56–73. Oxford: Oxford University Press.Google Scholar
Cook, Lory S., David E. Bowen, Richard B. Chase, Sriram Dasu, Doug M. Stewart, and David A. Tansik
2002 “Human resource issues in service design.” Journal of Operations Management 20 (2): 159–174. DOI logoGoogle Scholar
Eggins, Suzanne
2004 An Introduction to Systemic Functional Linguistics. London & New York: Continuum.Google Scholar
Halliday, M. A. K.
1994 An Introduction to Functional Grammar. London: Edward Arnold.Google Scholar
Hao, Jing, and Sally Humphrey
2012 “The role of ‘coupling’ in biological experimental reports.” Linguistics and the Human Sciences 5 (2): 169–194. DOI logoGoogle Scholar
Ho-Dac, Nga N., Stephen J. Carson, and William L. Moore
2013 “The effects of positive and negative online customer reviews: do brand strength and category maturity matter?” Journal of Marketing 77 (6): 37–53. DOI logoGoogle Scholar
Hoffman, Lily M., and Jiri Musil
2009 “Prague, tourism and the post-industrial city.” A Great Cities Institute Working Papers. Chicago: University of Illinois.Google Scholar
Hood, Susan
2012 “Voice and stance as APPRAISAL: Persuading and positioning in research writing across intellectual fields.” In Stance and Voice in Written Academic Genres, ed. by Ken Hyland and Carmen S. Guinda. London: Palgrave Macmillan. DOI logoGoogle Scholar
Hunston, Susan
2011 Corpus Approaches to Evaluation: Phraseology and Evaluative Language. New York & London: Routledge.Google Scholar
Hyland, Ken
1998 “Exploring corporate rhetoric: Metadiscourse in the CEO’s letter.” Journal of Business Communication 35 (2): 224–244. DOI logoGoogle Scholar
2005 Metadiscourse: Exploring Interaction in Writing. London and New York: Continuum.Google Scholar
Jackendoff, Ray
1997 “Twistin’ the night away.” Language 73 (3): 534–559. DOI logoGoogle Scholar
Jeong, Miyoung, and Myunghee M. Jeon
2008 “Customer reviews of hotel experiences through consumer generated media (CGM).” Journal of Hospitality & Leisure Marketing 17 (1–2): 121–138. DOI logoGoogle Scholar
Kachniewska, Magdalena
2002 Zarządzanie Jakością Usług Turystycznych [Quality Management in the Tourist Industry]. Warszawa: Difin.Google Scholar
Kaltenbacher, Martin
2006 “Culture related linguistic differences in tourist websites: the emotive and the factual – a corpus analysis within the framework of APPRAISAL.” In System and Corpus: Exploring Connections, ed. by Geoff Thompson and Susan Hunston, 269–92. London: Equinox.Google Scholar
Keuc, Natasha
2014 “Factors influencing the decision on the measurement of customer satisfaction.” In Economic and Social Development: Book of Proceedings, ed. by Ivica Filipovic, Marina Klacmer Calopa, and Fran Galetic, 552–558. Varazdin: Varazdin Development and Entrepreneurship Agency.Google Scholar
Li, Huiying, Qiang Ye, and Rob Law
2013 “Determinants of Customer Satisfaction in the Hotel Industry: An Application of Online Review Analysis.” Asia Pacific Journal of Tourism Research 18 (7): 784–802. DOI logoGoogle Scholar
Lin, Ingrid Y.
2004 “Evaluating a servicescape: the effect of cognition and emotion.” International Journal of Hospitality Management 23 (2): 163–178. DOI logoGoogle Scholar
Martin, James R.
2000 “Beyond exchange: Appraisal systems in English.” In Evaluation in Text. Authorial Stance and the Construction of Discourse, ed. by Susan Hunston and Geoff Thompson, 142–175. Oxford: Oxford University Press.Google Scholar
Martin, James R., and David Rose
2003 Working with Discourse. Meaning beyond the Clause. London: Continuum.Google Scholar
Martin, James R., and Peter R. R. White
2005 The Language of Evaluation: Appraisal in English. Basingstoke: Palgrave Macmillan. DOI logoGoogle Scholar
Martin-Fuentes, Eva, and Juan P. Mellinas
2018 “Hotels that most rely on Booking. com–online travel agencies (OTAs) and hotel distribution channels.” Tourism Review. DOI logoGoogle Scholar
Mel’cuk, Igor
1996 “Lexical functions: A tool for the description of lexical relations in a lexicon.” In Lexical Junctions in Lexicography and Natural Language Processing, ed. by Leo L. Wanner. 87-102. Amsterdam & Philadelphia: John Benjamins.Google Scholar
Mellinas, Juan P.
2019 “Dependency of Spanish urban hotels on Booking. com.” Tourism Analysis 24 (1): 3–12. DOI logoGoogle Scholar
Mohsin, Asad, and Tim Lockyer
2010 “Customer perceptions of service quality in luxury hotels in New Delhi, India: an exploratory study.” International Journal of Contemporary Hospitality Management 22 (2): 160–173. DOI logoGoogle Scholar
Oteíza, Teresa
2017 “The appraisal framework and discourse analysis”. In The Routledge Handbook of Systemic Functional Linguistics, ed. by Tom Bartlett and Gerard O’Grady, 481–496. New York: Routledge.Google Scholar
Oxford English Dictionary (OED)
URL: [URL] Accessed July 18, 2021.
Ozimek, Irena, Julita Szlachciuk, Olena Kulykovets and Natalia Przeździecka-Czyżewska
2017 “Factors Influencing the Choice and Quality Assessment of Hotel Facilities in Poland”. Folia Oeconomica Stetinensia 17 (1): 170–181. DOI logoGoogle Scholar
Pang, Bo, and Lillian Lee
2008 “Opinion Mining and Sentiment Analysis.” Foundations and Trends in Information Retrieval 2 (1–2): 1–135. DOI logoGoogle Scholar
Pawley, Andrew, and Frances H. Syder
1983 “Two puzzles for linguistic theory: nativelike selection and nativelike fluency.” In Language and Communication 7(1), ed. by Jack C. Richards and Richard W. Schmidt, 191–226. London: Longman.Google Scholar
Pierini, Patrizia
2009 “Adjectives in tourism English on the web: a corpus-based study.” CÍRCULO de Lingüística Aplicada a la Comunicación 401: 93–116.Google Scholar
Reuland, Ruud, Janet Choudry, and Ans Fagel
1985 “Research in the field of hospitality.” International Journal of Hospitality Management 4 (4): 141–146. DOI logoGoogle Scholar
Ryu, Kisang, and Soocheong S. Jang
2007 “The effect of environmental perceptions on behavioral intentions through emotions: The case of upscale restaurants.” Journal of Hospitality & Tourism Research 31 (1): 56–72. DOI logoGoogle Scholar
Sammons, Gail, Pat Moreo, Lori F. Benson, and Fred Demicco
1999 “Analysis of female business travelers’ selection of lodging accommodations.” Journal of Travel & Tourism Marketing 8 (1): 65–83. DOI logoGoogle Scholar
Silver, Marc
2003 “The stance of stance: a critical look at ways stance is expressed and modeled in academic discourse.” Journal of English for Academic Purposes 2 (4): 359–374. DOI logoGoogle Scholar
Sinclair, John
1991 Corpus, Concordance and Collocation. Oxford: Oxford University Press.Google Scholar
Smith, Trevor, and Andrew J. Spencer
2011 “Predictors of value for money in Jamaican all-inclusive hotels.” International Journal of Humanities and Social Science 1 (4): 93–102.Google Scholar
Sparks, Beverley
2001 “Managing service failure through recovery.” In Service quality management in hospitality, tourism and leisure, ed. by Jay Kandampully, Connie Mok, and Beverley Sparks, 193–219. New York & London: Routledge.Google Scholar
Su, Hang
2016 “How products are evaluated? Evaluation in customer review texts.” Language Resources and Evaluation 50 (3): 475–495. DOI logoGoogle Scholar
Thompson, Geoff
1996 Introducing Functional Grammar. London: Edward Arnold.Google Scholar
2014 “AFFECT and emotion, target-value mismatches, and Russian dolls: Refining the APPRAISAL model”. In Evaluation in context, ed. by Geoff Thompson and Laura Alba-Juez, 47–66. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Thompson, Geoff, and Laura Alba-Juez
2014 Evaluation in Context. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Tian, Youfei
2013 “Engagement in online hotel reviews: A comparative study.” Discourse, Context & Media 2 (4): 184–191. DOI logoGoogle Scholar
Torres, Edwin N., and Sheryl Kline
2013 “From customer satisfaction to customer delight.” International Journal of Contemporary Hospitality Management 25 (5): 642–659. DOI logoGoogle Scholar
Tuan, Nham P., and Nguyen T. H. Linh
2014 “Impact of service quality performance on customer satisfaction: A case study of Vietnam’s five star hotel.” ABAC Journal 34 (3): 53–70.Google Scholar
Vásquez, Camilla
2011 “Complaints online: The case of TripAdvisor.” Journal of Pragmatics 43 (6): 1707–1717. DOI logoGoogle Scholar
Wang, Hongway, Pei Yin, Lijuan Zheng, and James N. K. Liu
2014 “Sentiment classification of online reviews: using sentence-based language model.” Journal of Experimental & Theoretical Artificial Intelligence 26 (1): 13–31. DOI logoGoogle Scholar
White, Peter R. R.
2002 “Appraisal – the language of evaluation and stance.” In The Handbook of Pragmatics, ed. by Jef Verschueren, Jan-Ola Östman, Jan Blommaert, and Chris Bulcaen, 1–27. Amsterdam: John Benjamins.Google Scholar
2003 “Beyond modality and hedging: a dialogic view of the language of intersubjective stance.” Appraisal: Special Edition of Text 23 (2): 259–84. DOI logoGoogle Scholar
2011 “Appraisal.” In Discursive Pragmatics, ed. by Jan Zienkowski, Jan-Ola Östman, and Jef Verschueren, 14–36. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Yang, Yang, Zhenxing Mao, and Jingyin Tang
2018 “Understanding guest satisfaction with urban hotel location.” Journal of Travel Research 57 (2): 243–259. DOI logoGoogle Scholar