Cover not available
Article published In:
Language, Context and Text
Vol. 4:2 (2022) ► pp.335–374

Emotion in speech

A systemic functional semiotic approach to the vocalisation of affect

ORCID logo | University of Wollongong | Universidad Nacional de La Pampa
ORCID logo | University of Wollongong
ORCID logo | University of Wollongong

This paper explores how emotion is conveyed in spoken language based on a sample of three stories for children read aloud by a trained storyteller. It draws on both Martin & White’s appraisal framework (2005) and a systematic account of vocal features. Interpreting and profiling emotion is a challenge both for researchers interested in spoken language and for English-as-an-additional-language teachers, when working to improve students’ spoken expression of emotion. Underpinning this challenge is the absence of a unified theoretical stance and a consequent lack of shared analytical tools for describing how speakers use semiotic resources to convey emotion in their speech. A systemic functional semiotic approach can make visible the co-patterning of wordings and vocalisations that express emotion.

Keywords: systemic functional semiotics (SFS), appraisal analysis, spoken language, vocal features, co-patterning of wordings and vocalisations, emotion in speech, interpreting and profiling emotion, multi-stratal analysis, multisemiotic interaction, storytelling, children’s stories, English as additional language (EAL)
Article outline
  • 1.Introduction
  • 2.The study
    • 2.1Methodology
      • 2.1.1Data collection and data preparation
    • 2.2Theoretical framework for the analysis
      • 2.2.1Emotion in speech: SFS perspective
      • 2.2.2Analysis of evaluative language
      • 2.2.3Analysis of vocal features
        • 2.2.3.1Suprasegmental features
        • 2.2.3.2Paralinguistic features: Definition and analytical technique
  • 3.Results and discussion
    • 3.1Preliminary vocal profiles of inscribed affect
      • 3.1.1Vocal profiles for inscribed affect and vocal emotion profiles outside the SFS tradition
      • 3.1.2Prediction value of profiles and teaching implications
      • 3.1.3Association between affect categories and paralinguistic features
      • 3.1.4Additional understandings gained from associating affect categories and vocal profiles
      • 3.1.5Distinguishing affect types: The example of ‘unhappy’
  • 4.Conclusions and pedagogical implications
  • Acknowledgements
  • Notes
  • References
Published online: 29 September 2022
DOI logo
https://doi.org/10.1075/langct.21012.ari
References
Ariztimuño, Lilian I.
2017 The expression of emotion in fairy tales: A multimodal approach to improve EFL students’ oral renderings. Phonetics Teaching and Learning Conference (PTLC2017) Proceedings, 11–15.Google Scholar
Arnfield, Simon, Peter Roach, Jane Setter, Peter Greasley & Dave Horton
1995 Emotional stress and speech tempo variation. ESCA-NATO Tutorial and research workshop on speech under stress, 13–15.Google Scholar
Bednarek, Monika
2008 Emotion talk in corpora. Basingstoke/New York: Palgrave Macmillan. DOI logoGoogle Scholar
Boersma, Paul & David Weenink
2021 Praat: Doing phonetics by computer [Computer program]. Version 6.0. 381. https://​www​.fon​.hum​.uva​.nl​/praat​/download​_win​.html
Bombelli, Griselda, Lidia R. Soler & María S. Waasaf
2013 The language of Evaluation: Paralinguistic features as a phonological domain for Appraisal. DELTA Documentação de Estudos em Lingüística Teórica e Aplicada 29 (2). 267–280. DOI logoGoogle Scholar
Caldwell, David
2014 The interpersonal voice: Applying appraisal to the rap and sung voice. Social Semiotics 24 (1). 40–55. DOI logoGoogle Scholar
Cruttenden, Alan
2008 Gimson’s pronunciation of English (7th edition). London: Hodder Education.Google Scholar
Crystal, David & Randolph Quirk
1964 Systems of prosodic and paralinguistic features in English. The Hague: De Gruyter Mouton. DOI logoGoogle Scholar
Douglas-Cowie, Ellen, Nick Campbell, Roddy Cowie & Peter Roach
2003 Emotional speech: Towards a new generation of databases. Speech Communication 40 (1). 33–60. DOI logoGoogle Scholar
Eggins, Suzanne & Diana Slade
2005 Analysing casual conversation. London: Equinox.Google Scholar
Ekman, Paul
1999 Basic emotions. In Tim Dalgleish & Mick Power (eds.), Handbook of cognition and emotion. Chichester: John Wiley & Sons. DOI logoGoogle Scholar
Germani, Miriam P. & Lucía I. Rivas
2017 A genre approach to prosody: Teaching intonation from a discourse perspective. Phonetics Teaching and Learning Conference (PTLC2017) Proceedings, 49–53.Google Scholar
Halliday, Michael A. K.
1970 A course in spoken English: Intonation. London: OUP.Google Scholar
1978 Language as social semiotic: The social interpretation of language and meaning. London: Hodder Education.Google Scholar
2003 On language and linguistics, volume 3 in the collected works of M.A.K Halliday. Edited by Jonathan J. Webster. London: Continuum.Google Scholar
Halliday, Michael A. K. & William S. Greaves
2008 Intonation in the grammar of English. Equinox.Google Scholar
Halliday, Michael A. K. & Ruqaiya Hasan
1985 Language, context, and text: Aspects of language in a social-semiotic perspective. Waurn Ponds, Victoria: Deakin University Press.Google Scholar
Halliday, Michael A. K. & Christian M. I. M. Matthiessen
2014 Halliday’s introduction to functional grammar. London: Routledge. DOI logoGoogle Scholar
Hasan, Ruqaiya
1984 The nursery tale as a genre. Nottingham Linguistic Circular, 71–102.Google Scholar
1985 Linguistics, language, and verbal art. Oxford: Oxford University Press.Google Scholar
Humphrey, Sally & Shoshana Dreyfus
2012 Exploring the interpretive genre in applied linguistics. Indonesian Journal of SFL 1 (2). 156–174.Google Scholar
Jewitt, Carey, Jeff Bezemer & Kay O’Halloran
2016 Introducing multimodality. London/New York: Routledge. DOI logoGoogle Scholar
Johnstone, Tom, Van Reekum, Carien M. & Klaus R. Scherer
2001 Vocal expression correlates of appraisal processes. In Klaus R. Scherer, Angela Schorr & Tom Johnstone (eds.), Appraisal processes in emotion: Theory, methods, research, 271–284. New York: Oxford University Press.Google Scholar
Juslin, Patrik N. & Klaus R. Scherer
2005 Vocal expression of affect. In Jinni A. Harrigan, Robert Rosenthal & Klaus R. Scherer (eds.), The new handbook of methods in nonverbal behavior research. Oxford: Oxford University Press. DOI logoGoogle Scholar
Kress, Gunther R.
2010 Multimodality: A social semiotic approach to contemporary communication. New York: Routledge.Google Scholar
Laukka, Petri, Hillary Anger Elfenbein, Nutankumar S. Thingujam, Thomas Rockstuhl, Frederick. K. Iraki, Wanda Chui & Jean Althoff
2016 The expression and recognition of emotions in the voice across five nations: A lens model analysis based on acoustic features. Journal of Personality and Social Psychology 111 (5). 686–705. DOI logoGoogle Scholar
Lorette, Parnelle & Jean-Marc Dewaele
2019 The relationship between bi/multilingualism, nativeness, proficiency and multimodal emotion recognition ability. International Journal of Bilingualism 23 (6). 1502–1516. DOI logoGoogle Scholar
Macken-Horarik, Mary
2003 Appraisal and the special instructiveness of narrative. Text & Talk 23 (2). 285–312. DOI logoGoogle Scholar
Macken-Horarik, Mary & Anne Isaac
2014 Appraising appraisal. In Geoff Thompson & Laura Alba-Juez (eds.), Evaluation in context. Amsterdam/Philadelphia: John Benjamins. DOI logoGoogle Scholar
Mackenzie, J. Lachlan & Laura Alba-Juez
2019 Emotion in discourse. Amsterdam/Philadelphia: John Benjamins. DOI logoGoogle Scholar
Martin, James. R.
2017 The discourse semantics of attitudinal relations: Continuing the study of lexis. Russian Journal of Linguistics 21 (1). 22–47. DOI logoGoogle Scholar
Martin, James R. & David Rose
2008 Genre relations: Mapping culture. London/Oakville: Equinox.Google Scholar
Martin, James R. & Peter R. R. White
2005 The language of evaluation. Appraisal in English. Houndmills, UK: Palgrave Macmillan. DOI logoGoogle Scholar
Martin, James R. & Michele Zappavigna
2019 Embodied meaning: A systemic functional perspective on paralanguage. Functional Linguistics 6 (1). 1–33. DOI logoGoogle Scholar
Matthiessen, Christian M. I. M.
2009 Multisemiosis and context-based register typology. In Eija Ventola & Arsenio J. M. Guijarro (eds.), The world told and the world shown: Multisemiotic issues. 11–38. London: Palgrave Macmillan.Google Scholar
Mills, Kathy A. & Len Unsworth
2016 Multimodal Literacy: Oxford Research Encyclopedia of Education. DOI logoGoogle Scholar
2017 iPad animations: Powerful multimodal practices for adolescent literacy and emotional language. Journal of Adolescent & Adult Literacy 61 (6). 609–620. DOI logoGoogle Scholar
Noad, Betty & Georgina Barton
2018 Emotion resonance and divergence: A semiotic analysis of music and sound in “The Lost Thing” an animated short film and “Elizabeth” a film trailer. Social Semiotics 30 (2). 206–224. DOI logoGoogle Scholar
O’Grady, Gerard
2010 A grammar of spoken English: The intonation of increments. London: Continuum.Google Scholar
2020 Is there a role for prosody within register studies? And if so what and how? Language, Context and Text 2 (1). 59–90. DOI logoGoogle Scholar
Painter, Clare, James. R. Martin & Len Unsworth
2013 Reading visual narratives: Image analysis of children’s picture books. Sheffield: Equinox.Google Scholar
Pavlenko, Aneta
2007 Emotions and multilingualism. Cambridge: Cambridge University Press.Google Scholar
Rintell, Ellen. M.
1984 But how did you feel about that? The learner’s perception of emotion in speech. Applied Linguistics 5 (3). 255–264. DOI logoGoogle Scholar
Roach, Peter
2000 Techniques for the phonetic description of emotional speech. In ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion. 53–59.Google Scholar
2009 English phonetics and phonology: A practical course. Cambridge: Cambridge University Press.Google Scholar
Roach, Peter, Richard Stibbard, Jane Osborne, Simon Arnfield & Jane Setter
1998 Transcription of prosodic and paralinguistic features of emotional speech. Journal of the International Phonetic Association 28 (1–2). 83–94. DOI logoGoogle Scholar
Royce, Terry D.
2007 Multimodal communicative competence in second language contexts. In Terry D. Royce & Wendy L. Bowcher (eds.), New directions in the analysis of multimodal discourse. 361–390. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
Scherer, Klaus R.
2003 Vocal communication of emotion: A review of research paradigms. Speech Communication 40 (1–2). 227–256. DOI logoGoogle Scholar
2019 Acoustic patterning of emotion vocalizations. In Sascha Fruhholz & Pascal Belin (eds.), The Oxford handbook of voice perception. 61–91. Oxford: OUP.Google Scholar
Scherer, Klaus R. & Heiner Ellgring
2007 Multimodal expression of emotion: Affect programs or componential appraisal patterns? Emotion 7 (1). 158–171. DOI logoGoogle Scholar
Schuller, Björn, Anton Batliner, Stefan Steidl & Dino Seppi
2011 Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge. Speech Communication 53 (9–10). 1062–1087. DOI logoGoogle Scholar
Smith, Bradley A.
2012 Speech and writing: Intonation within multimodal studies. In Kay. L. O’Halloran & Bradley A. Smith (eds.), Multimodal studies: Exploring issues and domains. 291–313. London: Routledge.Google Scholar
Smith, Bradley A. & William S. Greaves
2015 Intonation. In Jonathan J. Webster (ed.), The Bloomsbury companion to MAK Halliday. 291–313. London: Bloomsbury.Google Scholar
Storynory
2013 King Thrushbeard, The elves and the shoemaker, Rumpelstiltskin. Available at https://​www​.storynory​.com​/category​/fairy​-tales​/brothers​-grimm/. (17 July, 2013.)
Styler, Will
2015 Using Praat for linguistic research. University of Colorado at Boulder Phonetics Laboratory. http://​savethevowels​.org​/praat
Tench, Paul
1996 The intonation systems of English. London: Cassell.Google Scholar
2014 A systemic functional model of the intonation of clauses in English. Readings in English Phonetics and Phonology (1). 261–296.Google Scholar
Unsworth, Len & Kathy A. Mills
2020 English language teaching of attitude and emotion in digital multimodal composition. Journal of Second Language Writing (47). 100712. DOI logoGoogle Scholar
van Leeuwen, Theo
1999 Speech, music, sound. Basingstoke: Macmillan. DOI logoGoogle Scholar
Zappavigna, Michele, Chris Cléirigh, Paul Dwyer & James R. Martin
2010 Visualising appraisal prosody. In Ahmar Mahboob & Naomi K. Knight (eds.), Appliable Linguistics: Texts, Contexts, and Meanings, 150–167. London & New York: Continuum.Google Scholar