Audiovisual translation as orchestration of multimodal synergies: Expendability of interjections in intralingual film subtitling

Yi Jing
Harbin Institute of Technology , Shenzhen

Abstract

This study explores the coordination between subtitles and other semiotic resources in films, from within a systemic functional semiotic framework. The paper focuses on the subtitling of interjections (e.g., ‘oh’, ‘wow’, ‘yay’) to examine how subtitles are synthesized in films with respect to the presence of facial expressions and/or bodily gestures which might be perceived as conveying similar meanings. The analysis of multiple versions of intralingual subtitles of two English-language films shows that the seemingly random omission of soundtrack interjections from subtitles is patterned to a considerable degree – those interjections which are concurrent with semiotically correlated actions were frequently omitted. The findings suggest that subtitling is a complex process involving synergy between subtitles and other semiotic resources, which calls for interdisciplinary research integrating translation, multimodality, and linguistics.

Keywords
Publication history
Table of contents

The study of audiovisual translation (AVT), which deals with multimodal texts, has naturally been closely aligned with multimodal approaches (e.g., Chaume 1997; Taylor 2003, 2004, 2013, 2016; Gambier 2006; Perego 2009; Burczynska 2012). This article contributes to the growing body of AVT research, and explores how subtitles are coordinated with other semiotic resources in films. This study focuses on interjections (e.g., ‘oh’, ‘wow’, ‘yay’, ‘aha’, ‘ouch’, ‘oops’, ‘hurray’), and examines the English intralingual subtitling of soundtrack interjections with respect to the presence of concurrent facial expressions and bodily gestures. For instance, it might be possible that where the visual shows a facial expression of raised eyebrows and widened eyes and mouth, the concurrent spoken interjection ‘wow’ might be considered expendable for subtitling. The assumption is that the meaning conveyed via the facial expression is analogous, or at least related, to the meaning conveyed via the utterance (in this case, ‘wow’), which, according to Gottlieb (1994, 273) may be regarded as a case of “intersemiotic redundancy.” This study interprets this hypothesis of intersemiotic redundancy from a systemic functional semiotic perspective, and suggests that intralingual film subtitling can be conceptualized as a process by which subtitlers orchestrate multimodal complementarities between their subtitles and other semiotic resources, such as facial expressions and bodily gestures.

Full-text access is restricted to subscribers. Log in to obtain additional credentials. For subscription information see Subscription & Price. Direct PDF access to this article can be purchased through our e-platform.

Filmography

DeMicco, Kirk, and Chris Sanders
2013The Croods. DreamWorks Animation. USA: 20th Century Fox, DVD.Google Scholar
Anderson, Wes
2014The Grand Budapest Hotel. Fox Searchlight Pictures. USA: 20th Century Fox, DVD. Google Scholar

References

Ameka, Felix K.
1992 “Interjections: The Universal Yet Neglected Part of Speech.” Journal of Pragmatics 18 (2–3): 101–118. Crossref logoGoogle Scholar
2006 “Interjections.” In Encyclopedia of Language & Linguistics, edited by Keith Brown, 743–746. Oxford: Elsevier. Crossref logoGoogle Scholar
Bogucki, Łukasz
2004 “The Constraint of Relevance in Subtitling.” The Journal of Specialised Translation (1): 71–88.Google Scholar
Burczynska, Paulina
2012Multimodality and Audiovisual Translation: Subtitling Humor. Saarbrücken: Lambert Academic Publishing.Google Scholar
Chaume, Frederic
1997 “Translating Non-verbal Information in Dubbing.” In Nonverbal Communication and Translation: New Perspectives and Challenges in Literature, Interpretation and the Media, edited by Fernando Poyatos, 315–326. Amsterdam: John Benjamins. Crossref logoGoogle Scholar
Cuenca, Maria Josep
2006 “Interjections and Pragmatic Errors in Dubbing.” Meta 51 (1): 20–35. Crossref logoGoogle Scholar
de Linde, Zoé
1995 “ ‘Read My Lips’ Subtitling Principles, Practices, and Problems.” Perspectives 3 (1): 9–20. Crossref logoGoogle Scholar
de Linde, Zoé, and Neil Kay
1999The Semiotics of Subtitling. Manchester: St. Jerome.Google Scholar
Eastman, Carol M.
1992 “Swahili Interjections: Blurring Language-Use/Gesture-Use Boundaries.” Journal of Pragmatics 18 (2–3): 273–287. Crossref logoGoogle Scholar
Ekman, Paul
2003Emotions Revealed: Recognizing Faces and Feelings to Improve Communication and Emotional Life. New York: Times Books.Google Scholar
Ekman, Paul, and Wallace V. Friesen
2003Unmasking the Face: A Guide to Recognizing Emotions from Facial Expressions. Cambridge, MA: Malor Books.Google Scholar
Feng, Dezheng, and Kay L. O’Halloran
2012 “Representing Emotive Meaning in Visual Images: A Social Semiotic Approach.” Journal of Pragmatics 44 (14): 2067–2084. Crossref logoGoogle Scholar
Gambier, Yves
2006 “Multimodality and Audiovisual Translation.” In Proceedings of the Marie Curie Euroconferences: MuTra – Audiovisual Translation Scenarios, Copenhagen, 1–5 May 2006, edited by Mary Carroll, Heidrun Gerzymisch-Arbogast, and Sandra Nauert. Accessed 14 December 2017. http://​www​.euroconferences​.info​/proceedings​/2006​_Proceedings​/2006​_Gambier​_Yves​.pdf
2013 “The Position of Audiovisual Translation Studies.” In The Routledge Handbook of Translation Studies, edited by Carmen Millán and Francesca Bartrina, 45–59. Abingdon: Routledge. Google Scholar
Georgakopoulou, Panayota
2009 “Subtitling for the DVD Industry.” In Audiovisual Translation: Language Transfer on Screen, edited by Jorge Díaz Cintas and Gunilla Anderman, 21–35. Basingstoke: Palgrave Macmillan. Crossref logoGoogle Scholar
Goddard, Cliff
2014 “Interjections and Emotion (with Special Reference to ‘Surprise’ and ‘Disgust’).” Emotion Review 6 (1): 53–63. Crossref logoGoogle Scholar
Gottlieb, Henrik
1994 “Subtitling: People Translating People.” In Teaching Translation and Interpreting 2: Insights, Aims, Visions. Papers from the Second Language International Conference, Elsinore, 1993, edited by Cay Dollerup and Annette Lindegaard, 261–274. Amsterdam: John Benjamins. Crossref logoGoogle Scholar
2018 “Semiotics and Translation.” In The Routledge Handbook of Translation Studies and Linguistics, edited by Kirsten Malmkjær, 45–63. Abingdon: Routledge.Google Scholar
Halliday, M. A. K., and Christian M. I. M. Matthiessen
2014Halliday’s Introduction to Functional Grammar. 4th ed. Abingdon: Routledge. Crossref logoGoogle Scholar
Jing, Yi
2020 “Interpersonal Functions of Interjections: A Systemic Functional Perspective.” Functions of Language. Published electronically, July 6. Crossref logoGoogle Scholar
Forthcoming. “Visual Affect in Films: A Semiotic Approach.” Semiotica.
Kovačič, Irena
1994 “Relevance as a Factor in Subtitling Reductions.” In Teaching Translation and Interpreting 2: Insights, Aims, Visions. Papers from the Second Language International Conference. Elsinore, 1993, edited by Cay Dollerup and Annette Lindegaard, 245–251. Amsterdam: John Benjamins. Crossref logoGoogle Scholar
1998 “Six Subtitlers – Six Subtitling Texts.” In Unity in Diversity? Current Trends in Translation Studies, edited by Lynne Bowker, Michael Cronin, Dorothy Kenny, and Jennifer Pearson, 75–82. Manchester: St. Jerome.Google Scholar
Luyken, Georg-Michael, Thomas Herbst, Jo Langham-Brown, Helen Reid, and Hermann Spinhof
1991Overcoming Language Barriers in Television: Dubbing and Subtitling for the European Audience. Manchester: European Institute for the Media.Google Scholar
Maestri, George
2005Digital Character Animation 3. Berkeley, CA: New Riders.Google Scholar
Martin, James R., and Peter R. R. White
2005The Language of Evaluation: Appraisal in English. London: Palgrave Macmillan. Crossref logoGoogle Scholar
Martinec, Radan
2001 “Interpersonal Resources in Action.” Semiotica 2001 (135): 117–145. Crossref logoGoogle Scholar
Matamala, Anna
2009 “Interjections in Original and Dubbed Sitcoms in Catalan: A Comparison.” Meta 54 (3): 485–502. Crossref logoGoogle Scholar
Matthiessen, Christian M. I. M.
2007 “The Multimodal Page: A Systemic Functional Exploration.” In New Directions in the Analysis of Multimodal Discourse, edited by Terry D. Royce and Wendy L. Bowcher, 1–62. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
McCloud, Scott
2006Making Comics: Storytelling Secrets of Comics, Manga and Graphic Novels. New York: Harper.Google Scholar
Pedersen, Jan
2005 “How Is Culture Rendered in Subtitles?” In Proceedings of the Marie Curie Euroconferences: MuTra – Challenges of Multidimensional Translation, Saarbrücken, 2–6 May 2005, edited by Heidrun Gerzymisch-Arbogast and Sandra Nauert. Accessed 14 December 2017. http://​www​.euroconferences​.info​/proceedings​/2005​_Proceedings​/2005​_Pedersen​_Jan​.pdf
2011Subtitling Norms for Television: An Exploration Focussing on Extralinguistic Cultural References. Amsterdam: John Benjamins. Crossref logoGoogle Scholar
Perego, Elisa
2009 “The Codification of Nonverbal Information in Subtitled Texts.” In New Trends in Audiovisual Translation, edited by Jorge Díaz Cintas, 58–69. Bristol: Multilingual Matters. Crossref logoGoogle Scholar
Quirk, Randolph, Sidney Greenbaum, Geoffrey Leech, and Jan Svartvik
1985A Comprehensive Grammar of the English Language. London: Longman.Google Scholar
Rosa, Alexandra Assis
2001 “Features of Oral and Written Communication in Subtitling.” In (Multi) Media Translation: Concepts, Practices, and Research, edited by Yves Gambier and Henrik Gottlieb, 213–221. Amsterdam: John Benjamins. Crossref logoGoogle Scholar
Stange, Ulrike
2016Emotive Interjections in British English: A Corpus-based Study on Variation in Acquisition, Function and Usage. Amsterdam: John Benjamins. Crossref logoGoogle Scholar
Szarkowska, Agnieszka
2013 “Towards Interlingual Subtitling for the Deaf and the Hard of Hearing.” Perspectives 21 (1): 68–81. Crossref logoGoogle Scholar
Taylor, Christopher
2003 “Multimodal Transcription in the Analysis, Translation and Subtitling of Italian Films.” The Translator 9 (2): 191–205. Crossref logoGoogle Scholar
2004 “Multimodal Text Analysis and Subtitling.” In Perspectives on Multimodality, edited by Eija Ventola, Cassily Charles, and Martin Kaltenbacher, 153–172. Amsterdam: John Benjamins. Crossref logoGoogle Scholar
2013 “Multimodality and Audiovisual Translation.” In Handbook of Translation Studies, Volume 4, edited by Yves Gambier and Luc van Doorslaer, 98–104. Amsterdam: John Benjamins. Crossref logoGoogle Scholar
2016 “The Multimodal Approach in Audiovisual Translation.” In Audiovisual Translation: Theoretical and Methodological Challenges, edited by Yves Gambier and Sara Ramos Pinto, special issue of Target 28 (2): 222–236. Crossref logoGoogle Scholar
Tian, Ping
2011Multimodal Evaluation: Sense and Sensibility in Anthony Browne’s Picture Books. PhD diss. University of Sydney.Google Scholar
Wilkins, David P.
1992 “Interjections as Deictics.” Journal of Pragmatics 18 (2–3): 119–158. Crossref logoGoogle Scholar
Zabalbeascoa, Patrick
1997 “Dubbing and the Nonverbal Dimension of Translation.” In Nonverbal Communication and Translation: New Perspectives and Challenges in Literature, Interpretation and the Media, edited by Fernando Poyatos, 327–342. Amsterdam: John Benjamins. Crossref logoGoogle Scholar