Cover not available
Article published In:
Text and Context Revisited Within a Multimodal Framework
Edited by Yves Gambier and Olli Philippe Lautenbacher
[Babel 70:1/2] 2024
► pp. 138–163

Towards a corpus-based approach to graphic elements in creative subtitling

A case study of the YouTube channel “Apenjie with Dawang”

| The Hong Kong Polytechnic University

This article explores how a corpus-based approach allows us to describe and analyze the multimodal complexity of graphic elements in creative subtitling. To this end, the article focuses on a YouTube channel, Apenjie with Dawang, featuring a dog and its owner. This channel’s subtitling strategies were experimental with multiple graphic elements (colors, positions, font sizes, and emojis). Informed by a social semiotic approach to multimodality, a corpus of 1,155 coupled pairs of Chinese-English subtitles was annotated for modal shifts and metafunctional shifts. Some major findings include: (a) emojis were much more likely to be added to the target subtitles for the animals than for the humans; (b) speaker-identifying graphic elements (color and position) were lost in the target subtitles, but emojis were systematically added to mark animals as the speakers; (c) the addition of emojis evoked complementary-interpersonal meanings, suggesting that the subtitlers might have prioritized audience engagement over textual fidelity; (d) although the target subtitles used fewer graphic elements, the semiotic meanings could be similar or complementary to those of the source subtitles. Based on these findings, the article also discusses the opportunities and challenges of a corpus-based approach to graphic elements in creative subtitling.

Keywords: creative subtitling, corpus, graphic elements, multimodality, metafunction, mode
Article outline
  • 1.Introduction
  • 2.Research background
    • 2.1A social semiotic approach to multimodality
    • 2.2A corpus-based approach to subtitles
  • 3.The study
    • 3.1Corpus
    • 3.2Coding framework
    • 3.3Coding and analytical procedures
  • 4.Results
    • 4.1Within-group comparisons
    • 4.2Inter-group comparisons
  • 5.Discussion
    • 5.1Graphic elements in creative subtitling
    • 5.2Opportunities and challenges of a corpus-based approach
  • 6.Conclusion
  • Notes
  • References
Published online: 15 December 2023
DOI logo
https://doi.org/10.1075/babel.00382.wu
References

Primary sources

Video links for the examples in Figure 3
Example 1 [URL] (2:44)
Example 2 [URL] (1:20)
Example 3 https://www.youtube.com/watch?v=G_rOvmYpwZ8 (0:12)
Example 4 [URL] (0:05)

Secondary sources

Agresti, Alan
2013 Categorical Data Analysis. Hoboken: John Wiley & Sons.Google Scholar
Baños, Rocío, Silvia Bruti, and Serenella Zanotti
2013 “Corpus Linguistics and Audiovisual Translation: In Search of an Integrated Approach.” Perspectives 21 (4): 483–490. DOI logoGoogle Scholar
Bassnett, Susan, Lawrence Venuti, Jan Pedersen, and Ivana Hostová
2022 “Translation and Creativity in the 21st Century.” World Literature Studies 14 (1): 3–17. DOI logoGoogle Scholar
Bateman, John
2008 Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents. London: Palgrave Macmillan. DOI logoGoogle Scholar
Bateman, John A.
2014 “Using Multimodal Corpora for Empirical Research.” In The Routledge Handbook of Multimodal Analysis, edited by Carey Jewitt, 238–252. London: Routledge.Google Scholar
Beckman, Rachel
2008 “An Out-of-Character Role for Subtitles.” Washington Post 14 November. [URL]
Bernardini, Silvia, and Adriano Ferraresi
2022 “Corpus Linguistics.” In The Routledge Handbook of Translation and Methodology, edited by Federico Zanettin and Christopher Rundle, 207–222. London: Routledge. DOI logoGoogle Scholar
Brookes, Gavin, and Tony McEnery
2020 “Corpus Linguistics.” In The Routledge Handbook of English Language and Digital Humanities, edited by Svenja Adolphs and Dawn Knight, 378–404. London: Routledge. DOI logoGoogle Scholar
Bruti, Silvia
2020 “Corpus Approaches and Audiovisual Translation.” In The Palgrave Handbook of Audiovisual Translation and Media Accessibility, edited by Łukasz Bogucki and Mikołaj Deckert, 381–395. Cham: Palgrave Macmillan. DOI logoGoogle Scholar
Bywood, Lindsay
2019 “Testing The Retranslation Hypothesis for Audiovisual Translation: The Films of Volker Schlöndorff Subtitled into English.” Perspectives 27 (6): 815–832. DOI logoGoogle Scholar
Chen, Yuping
2019 Translating Film Subtitles into Chinese: A Multimodal Study. Singapore: Springer. DOI logoGoogle Scholar
Cohn, Neil
2013 “Beyond Speech Balloons and Thought Bubbles: The Integration of Text and Image.” Semiotica 1971: 35–63. DOI logoGoogle Scholar
Dahne, Annie, and Roberta Piazza
2020 “ ‘Subtitles Have to Become My Ears Not My Eyes’ Pragmatic-Stylistic Choices behind Closed Captions for the Deaf and Hard of Hearing: The Example of Breaking Bad .” In Telecinematic Stylistics, edited by Christian Hoffmann and Monika Kirner-Ludwig, 285–306. London: Bloomsbury. DOI logoGoogle Scholar
Danesi, Marcel
2017 The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London: Bloomsbury. DOI logoGoogle Scholar
De Linde, Zoé, and Neil Kay
1999 “Processing Subtitles and Film Images: Hearing vs Deaf Viewers.” The Translator 5 (1): 45–60. DOI logoGoogle Scholar
Díaz Cintas, Jorge
2018 “ ‘Subtitling’s a Carnival’: New Practices in Cyberspace.” The Journal of Specialised Translation 301: 127–149.Google Scholar
Díaz Cintas, Jorge, and Aline Remael
2021 Subtitling: Concepts and Practices. London: Routledge.Google Scholar
Dwyer, Tessa
2017 Speaking in Subtitles: Revaluing Screen Translation. Edinburgh: Edinburgh University Press.Google Scholar
Field, Andy
2018 Discovering Statistics Using IBM SPSS Statistics, 5th ed. Los Angeles: Sage.Google Scholar
Foerster, Anna
2010 “Towards a Creative Approach in Subtitling: A Case Study.” In New Insights into Audiovisual Translation and Media Accessibility, edited by Jorge Díaz Cintas, Anna Matamala, and Josélia Neves, 81–98. Amsterdam: Rodopi. DOI logoGoogle Scholar
Fox, Wendy
2016 “Integrated Titles: An Improved Viewing Experience.” In Eyetracking and Applied Linguistics, edited by Silvia Hansen-Schirra and Sambor Grucza, 5–29. Berlin: Language Science Press.Google Scholar
2018 Can Integrated Titles Improve the Viewing Experience? Investigating the Impact of Subtitling on the Reception and Enjoyment of Film Using Eye Tracking and Questionnaire Data. Berlin: Language Science Press.Google Scholar
Gambier, Yves
2003 “Introduction: Screen Transadaptation: Perception and Reception. Special Issue on Screen Translation.” The Translator 9 (2): 171–190. DOI logoGoogle Scholar
2013 “The Position of Audiovisual Translation Studies.” In The Routledge Handbook of Translation Studies, edited by Carmen Millán and Francesca Bartrina, 45–59. London: Routledge. DOI logoGoogle Scholar
Guillot, Marie-Noëlle
2019 “Subtitling on the Cusp of its Futures.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-González, 31–47. London: Routledge.Google Scholar
Halliday, Michael Alexander Kirkwood
1978 Language as Social Semiotic: The Social Interpretation of Language and Meaning. London: Edward Arnold.Google Scholar
2001 “Towards a Theory of Good Translation.” In Exploring Translation and Multilingual Text Production: Beyond Content, edited by Erich Steiner and Colin Yallop, 13–18. Berlin: Mouton de Gruyter. DOI logoGoogle Scholar
Halliday, Michael Alexander Kirkwood, and Christian M. I. M. Matthiessen
2014 Halliday’s Introduction to Functional Grammar. London: Routledge. DOI logoGoogle Scholar
Hart, Christopher
2020 “Multimodal Discourse Analysis.” In Researching Discourse: A Student Guide, edited by Christopher Hart, 143–179. London: Routledge. DOI logoGoogle Scholar
Jewitt, Carey, Jeff Bezemer, and Kay O’Halloran
2016 Introducing Multimodality. London: Routledge. DOI logoGoogle Scholar
Kress, Gunther
2010 Multimodality: A Social Semiotic Approach to Contemporary Communication. London: Routledge.Google Scholar
2014 “What Is a Mode?” In The Routledge Handbook of Multimodal Analysis, edited by Carey Jewitt, 2nd ed., 60–75. London: Routledge.Google Scholar
2020 “Transposing Meaning: Translation in a Multimodal Semiotic Landscape.” In Translation and Multimodality: Beyond Words, edited by Monica Boria, Ángeles Carreres, María Noriega-Sánchez, and Marcus Tomalin, 24–48. London: Routledge.Google Scholar
Kress, Gunther, and Theo van Leeuwen
2006 Reading Images: The Grammar of Visual Design, 2nd ed. London: Routledge. DOI logoGoogle Scholar
Kruger, Jan-Louis
2021 “Translation, Multimodality and Cognition.” In The Routledge Handbook of Translation and Cognition, edited by Fabio Alves and Arnt Lykke Jakobsen, 433–444. London: Routledge.Google Scholar
Levshina, Natalia
2017 “Online Film Subtitles as A Corpus: An N-Gram Approach.” Corpora 12 (3): 311–338. DOI logoGoogle Scholar
Logi, Lorenzo, and Michele Zappavigna
2021 “A Social Semiotic Perspective on Emoji: How Emoji and Language Interact to Make Meaning in Digital Messages.” New Media & Society 25 (12): 3222–3246. DOI logoGoogle Scholar
Malamatidou, Sofia
2018 Corpus Triangulation: Combining Data and Methods in Corpus-Based Translation Studies. London: Routledge.Google Scholar
Matthiessen, Christian M. I. M.
2021 “Translation, Multilingual Text Production and Cognition Viewed in Terms of Systemic Functional Linguistics.” In The Routledge Handbook of Translation and Cognition, edited by Fabio Alves and Arnt Lykke Jakobsen, 517–544. London: Routledge.Google Scholar
McClarty, Rebecca
2012 “Towards A Multidisciplinary Approach in Creative Subtitling.” MonTI: Monografías de traducción e interpretación 41: 133–153. DOI logoGoogle Scholar
2014 “In Support of Creative Subtitling: Contemporary Context and Theoretical Framework.” Perspectives 22 (4): 592–606. DOI logoGoogle Scholar
McEnery, Tony, and Andrew Hardie
2012 Corpus Linguistics: Method, Theory and Practice. Cambridge: Cambridge University Press.Google Scholar
Miles, Matthew B., and A. Michael Huberman
1994 Qualitative Data Analysis: An Expanded Sourcebook. 2 ed. Thousand Oaks, CA: Sage.Google Scholar
Mubenga, Kajingulu
2009 “Towards a Multimodal Pragmatic Analysis of Film Discourse in Audiovisual Translation.” Meta 54 (3): 466–484. DOI logoGoogle Scholar
Nornes, Abé Mark
1999 “For an Abusive Subtitling.” Film Quarterly 52 (3): 17–34. DOI logoGoogle Scholar
Pavesi, Maria
2019 “Corpus-Based Audiovisual Translation Studies: Ample Room for Development.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-González, 315–333. London: Routledge.Google Scholar
2022 “Corpora and The Language of Films: Exploring Dialogue in English and Italian.” In The Routledge Handbook of Corpus Linguistics, 2nd ed., edited by Anne O’Keeffe and Michael J. McCarthy, 547–561. London: Routledge. DOI logoGoogle Scholar
Pedersen, Jan
2011 Subtitling Norms for Television: An Exploration Focussing on Extralinguistic Cultural References. Amsterdam and Philadelphia: John Benjamins Publishing Company. DOI logoGoogle Scholar
Perego, Elisa
2019 “Into the Language of Museum Audio Descriptions: A Corpus-Based Study.” Perspectives 27 (3): 333–349. DOI logoGoogle Scholar
Pérez-González, Luis
2007 “Appraising Dubbed Conversation: Systemic Functional Insights into the Construal of Naturalness in Translated Film Dialogue.” The Translator 13 (1): 1–38. DOI logoGoogle Scholar
2014 Audiovisual Translation: Theories, Methods and Issues. London: Routledge. DOI logoGoogle Scholar
Pinnow, Rachel J.
2015 “A Patriot is Respectful: (Re-)Examining the Architecture of Ideology in Educational Contexts.” In International Handbook of Semiotics, edited by Peter Pericles Trifonas, 1249–1269. New York: Springer. DOI logoGoogle Scholar
Pinto, Sara Ramos, and Aishah Mubaraki
2020 “Multimodal Corpus Analysis of Subtitling: The Case of Non-Standard Varieties.” Target 32 (3): 389–419. DOI logoGoogle Scholar
Remael, Aline, and Nina Reviers
2020 “Media Accessibility and Accessible Design.” In The Routledge Handbook of Translation and Technology, edited by Minako O’Hagan, 482–497. London: Routledge.Google Scholar
Reviers, Nina
2018 “Studying the Language of Dutch Audio Description: An Example of a Corpus-Based Analysis.” Translation and Translanguaging in Multilingual Contexts 41: 178–202. DOI logoGoogle Scholar
Romero-Fresco, Pablo
2009 “Naturalness in the Spanish Dubbing Language: A Case of Not-so-close Friends,” Meta 54 (1), 49–72. DOI logoGoogle Scholar
Secară, Alina
2011 “RU Ready 4 New Subtitles? Investigating the Potential of Social Translation Practices and Creative Spellings.” Linguistica Antverpiensia, New Series–Themes in Translation Studies 101: 153–171.Google Scholar
Soffritti, Marcello
2019 “Multimodal Corpora in Audiovisual Translation Studies.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-González, 334–349. London: Routledge.Google Scholar
Sun, Juhyung, Sarah Lasser, and Sun Kyong Lee
2022 “Understanding Emojis: Cultural Influences in Interpretation and Choice of Emojis.” Journal of International and Intercultural Communication 16 (3): 246–261.Google Scholar
Taylor, Christopher
2003 “Multimodal Transcription in the Analysis, Translation and Subtitling of Italian Films.” The Translator 9 (2): 191–205. DOI logoGoogle Scholar
2013 “Multimodality and Audiovisual Translation.” In Handbook of Translation Studies, vol. 41, edited by Yves Gambier and Luc van Doorslaer, 98–104. Amsterdam and Philadelphia: John Benjamins Publishing Company. DOI logoGoogle Scholar
2016 “The Multimodal Approach in Audiovisual Translation.” Target 28 (2): 222–236. DOI logoGoogle Scholar
2020 “Multimodality and Intersemiotic Translation.” In The Palgrave Handbook of Audiovisual Translation and Media Accessibility, edited by Łukasz Bogucki and Mikołaj Deckert, 83–99. Cham: Palgrave Macmillan. DOI logoGoogle Scholar
Thompson, Paul
2022 “Building A Specialised Audiovisual Corpus.” In The Routledge Handbook of Corpus Linguistics (2nd edition), edited by Anne O’Keeffe and Michael J. McCarthy, 75–88. London: Routledge. DOI logoGoogle Scholar
Tirkkonen-Condit, Sonja, and Jukka Mäkisalo
2007 “Cohesion in Subtitles: A Corpus-Based Study.” Across Languages and Cultures 8 (2): 221–230. DOI logoGoogle Scholar
Tognini-Bonelli, Elena
2001 Corpus Linguistics at Work. Amsterdam and Philadelphia: John Benjamins Publishing Company. DOI logoGoogle Scholar
Toury, Gideon
2012 Descriptive Translation Studies and Beyond, rev. ed. Amsterdam and Philadelphia: John Benjamins Publishing Company. DOI logoGoogle Scholar
Tuominen, Tiina, Catalina Jiménez Hurtado, and Anne Ketola
2018 “Why Methods Matter: Approaching Multimodality in Translation Research.” Linguistica Antverpiensia: New Series–Themes in Translation Studies 171: 1–21.Google Scholar
Valentini, Cristina
2007 “Developing AVT Corpora for a Quantitative Approach to Language Transfer in Cinematic Products.” In Selected Papers of the CETRA Research Seminar in Translation Studies 2006, edited by Francis Mus, 1–18. Leuven: CETRA.Google Scholar
Wu, Zhiwei, and Zhuojia Chen
2022 “Towards a Corpus-Driven Approach to Audiovisual Translation (AVT) Reception: A Case Study of YouTube Viewer Comments.” The Journal of Specialised Translation 381: 128–154.Google Scholar
Zanettin, Federico
2012 Translation-Driven Corpora: Corpus Resources for Descriptive and Applied Translation Studies. Manchester: St. Jerome.Google Scholar