Cover not available
Article published In:
Methodological Issues in Experimental Research in Audiovisual Translation and Media Accessibility
Edited by Gian Maria Greco, Anna Jankowska and Agnieszka Szarkowska
[Translation Spaces 11:1] 2022
► pp. 89–112

Experimental research in automatic subtitling

At the crossroads between machine translation and audiovisual translation

ORCID logo | Fondazione Bruno Kessler | University of Trento

Recent developments in neural machine translation, and especially speech translation, are gradually but firmly entering the field of audiovisual translation (AVT). Automation in subtitling is extending from a machine translation (MT) component to fully automatic subtitling, which comprises MT, auto-spotting and automatic segmentation. The rise of this new paradigm renders MT-oriented experimental designs inadequate for the evaluation and investigation of automatic subtitling, since they fail to encompass the multimodal nature and technical requirements of subtitling. This paper highlights the methodological gaps to be addressed by multidisciplinary efforts in order to overcome these inadequacies and obtain metrics and methods that lead to rigorous experimental research in automatic subtitling. It presents a review of previous experimental designs in MT for subtitling, identifies their limitations for conducting research under the new paradigm and proposes a set of recommendations towards achieving replicability and reproducibility in experimental research at the crossroads between AVT and MT.

Keywords: Subtitling, Neural Machine Translation, speech translation, evaluation, post-editing
Article outline
  • 1.Introduction
  • 2.From MT for subtitles to automatic subtitling
  • 3.Review of previous experimental approaches in MT for subtitling
    • 3.1Automatic evaluation
    • 3.2Human evaluation
    • 3.3Other experimental aspects
  • 4.Issues and challenges in conducting experimental research in automatic subtitling
    • 4.1Variable number of segments
    • 4.2Auto-spotting and segmentation come into play
    • 4.3Disentangling the variables
  • 5.Experiences and recommendations
    • 5.1Research in automatic subtitling should comprise all dimensions of subtitling
    • 5.2Improved interfaces
    • 5.3Towards reporting standards
    • 5.4Test data and benchmarks
    • 5.5Evaluation should be independent of generation
  • 6.Conclusions
  • Notes
  • References
Published online: 14 June 2022
DOI logo
https://doi.org/10.1075/ts.21021.kar
References
Álvarez, Aitor, Marina Balenciaga, Arantza del Pozo, Haritz Arzelus, Anna Matamala, and Carlos-D. Martínez-Hinarejos
2016 Impact of Automatic Segmentation on the Quality, Productivity and Self-reported Post-editing Effort of Intralingual Subtitles. In Proceedings of the Tenth International Conference on Language Re-sources and Evaluation (LREC’16), Portorož, Slovenia, pp. 3049–3053. European Language Resources Association (ELRA).Google Scholar
Alves, Fabio
2003 Tradução, cognição e contextualidade: Triangulando a interface processo produto no desempenho de tradutores novos [Translation, cognition and contextualisation: Triangulating the process-product interface in the performance of novice translators]. Translation, cognition D.E.L.T.A. 191, 71–108.Google Scholar
Armstrong, Stephen, Colm Caffrey, and Marian Flanagan
2006 Translating DVD Subtitles from English-German and English-Japanese Using Example-Based Machine Translation. In MuTra 2006–Audiovisual Translation Scenarios: Conference Proceedings.Google Scholar
American Psychological Association-APA
2010 Publication Manual of the American Psychological Association (6th edition ed.). Washington: APA.Google Scholar
Aziz, Wilker, Sheila C. M. de Sousa, and Lucia Specia
2012 Cross-lingual Sentence Compression for Subtitles. In 16th Annual Conference of the European Association for Machine Translation, EAMT, Trento, Italy, pp. 103–110.Google Scholar
Barrault, Loїc, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank
2018 Findings of the Third Shared Task on Multimodal Machine Translation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, Belgium, Brussels, pp. 304–323. Association for Computational Linguistics. DOI logoGoogle Scholar
Belz, Anya, Simon Mille, and David M. Howcroft
2020 Disentangling the properties of human evaluation methods: A classification system to support comparability, meta-evaluation and reproducibility testing. In Proceedings of the 13th International Conference on Natural Language Generation, Dublin, Ireland, pp. 183–194. Association for Computational Linguistics.Google Scholar
Bentivogli, Luisa, Nicola Bertoldi, Mauro Cettolo, Marcello Federico, Matteo Negri, and Marco Turchi
2016 On the Evaluation of Adaptive Machine Translation for Human Post-Editing. IEEE/ACM Transactions on Audio, Speech, and Language Processing 24(2), 388–399. DOI logoGoogle Scholar
Bentivogli, Luisa, Mauro Cettolo, Marcello Federico, and Christian Federmann
2018 Machine translation human evaluation: An investigation of evaluation based on Post-editing and its relation with Direct Assessment. In Proceedings of the 15th International Workshop on Spoken Language Translation (IWSLT 2018).Google Scholar
Bolaños-García-Escribano, Alejandro, Jorge Díaz-Cintas, and Serenella Massidda
2021 Latest advancements in audiovisual translation education. The Interpreter and Translator Trainer 15(1), 1–12. DOI logoGoogle Scholar
Burchardt, Aljoscha, Arle Lommel, Lindsay Bywood, Kim Harris, and Maja Popović
2016 Machine translation quality in an audiovisual context. Target 28(2), 206–221. DOI logoGoogle Scholar
Bywood, Lindsay, Panayota Georgakopoulou, and Thierry Etchegoyhen
2017 Embracing the threat: machine translation as a solution for subtitling. Perspectives 25(3), 492–508. DOI logoGoogle Scholar
Carroll, Mary and Jan Ivarsson
1998 Code of Good Subtitling Practice. Simrishamn: TransEdit.Google Scholar
de Sousa, Sheila C. M., Wilker Aziz, and Lucia Specia
2011 Assessing the Post-Editing Effort for Automatic and Semi-Automatic Translations of DVD Subtitles. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pp. 97–103. Association for Computational Linguistics.Google Scholar
Doddington, George
2002 Automatic Evaluation of Machine Translation Quality Using N-Gram Co-Occurrence Statistics. In Proceedings of the Second International Conference on Human Language Technology Research, HLT ’02, San Francisco, CA, USA, pp. 138–145. Morgan Kaufmann Publishers Inc. DOI logoGoogle Scholar
Doherty, Stephen
2016 The impact of translation technologies on the process and product of translation. International Journal of Communication 10 1, 947–969.Google Scholar
Díaz-Cintas, Jorge
2013 The technology turn in subtitling. In Proceedings from the 5th International Maastricht – Lódź Duo Colloquium on Translation and Meaning, Part 9 1, pp. 119–132.Google Scholar
Etchegoyhen, Thierry, Lindsay Bywood, Mark Fishel, Panayota Georgakopoulou, Jie Jiang, Gerard van Loenhout, Arantza del Pozo, Mirjiam S. Maučec, Anja Turner, and Martin Volk
2014 Machine Translation for Subtitling: A Large-Scale Evaluation. In Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC), pp. 46–53.Google Scholar
Facebook
2020 “New Automated Captions Powered by AI”. Last accessed: 30/06/2021. [URL]
Hadley, James, Maja Popović, Haithem Afli, and Andy Way
(eds.) 2019 Proceedings of the Qualities of Literary Machine Translation, Dublin, Ireland. European Association for Machine Translation.Google Scholar
Howcroft, David M., Anja Belz, Miruna Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser
2020 Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings of the 13th International Conference on Natural Language Generation, Dublin, Ireland, pp. 169–182. Association for Computational Linguistics.Google Scholar
Hu, Ke, Sharon O’Brien, and Dorothy Kenny
2019 A reception study of machine translated subtitles for MOOCs. Perspectives, 1–18.Google Scholar
Karakanta, Alina, Matteo Negri, and Marco Turchi
2020a Is 42 the Answer to Everything in Subtitling-oriented Speech Translation? In Proceedings of the 17th International Conference on Spoken Language Translation, Online, pp. 209–219. Association for Computational Linguistics. DOI logoGoogle Scholar
2020b MuST-Cinema: a Speech-to-Subtitles corpus. In Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France, pp. 3727–3734. European Language Resources Association.Google Scholar
Karamitroglou, Fotios
1998 A proposed set of subtitling standards in Europe. Translation Journal.Google Scholar
Koponen, Maarit, Umut Sulubacak, Kaisa Vitikainen, and Jörg Tiedemann
2020 MT for subtitling: User evaluation of post-editing productivity. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT 2020), pp. 115–124.Google Scholar
Krings, Hans P.
(2001) Repairing texts: Empirical investigations of machine translation post-editing process. The Kent State University Press.Google Scholar
Lison, Pierre and Jörg Tiedemann
2016 OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Portorož, Slovenia, pp. 923–929. European Language Resources Association (ELRA).Google Scholar
Lo, Chi-kiu
2019 YiSi – a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), Florence, Italy, pp. 507–513. Association for Computational Linguistics. DOI logoGoogle Scholar
Matusov, Evgeny, Gregor Leusch, Oliver Bender, and Hermann Ney
2005 Evaluating Machine Translation Output with Automatic Sentence Segmentation. In Proceedings of the Second International Workshop on Spoken Language Translation, Pittsburgh, Pennsylvania, USA.Google Scholar
Matusov, Evgeny, Patrick Wilken, and Yota Georgakopoulou
2019 Customizing Neural Machine Translation for Subtitling. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), Florence, Italy, pp. 82–93. Association for Computational Linguistics. DOI logoGoogle Scholar
Melero, Maite, Antoni Oliver, and Toni Badia
(2006) Automatic Multilingual Subtitling in the eTITLE Project. In Proceedings of ASLIB Translating and the Computer 28 1.Google Scholar
Moorkens, Joss and Sharon O’Brien
(2017) Assessing User Interface Needs of Post-Editors of Machine Translation. Human Issues in Translation Technology: The IATIS Yearbook, 109–130.Google Scholar
Nikolic, Kristijan and Lindsay Bywood
2021 Audiovisual Translation: The Road Ahead. Journal of Audiovisual Translation 4(1), 50–70. DOI logoGoogle Scholar
Orero, Pilar, Stephen Doherty, Jan-Louis Kruger, Anna Matamala, Jan Pedersen, Elisa Perego, Pablo Romero-Fresco, Sara Rovira-Esteva, Olga Soler-Vilageliu, and Agnieszka Szarkowska
2018 Conducting experimental research in audiovisual translation (AVT): A position paper. Journal of Specialised Translation 301, 105–126.Google Scholar
Orrego-Carmona, David, Łukasz Dutka, and Agnieszka Szarkowska
2018 Using translation process research to explore the creation of subtitles: an eye-tracking study comparing professional and trainee subtitlers. Journal of Specialised Translation 0(30), 150–180.Google Scholar
O’Brien, Sharon
2021 Translation, human–computer interaction and cognition. In Fabio Alves and Arnt Lykke Jakobsen (eds.), The Routledge Handbook of Translation and Cognition, pp. 376–388. Routledge.Google Scholar
O’Hagan, Minako
2003 Can language technology respond to the subtitler’s dilemma? – a preliminary study. In Proceedings of the 25th International Conference on Translation and the Computer.Google Scholar
Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu
2002 BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pp. 311–318. Association for Computational Linguistics.Google Scholar
Pedersen, Jan
2017 The FAR Model: Assessing Quality in Interlingual Subtitling. The Journal of Specialised Translation 281, 210–229.Google Scholar
Piperidis, Stelios, Iason Demiros, and Prokopis Prokopidis
2005 Infrastructure for a Multilingual Subtitle Generation System. In 9th International Symposium on Social Communication, pp. 24–28.Google Scholar
Popović, Maja
2015 chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisbon, Portugal, pp. 392–395. Association for Computational Linguistics. DOI logoGoogle Scholar
Popowich, Fred, Paul McFetridge, Davide Turcato, and Janine Toole
2000 Machine Translation of Closed Captions. Machine Translation, 311–341. DOI logoGoogle Scholar
Rei, Ricardo, Craig Stewart, Ana C. Farinha, and Alon Lavie
2020 COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 2685–2702. Association for Computational Linguistics. DOI logoGoogle Scholar
Snover, Matthew, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul
2006 A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, Cambridge, Massachusetts, USA, pp. 223–231. Association for Machine Translation in the Americas.Google Scholar
Szarkowska, Agnieszka, Jorge Diaz-Cintas, and Olivia Gerber-Morón
2020 Quality is in the eye of the stakeholders: what do professional subtitlers and viewers think about subtitling? Universal Access in the Information Society 201, 661–675. DOI logoGoogle Scholar
Tadmor-Ramanovich, Michelle and Nadav Bar
2019 “On-Device Captioning with Live Caption.” Google AIBlog. Last accessed: 30/06/2021. [URL]
Volk, Martin, Rico Sennrich, Christian Hardmeier, and Frida Tidström
2010 Machine Translation of TV Subtitles for Large Scale Production. In Ventsislav Zhechev (ed.), Proceedings of the Second Joint EM+/CNGL Workshop ”Bringing MT to the User: Research on Integrating MT in the Translation Industry (JEC’10), Denver, pp. 53–62.Google Scholar
Wang, Weiyue, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney
2016 CharacTer: Translation Edit Rate on Character Level. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, Berlin, Germany, pp. 505–510. Association for Computational Linguistics. DOI logoGoogle Scholar
Wang, Xin, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang
2019 VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research. In The IEEE International Conference on Computer Vision (ICCV). DOI logoGoogle Scholar