Cover not available
Article published In:
FORUM
Vol. 21:2 (2023) ► pp.236–257

A quality assessment of Korean–English patent machine translation

Automatic and human evaluations of K2E-PAT, Patent Translate and WIPO Translate translations

| Ewha Womans University
| Ewha Womans University

This paper aims to investigate the quality Korean–English patent translations by three machine translation (MT) engines based on automatic and human evaluations of Korean to English Patent Automatic Translation (K2E-PAT), a pattern-based statistical MT; and Patent Translate and WIPO Translate, both neural MTs. For title translations, WIPO Translate scored the highest in automatic and human evaluations, while results were mixed for the other two MTs. K2E-PAT slightly outperformed Patent Translate in automatic evaluation, whereas Patent Translate outperformed K2E-PAT in human evaluation. For abstract translations, Patent Translate scored the highest in automatic evaluation, followed by WIPO Translate and K2E-PAT. In human evaluation, the ranking order was the same as that of title translations, with WIPO Translate scoring the highest on average. The results indicated correlations between automatic and human evaluations, and the NMTs subject to the current study still do not render satisfactory gist translation from Korean to English.

Keywords: patent machine translation, machine translation quality evaluation, K2E-PAT, Patent Translate, WIPO Translate
Article outline
  • 1.Introduction
  • 2.Evaluation of machine translation
  • 3.Patent machine translation quality assessment
  • 4.Methods
  • 5.Results
    • 5.1Translation of titles
    • 5.2Translation of abstracts
    • 5.3Error analysis
      • 5.3.1Error types
      • 5.3.2Frequent errors in title translations
      • 5.3.3Frequent errors in abstract translations
  • 6.Discussion and conclusions
  • Notes
  • References
Published online: 14 November 2023
DOI logo
https://doi.org/10.1075/forum.00030.lee
References
Adams, Stephen
2020 Information Sources in Patents. Berlin: Walter de Gruyter GmbH & Co KG. DOI logoGoogle Scholar
Banerjee, Satanjeev, and Alon Lavie
2005 “METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments.” In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, Ann Arbor, Michigan, 29 June, ed. by Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss, 65–72. New Jersey: The Association for Computational Linguistics.Google Scholar
Bazrafshan, Marzieh
2014 Semantic Features for Statistical Machine Translation. PhD diss. University of Rochester.
Brkić, Marija, Sanja Seljan, and Tomislav Vičić
2013 “Automatic and Human Evaluation on English-Croatian Legislative Test Set.” In Computational Linguistics and Intelligent Text Processing. ed. by Alexander Gelbukh, 311–317. Berlin, Heidelberg: Springer. DOI logoGoogle Scholar
Callison-Burch, Chris, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder
2007 “(Meta-)evaluation of Machine Translation.” In Proceedings of the Second Workshop on Statistical Machine Translation, Prague, Czech Republic, 23 June, ed. by Chris Callison-Burch, Philipp Koehn, Christof Monz, and Cameron Shaw Fordyce, 136–158. Pennsylvania: Association for Computational Linguistics. DOI logoGoogle Scholar
Castilho, Sheila, et al.
2017 “Is Neural Machine Translation the New State of the Art?” The Prague Bulletin of Mathematical Linguistics 1081: 109–120. DOI logoGoogle Scholar
2018 “Approaches to Human and Machine Translation Quality Assessment.” In Translation Quality Assessment, ed. by Joss Moorkens, Sheila Castilho, Federico Gaspari and Stephen Doherty, 9–38. Cham: Springer. DOI logoGoogle Scholar
Chatzikoumi, Eirini
2020 “How to Evaluate Machine Translation: A Review of Automated and Human Metrics.” Natural Language Engineering 261: 137–161. DOI logoGoogle Scholar
Choi, Hyoeun, and Jieun Lee
2017 “A Study on the Evaluation of Korean-English Patent Machine Translation–Focusing on KIPRIS K2E-PAT Translation.” Interpretation and Translation 19 (1): 139–178. DOI logoGoogle Scholar
Choi, Hyoeun, Chung-ho Lee, and Jun-ho Lee
2023 “Evaluation of English-Korean Patent Machine Translations by a Patent-Specific NMT Engine Using AutoML.” Journal of Translation Studies 24(2): 101–130.Google Scholar
Choi, YooChan
2009 “Korean to English Patent Automatic Translation (K2E-PAT) and Cross-Lingual Retrieval on KIPRIS.” World Patent Information 311: 135–136. DOI logoGoogle Scholar
Coughlin, Deborah
2003 “Correlating Automated and Human Assessments of Machine Translation Quality.” Paper presented at MT Summit IX, New Orleans, US.
Encyclopaedia Britannica
2022 “Electrostriction.” Accessed April 4, 2022. [URL]
European Patent Office
2021 “Patent Translate.” Accessed February 17, 2022. [URL]
Forcada, Mikel L.
2017 “Making Sense of Neural Machine Translation.” Translation Spaces 6 (2): 291–309. DOI logoGoogle Scholar
Kinoshita, Satoshi, Tadaaki Oshio, and Tomoharu Mitsuhashi
2017 “Comparison of SMT and NMT trained with large Patent Corpora: Japio at WAT2017.” In Proceedings of the 4th Workshop on Asian Translation, Taipei, Taiwan, 27 November–1 December, ed. by Toshiaki Nakazawa et al., 140–145. Singapore: Asian Federation of Natural Language Processing.Google Scholar
Koehn, Philipp, and Rebecca Knowles
2017 “Six Challenges for Neural Machine Translation.” In Proceedings of the First Workshop on Neural Machine Translation, Vancouver, Canada, 4 August, ed. by Minh-Thang Luong, Alexandra Birch, Grahan Neubig, and Andrew Finch, 28–39. Pennsylvania: Association for Computational Linguistics. DOI logoGoogle Scholar
Koehn, Philipp et al.
2007 “Moses: Open-Source Toolkit for Statistical Machine Translation.” Paper presented at Annual Meeting of the Association for Computational Linguistics (ACL), demonstration session.
Korea Institute of Patent Information (KIPI)
(2018) “KIPRIS Brochure.” Accessed February 17, 2022. [URL]
Korea Intellectual Property Office (KIPO) and Korea Institute of Patent Information (KIPI)
2021 “Intellectual Property Statistics Service.” Accessed February 17, 2022. [URL]
Lavie, Alon, and Abhaya Agarwal
2007 “METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments.” In Proceedings of the Second Workshop on Statistical Machine Translation, Prague, Czech Republic, 23 June, ed. by Chris Callison-Burch, Philipp Koehn, Christof Monz, and Cameron Shaw Fordyce, 228–231. Pennsylvania: Association for Computational Linguistics. DOI logoGoogle Scholar
Lavie, Alon
2013 “MT Evaluation: Human Measures and Assessment Methods.” Accessed February 24, 2022. [URL]
Lee, Jieun, and Hyoeun Choi
2023 “A Case Study of the Evaluation of Legal and Patent Korean-English Machine Translations by a Domain-Specific NMT Engine.” Journal of Translation Studies 24(1): 9–37.Google Scholar
Lommel, Arle
2018 “Metrics for Translation Quality Assessment: A Case for Standardising Error Typologies.” In Translation Quality Assessment, ed. by Joss Moorkens, Sheila Castilho, Federico Gaspari, and Stephen Doherty, 109–208. Cham: Springer. DOI logoGoogle Scholar
Long, Zi et al.
2016 “Translation of Patent Sentences with a Large Vocabulary of Technical Terms Using Neural Machine Translation.” In Proceedings of the 3rd Workshop on Asian Translation (WAT2016), Osaka, Japan, 11–16 December, 47–57. Pennsylvania: Association for Computational Linguistics.Google Scholar
Matsutani, Yohei
2019 “Utilization of Machine Translation in the Japan Patent Office toward Improvement of Accessibility to Patent Information.” Accessed February 17, 2022. [URL]
Nurminen, Mary
2019 “Decision-making, risk, and gist machine translation in the work of patent professionals.” In Proceedings of the 8th Workshop on Patent and Scientific Literature Translation, Dublin, Ireland, 20 August, ed. by Takehito Utsuro, Katsuhito Sudoh, and Takashi Tsunakawa, 32–42. European Association for Machine Translation.Google Scholar
2020 “Raw Machine Translation Use by Patent Professionals: A Case of Distributed Cognition.” Translation, Cognition & Behaviour 3 (1): 100–121. DOI logoGoogle Scholar
Olohan, Maeve
2015 Scientific and Technical Translation. London: Routledge. DOI logoGoogle Scholar
Panic, Milica
2020 “Automated MT Evaluation Metrics.” Accessed February 17, 2022. [URL]
Poibeau, Thierry
2017 Machine Translation. Cambridge: The MIT Press. DOI logoGoogle Scholar
Popović, Maja
2017 “Comparing Language Related Issues for NMT and PBMT between German and English.” Prague Bull Math Linguist 108 (1): 209–220. DOI logoGoogle Scholar
2018 “Error Classification and Analysis for Machine Translation Quality Assessment.” In Translation Quality Assessment, ed. by Joss Moorkens, Sheila Castilho, Federico Gaspari, and Stephen Doherty, 129–158. Cham: Springer. DOI logoGoogle Scholar
Premoli, Valeria, Elena Murgulo, and Diego Cresceri
2019 “MTPE in Patents: A Successful Business Story.” In Proceedings of MT Summit XVII, vol. 2, Dublin, Ireland, 19–23 August, edited by Mikel Forcada, Andy Way, Barry Haddow, and Rico Sennrich, 36–41. European Association for Machine Translation.Google Scholar
Qun, Liu, and Zhang Xiaojun
2014 “Machine Translation.” In The Routledge Encyclopedia of Translation Technology, ed. by Chan Sin-Wai, 105–119. London: Routledge.Google Scholar
Rossi, Laura, and Dion Wiggins
2013 “Applicability and Application of Machine Translation Quality Metrics in the Patent Field.” World Patent Information 351: 115–125. DOI logoGoogle Scholar
Snover, Matthew et al.
2006 “A Study of Translation Edit Rate with Targeted Human Annotation.” In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, Cambridge, USA., 8–12 August. ed. by Laurie Gerber, 223–231. Association for Machine Translation in the Americas.Google Scholar
Specia, Lucia, Dhwaj Raj, and Marco Turchi
2010 “Machine Translation Evaluation versus Quality Estimation.” Machine Translation 241: 39–50. DOI logoGoogle Scholar
Tinsley, John et al.
2012 “IP Translator: Facilitating Patent Search with Machine Translation.” In Proceedings of the 10th Conference of the Association for Machine Translation in the Americas: Commercial MT User Program, San Diego, USA., 28 October–1 November. Association for Machine Translation in the Americas. Accessed July 31, 2022. [URL]
Toral, Antonio, and Víctor M. Sánchez-Cartagena
2017 “A Multifaceted Evaluation of Neural versus Phrase-based Machine Translation for 9 Language Directions.” In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Valencia, Spain, 3–7 April, edited by Mirella Lapata, Phil Blunsom, and Alexander Koller, 1063–1073. Pennsylvania: Association for Computational Linguistics. DOI logoGoogle Scholar
Wang, Xiaolin, Andrew, Finch Masao Utiyama, Taro Watanabe, and Eiichiro Sumita
2014 “The NICT translation system for IWSLT 2014.” In Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign, 139–142.
Wang, Dan
2009 “Chinese to English Automatic Patent Machine Translation at SIPO.” World Patent Information 31(2): 137–139. DOI logoGoogle Scholar
World Intellectual Property Organization (WIPO)
2014 “Patent Cooperation Treaty: Yearly Review.” Accessed February 17, 2022. [URL]
World Intellectual Property Organization
2020a “WIPO Translate: Terms and Conditions for the Usage and User Guide.” Accessed February 17, 2022. [URL]
2020b “WIPO IP Facts and Figures 2020.” Accessed February 17, 2022. [URL]
2020c “World Intellectual Property Indicators 2020.” Accessed February 17, 2022. [URL]
Ying, Cheng et al.
2021 “Errors of Machine Translation of Terminology in the Patent Text from English into Chinese.” ASP Transactions on Computers 1 (1): 12–17. DOI logoGoogle Scholar