User-driven assessment of commercial term extractors
In this paper, we address the system evaluation issue for commercial term extraction tools from the users’
perspective. We first revisit the gold standard approach commonly practised among researchers, and discuss the challenges it may
pose on end users, taking translators as a typical example. Considering the very different motivations and needs of users and
researchers, a user-driven approach is proposed as a variation and alternative to the gold standard approach to allow users to
assess and understand the performance of commercial tools more objectively. Its feasibility and usefulness are demonstrated by
deploying a benchmarking dataset of English-Chinese financial terms, produced by multiple annotators, in a case study with SDL
MultiTerm Extract. The results also provide insight for future development of term extractors designed for translators, which will
hopefully generate more accurate candidates, offer more customised features, enable better user experience, and enjoy wider
popularity as a computer-aided translation tool.
Article outline
- 1.Introduction
- 2.Related work
- 2.1Automatic term extraction
- 2.2The issue of system evaluation
- 3.Creating the user-made benchmark
- 3.1The corpus
- 3.2English-Chinese financial terms in existing resources
- 3.3Term annotation guidelines
- Scope of terms
- Form of terms
- Span of terms
- 3.4The annotation and the resulting benchmark
- 4.Assessing systems with user-driven benchmarks
- 4.1SDL MultiTerm Extract
- 4.2Monolingual English term extraction
- 4.3Monolingual Chinese term extraction
- 4.4Bilingual English-Chinese term extraction
- 5.Discussion
- 5.1User-driven approach to accommodate individual needs
- 5.2An informal comparison with research-based systems
- 6.Conclusion
-
References