Jinbiao Yang

List of John Benjamins publications for which Jinbiao Yang plays a role.

Article

Cover not available
Yang, Jinbiao 2024 Rethinking tokenization: Crafting better tokenizers for large language models The Unit of Processing in Chinese, Wang, Tianlin (ed.), pp. 94–109 | Article
Tokenization significantly influences language models (LMs)’ performance. This paper traces the evolution of tokenizers from word-level to subword-level, analyzing how they balance tokens and types to enhance model adaptability while controlling complexity. Despite subword tokenizers like Byte… read more