Chapter published in:
Parallel Corpora for Contrastive and Translation Studies: New resources and applications
Edited by Irene Doval and M. Teresa Sánchez Nieto
[Studies in Corpus Linguistics 90] 2019
► pp. 281–298

Normalization of shorthand forms in French text messages using word embedding and machine translation

| Neue Zürcher Zeitung, KOF Swiss Economic Institute
| Neue Zürcher Zeitung, KOF Swiss Economic Institute
This chapter focuses on the normalization of abbreviations and shorthand forms used in French text messages. These forms are difficult to normalize, as they mostly cannot be resolved by typical spell checkers and dictionary lookups. Firstly, we aligned normalized and non-normalized French text messages and built a parallel corpus. We applied two popular approaches for text normalization, namely multilingual word embeddings, and character-based machine translation. We compare our results and observe the efficacy of our models while normalizing deletions, substitutions, repetitions, swaps, and insertions, made to canonical forms. This is the first paper that uses Multivec and the Belgian SMS corpus collected under the SMS4Science Project. The unsupervised machine learning approach makes the system highly flexible, easily adaptable and provides a domain-independent method of text normalization.
Keywords: SMS, parallel corpus, French, abbreviation/shorthand form normalization, unsupervised learning, character-based machine translation, distributional semantics, Multivec, neural networks, deep learning, word embeddings
Article outline
  • 1.Introduction
  • 2.Previous work
  • 3.Corpus and preprocessing
    • 3.1Corpus
    • 3.2Preprocessing
  • 4.Methodologies, tools and experiments
    • 4.1Methodologies
    • 4.2Tools and experiments
      • multivec
      • moses
  • 5.Results analysis
  • 6.Conclusion
  • 7.Future work
  • Acknowledgment
  • Notes
  • References
Published online: 20 March 2019
https://doi.org/10.1075/scl.90.17gho
References
Beaufort, Richard, Roekhaut, Sophie, Cougnon, Louise-Amélie & Fairon, Cédrick
2010 A hybrid rule/model-based finite-state framework for normalizing SMS Messages. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, 770–779.Google Scholar
Bérard, Alexandre, Servan, Christophe, Pietquin, Olivier & Besacier, Laurent
2016 Multivec: A multilingual and multilevel representation learning toolkit for NLP. The 10th edition of the Language Resources and Evaluation Conference, 4188–4192.Google Scholar
Bird, Steven, Loper, Edward & Klein, Ewan
2009 Natural Language Processing with Python. San Francisco CA: O’Reilly Media.Google Scholar
Bojanowski, Piotr, Grave, Edouard, Joulin, Armand & Mikolov, Tomas
2016 Enriching Word Vectors with Subword Information. https://​arxiv​.org​/abs​/1607​.04606> (13 May 2017).Google Scholar
Choudhury, Monojit, Saraf, Rahul, Jain, Vijit, Sudeshna, Sarkar & Basu, Anupam
2007 Investigation and modeling of the structure of texting language. International Journal on Document Analysis and Recognition 10(3): 157–174. CrossrefGoogle Scholar
De Clercq Orphée, Schulz, Sarah, Desmet, Bart, Lefever, Els, Hoste, Véronique
2013 Normalization of Dutch user-generated content. Proceedings of 9th International Conference on Recent Advances in Natural Language Processing, 179–188. Berlin: Springer.Google Scholar
Fairon, Cécrick, Klein, Jean R. & Paumier, Sébastien
2007 Le langage SMS: étude d'un corpus informatisé à partir de l'enquête ‘Faites don de vos SMS à la science’. Louvain-la-Neuve: Presses universitaires de Louvain.Google Scholar
Firth, John R.
1957 A synopsis of linguistic theory 1930–1955. Studies in Linguistic Analysis, 1–32. Oxford: Blackwell.Google Scholar
Jurafsky, Daniel & Martin, James H.
2014 Speech and Language Processing. Englewood Cliffs NJ: Prentice Hall.Google Scholar
Kobus, Catherine, Yvon, François & Damnati, Géraldine
2008 Normalizing SMS: Are two metaphors better than one? Proceedings of the 22nd International Conference on Computational Linguistics 1, 441–448.Google Scholar
Koch, Peter & Oesterreicher, Wulf
2001 Gesprochene und geschriebene Sprache. Französisch, Italienisch, Spanisch. Berlin: De Gruyter.Google Scholar
Koehn, Philipp, Hoang, Hieu, Birch, Alexandra, Callison-Burch, Chris, Federico, Marcello, Bertoldi, Brooke Cowan, Nicola, Shen, Wade, Moran, Christine, Zens, Richard, Dyer, Chris, Bojar, Ondřej, Constantin, Alexandra & Herbst, Evan
2007 Moses: Open source toolkit for statistical machine translation. Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions, 177–180. CrossrefGoogle Scholar
Li, Chen & Liu, Yang
2012 Normalization of text messages using character-& phone-based machine translation approaches. Proceedings of 13th Annual Conference of the International Speech Communication Association, 2330–2333.Google Scholar
Mikolov, Tomas, Chen, Kai, Corrado, Greg & Dean, Jeffrey
2013a Efficient estimation of word representations in vector space. The Workshop Proceedings of the International Conference on Learning Representations. https://​arxiv​.org​/abs​/1301​.3781> (13 May 2017).Google Scholar
Mikolov, Thomas, Ilya, Sutskever, Chen, Kai, Corrado, Greg & Dean, Jeffrey
2013b Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems.https://​arxiv​.org​/pdf​/1310​.4546​.pdf> (13 May 2017).Google Scholar
Och, Franz Josef & Ney, Hermann
2003 A systematic comparison of various statistical alignment models. Computational Linguistics 29(1): 19–51. CrossrefGoogle Scholar
Pennell, Deana L. & Liu, Yang
2011 A character-level machine translation approach for normalization of SMS Abbreviations. Proceedings of International Joint Conference on Natural Language Processing (IJCNLP): 974–982.Google Scholar
Rong, Xin
2014 word2vec parameter learning explained. https://​arxiv​.org​/abs​/1411​.2738> (13 May 2017).Google Scholar
sms4science project
2004 <http://​www​.sms4science​.org​/?q​=en> (13 May 2017).
Sridhar, V. K. R.
2015 Unsupervised text normalization using distributed representations of words and phrases. Proceedings of the 2015 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL): 8–16.Google Scholar
Van Compernolle, Rémi A.
2010 The (slightly more) productive use of ne in Montreal French chat. Language Sciences 32(4): 447–463. CrossrefGoogle Scholar
Yvon, François
2010 Rewriting the orthography of SMS messages. Natural Language Engineering 16(2): 133–159. CrossrefGoogle Scholar