Document and Word-level Language Identification for Noisy User Generated Text

Zhanibek Kozhirbayev, Zhandos Yessenbayev, Aibek Makazhanov

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

We present herein our work on language identification applied to comments left by the readers of online news sites popular in Kazakhstan. Such comments are typically written in one of the two languages spoken widely in the area (Kazakh and Russian) and sometimes-in a mixture of both. Code-switching (mixing languages) makes it desirable to identify language not only on document, but also on individual word level. We approach both tasks in a single two-step framework, performing unsupervised normalization and Nave Bayes text classification procedures successively. Moreover, we applied deep learning model based on recurrent networks with LSTM cell in order to classify text. Our results suggest improvement over the state-of-the-art for Kazakh language.

Original languageEnglish
Title of host publicationIEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781538664674
DOIs
Publication statusPublished - Oct 2018
Event12th IEEE International Conference on Application of Information and Communication Technologies, AICT 2018 - Almaty, Kazakhstan
Duration: Oct 17 2018Oct 19 2018

Publication series

NameIEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings

Conference

Conference12th IEEE International Conference on Application of Information and Communication Technologies, AICT 2018
CountryKazakhstan
CityAlmaty
Period10/17/1810/19/18

Keywords

  • code-switching
  • language identification
  • normalization
  • user generated content

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Computer Vision and Pattern Recognition
  • Information Systems and Management
  • Health Informatics
  • Information Systems

Fingerprint Dive into the research topics of 'Document and Word-level Language Identification for Noisy User Generated Text'. Together they form a unique fingerprint.

  • Cite this

    Kozhirbayev, Z., Yessenbayev, Z., & Makazhanov, A. (2018). Document and Word-level Language Identification for Noisy User Generated Text. In IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings [8747138] (IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICAICT.2018.8747138