Abstract
We present herein our work on language identification applied to comments left by the readers of online news sites popular in Kazakhstan. Such comments are typically written in one of the two languages spoken widely in the area (Kazakh and Russian) and sometimes-in a mixture of both. Code-switching (mixing languages) makes it desirable to identify language not only on document, but also on individual word level. We approach both tasks in a single two-step framework, performing unsupervised normalization and Nave Bayes text classification procedures successively. Moreover, we applied deep learning model based on recurrent networks with LSTM cell in order to classify text. Our results suggest improvement over the state-of-the-art for Kazakh language.
| Original language | English |
|---|---|
| Title of host publication | IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings |
| Publisher | Institute of Electrical and Electronics Engineers Inc. |
| ISBN (Electronic) | 9781538664674 |
| DOIs | |
| Publication status | Published - Oct 1 2018 |
| Event | 12th IEEE International Conference on Application of Information and Communication Technologies, AICT 2018 - Almaty, Kazakhstan Duration: Oct 17 2018 → Oct 19 2018 |
Publication series
| Name | IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings |
|---|
Conference
| Conference | 12th IEEE International Conference on Application of Information and Communication Technologies, AICT 2018 |
|---|---|
| Country | Kazakhstan |
| City | Almaty |
| Period | 10/17/18 → 10/19/18 |
Fingerprint
Keywords
- code-switching
- language identification
- normalization
- user generated content
ASJC Scopus subject areas
- Computer Networks and Communications
- Computer Science Applications
- Computer Vision and Pattern Recognition
- Information Systems and Management
- Health Informatics
- Information Systems
Cite this
Document and Word-level Language Identification for Noisy User Generated Text. / Kozhirbayev, Zhanibek; Yessenbayev, Zhandos; Makazhanov, Aibek.
IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2018. 8747138 (IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings).Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
}
TY - GEN
T1 - Document and Word-level Language Identification for Noisy User Generated Text
AU - Kozhirbayev, Zhanibek
AU - Yessenbayev, Zhandos
AU - Makazhanov, Aibek
PY - 2018/10/1
Y1 - 2018/10/1
N2 - We present herein our work on language identification applied to comments left by the readers of online news sites popular in Kazakhstan. Such comments are typically written in one of the two languages spoken widely in the area (Kazakh and Russian) and sometimes-in a mixture of both. Code-switching (mixing languages) makes it desirable to identify language not only on document, but also on individual word level. We approach both tasks in a single two-step framework, performing unsupervised normalization and Nave Bayes text classification procedures successively. Moreover, we applied deep learning model based on recurrent networks with LSTM cell in order to classify text. Our results suggest improvement over the state-of-the-art for Kazakh language.
AB - We present herein our work on language identification applied to comments left by the readers of online news sites popular in Kazakhstan. Such comments are typically written in one of the two languages spoken widely in the area (Kazakh and Russian) and sometimes-in a mixture of both. Code-switching (mixing languages) makes it desirable to identify language not only on document, but also on individual word level. We approach both tasks in a single two-step framework, performing unsupervised normalization and Nave Bayes text classification procedures successively. Moreover, we applied deep learning model based on recurrent networks with LSTM cell in order to classify text. Our results suggest improvement over the state-of-the-art for Kazakh language.
KW - code-switching
KW - language identification
KW - normalization
KW - user generated content
UR - http://www.scopus.com/inward/record.url?scp=85070232678&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85070232678&partnerID=8YFLogxK
U2 - 10.1109/ICAICT.2018.8747138
DO - 10.1109/ICAICT.2018.8747138
M3 - Conference contribution
AN - SCOPUS:85070232678
T3 - IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings
BT - IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings
PB - Institute of Electrical and Electronics Engineers Inc.
ER -