Document and Word-level Language Identification for Noisy User Generated Text

Zhanibek Kozhirbayev; Zhandos Yessenbayev; Aibek Makazhanov

doi:10.1109/ICAICT.2018.8747138

Document and Word-level Language Identification for Noisy User Generated Text

Zhanibek Kozhirbayev, Zhandos Yessenbayev, Aibek Makazhanov

Laboratory of computational materials science for energy application

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

4 Citations (Scopus)

Abstract

We present herein our work on language identification applied to comments left by the readers of online news sites popular in Kazakhstan. Such comments are typically written in one of the two languages spoken widely in the area (Kazakh and Russian) and sometimes-in a mixture of both. Code-switching (mixing languages) makes it desirable to identify language not only on document, but also on individual word level. We approach both tasks in a single two-step framework, performing unsupervised normalization and Nave Bayes text classification procedures successively. Moreover, we applied deep learning model based on recurrent networks with LSTM cell in order to classify text. Our results suggest improvement over the state-of-the-art for Kazakh language.

Original language	English
Title of host publication	IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781538664674
DOIs	https://doi.org/10.1109/ICAICT.2018.8747138
Publication status	Published - Oct 2018
Event	12th IEEE International Conference on Application of Information and Communication Technologies, AICT 2018 - Almaty, Kazakhstan Duration: Oct 17 2018 → Oct 19 2018

Publication series

Name	IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings

Conference

Conference	12th IEEE International Conference on Application of Information and Communication Technologies, AICT 2018
Country	Kazakhstan
City	Almaty
Period	10/17/18 → 10/19/18

Keywords

code-switching
language identification
normalization
user generated content

ASJC Scopus subject areas

Computer Networks and Communications
Computer Science Applications
Computer Vision and Pattern Recognition
Information Systems and Management
Health Informatics
Information Systems

Access to Document

10.1109/ICAICT.2018.8747138

Fingerprint Dive into the research topics of 'Document and Word-level Language Identification for Noisy User Generated Text'. Together they form a unique fingerprint.

Cite this

Kozhirbayev, Z., Yessenbayev, Z., & Makazhanov, A. (2018). Document and Word-level Language Identification for Noisy User Generated Text. In IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings [8747138] (IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICAICT.2018.8747138

Document and Word-level Language Identification for Noisy User Generated Text. / Kozhirbayev, Zhanibek ; Yessenbayev, Zhandos ; Makazhanov, Aibek.

IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2018. 8747138 (IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Kozhirbayev, Z , Yessenbayev, Z & Makazhanov, A 2018, Document and Word-level Language Identification for Noisy User Generated Text. in IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings., 8747138, IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings, Institute of Electrical and Electronics Engineers Inc., 12th IEEE International Conference on Application of Information and Communication Technologies, AICT 2018, Almaty, Kazakhstan, 10/17/18. https://doi.org/10.1109/ICAICT.2018.8747138

Kozhirbayev Z , Yessenbayev Z , Makazhanov A. Document and Word-level Language Identification for Noisy User Generated Text. In IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2018. 8747138. (IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings). https://doi.org/10.1109/ICAICT.2018.8747138

Kozhirbayev, Zhanibek ; Yessenbayev, Zhandos ; Makazhanov, Aibek. / Document and Word-level Language Identification for Noisy User Generated Text. IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2018. (IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings).

@inproceedings{c41cf13c55804fe58252a1cbc22c1283,

title = "Document and Word-level Language Identification for Noisy User Generated Text",

abstract = "We present herein our work on language identification applied to comments left by the readers of online news sites popular in Kazakhstan. Such comments are typically written in one of the two languages spoken widely in the area (Kazakh and Russian) and sometimes-in a mixture of both. Code-switching (mixing languages) makes it desirable to identify language not only on document, but also on individual word level. We approach both tasks in a single two-step framework, performing unsupervised normalization and Nave Bayes text classification procedures successively. Moreover, we applied deep learning model based on recurrent networks with LSTM cell in order to classify text. Our results suggest improvement over the state-of-the-art for Kazakh language.",

keywords = "code-switching, language identification, normalization, user generated content",

author = "Zhanibek Kozhirbayev and Zhandos Yessenbayev and Aibek Makazhanov",

note = "Funding Information: ACKNOWLEDGMENT This work has been supported by Nazarbayev University research grant 129-2017/022-2017 and the Committee of Science of the Ministry of Education and Science of the Republic of Kazakhstan under the research grant AP05134272.; 12th IEEE International Conference on Application of Information and Communication Technologies, AICT 2018 ; Conference date: 17-10-2018 Through 19-10-2018",

year = "2018",

month = oct,

doi = "10.1109/ICAICT.2018.8747138",

language = "English",

series = "IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings",

address = "United States",

}

TY - GEN

T1 - Document and Word-level Language Identification for Noisy User Generated Text

AU - Kozhirbayev, Zhanibek

AU - Yessenbayev, Zhandos

AU - Makazhanov, Aibek

N1 - Funding Information: ACKNOWLEDGMENT This work has been supported by Nazarbayev University research grant 129-2017/022-2017 and the Committee of Science of the Ministry of Education and Science of the Republic of Kazakhstan under the research grant AP05134272.

PY - 2018/10

Y1 - 2018/10

N2 - We present herein our work on language identification applied to comments left by the readers of online news sites popular in Kazakhstan. Such comments are typically written in one of the two languages spoken widely in the area (Kazakh and Russian) and sometimes-in a mixture of both. Code-switching (mixing languages) makes it desirable to identify language not only on document, but also on individual word level. We approach both tasks in a single two-step framework, performing unsupervised normalization and Nave Bayes text classification procedures successively. Moreover, we applied deep learning model based on recurrent networks with LSTM cell in order to classify text. Our results suggest improvement over the state-of-the-art for Kazakh language.

AB - We present herein our work on language identification applied to comments left by the readers of online news sites popular in Kazakhstan. Such comments are typically written in one of the two languages spoken widely in the area (Kazakh and Russian) and sometimes-in a mixture of both. Code-switching (mixing languages) makes it desirable to identify language not only on document, but also on individual word level. We approach both tasks in a single two-step framework, performing unsupervised normalization and Nave Bayes text classification procedures successively. Moreover, we applied deep learning model based on recurrent networks with LSTM cell in order to classify text. Our results suggest improvement over the state-of-the-art for Kazakh language.

KW - code-switching

KW - language identification

KW - normalization

KW - user generated content

UR - http://www.scopus.com/inward/record.url?scp=85070232678&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85070232678&partnerID=8YFLogxK

U2 - 10.1109/ICAICT.2018.8747138

DO - 10.1109/ICAICT.2018.8747138

M3 - Conference contribution

AN - SCOPUS:85070232678

T3 - IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings

BT - IEEE 12th International Conference on Application of Information and Communication Technologies, AICT 2018 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 12th IEEE International Conference on Application of Information and Communication Technologies, AICT 2018

Y2 - 17 October 2018 through 19 October 2018

ER -