Cyrillic manual alphabet recognition in RGB and RGB-D data for sign language interpreting robotic system (SLIRS)

Nazgul Tazhigaliyeva; Nazerke Kalidolda; Alfarabi Imashev; Shynggys Islam; Kairat Aitpayev; German I. Parisi; Anara Sandygulova

doi:10.1109/ICRA.2017.7989526

Cyrillic manual alphabet recognition in RGB and RGB-D data for sign language interpreting robotic system (SLIRS)

Nazgul Tazhigaliyeva
, Nazerke Kalidolda
, Alfarabi Imashev
, Shynggys Islam
, Kairat Aitpayev
, German I. Parisi
, Anara Sandygulova

Department of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

8 Citations (Scopus)

Abstract

Deaf-mute communities around the world experience a need in effective human-robot interaction system that would act as an interpreter in public places such as banks, hospitals, or police stations. The focus of this work is to address the challenges presented to hearing-impaired people by developing an interpreting robotic system required for effective communication in public places. To this end, we utilize a previously developed neural network-based learning architecture to recognize Cyrillic manual alphabet, which is used for fingerspelling in Kazakhstan. In order to train and test the performance of the recognition system, we collected four datasets comprising of static and motion RGB and RGB-D data of 33 manual gestures. After applying them to standard machine learning algorithms as well as to our previously developed learning-based method, we achieved an average accuracy of 93% for a complete alphabet recognition by modeling motion depth data.

Original language	English
Title of host publication	ICRA 2017 - IEEE International Conference on Robotics and Automation
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	4531-4536
Number of pages	6
ISBN (Electronic)	9781509046331
DOIs	https://doi.org/10.1109/ICRA.2017.7989526
Publication status	Published - Jul 21 2017
Event	2017 IEEE International Conference on Robotics and Automation, ICRA 2017 - Singapore, Singapore Duration: May 29 2017 → Jun 3 2017

Publication series

Name	Proceedings - IEEE International Conference on Robotics and Automation
ISSN (Print)	1050-4729

Conference

Conference	2017 IEEE International Conference on Robotics and Automation, ICRA 2017
Country/Territory	Singapore
City	Singapore
Period	5/29/17 → 6/3/17

Funding

The authors would like to thank the stuff of the community fund “Yunsz Alem” (Silent World) for the provided help with the project. This work was funded by the School of Science and Technology, Nazarbayev University, Kazakhstan.

ASJC Scopus subject areas

Software
Control and Systems Engineering
Artificial Intelligence
Electrical and Electronic Engineering

Access to Document

10.1109/ICRA.2017.7989526

Cite this

Tazhigaliyeva, N., Kalidolda, N., Imashev, A., Islam, S., Aitpayev, K., Parisi, G. I., & Sandygulova, A. (2017). Cyrillic manual alphabet recognition in RGB and RGB-D data for sign language interpreting robotic system (SLIRS). In ICRA 2017 - IEEE International Conference on Robotics and Automation (pp. 4531-4536). Article 7989526 (Proceedings - IEEE International Conference on Robotics and Automation). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICRA.2017.7989526

Cyrillic manual alphabet recognition in RGB and RGB-D data for sign language interpreting robotic system (SLIRS). / Tazhigaliyeva, Nazgul; Kalidolda, Nazerke; Imashev, Alfarabi et al.
ICRA 2017 - IEEE International Conference on Robotics and Automation. Institute of Electrical and Electronics Engineers Inc., 2017. p. 4531-4536 7989526 (Proceedings - IEEE International Conference on Robotics and Automation).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Tazhigaliyeva, N, Kalidolda, N, Imashev, A, Islam, S, Aitpayev, K, Parisi, GI & Sandygulova, A 2017, Cyrillic manual alphabet recognition in RGB and RGB-D data for sign language interpreting robotic system (SLIRS). in ICRA 2017 - IEEE International Conference on Robotics and Automation., 7989526, Proceedings - IEEE International Conference on Robotics and Automation, Institute of Electrical and Electronics Engineers Inc., pp. 4531-4536, 2017 IEEE International Conference on Robotics and Automation, ICRA 2017, Singapore, Singapore, 5/29/17. https://doi.org/10.1109/ICRA.2017.7989526

Tazhigaliyeva N, Kalidolda N, Imashev A, Islam S, Aitpayev K, Parisi GI et al. Cyrillic manual alphabet recognition in RGB and RGB-D data for sign language interpreting robotic system (SLIRS). In ICRA 2017 - IEEE International Conference on Robotics and Automation. Institute of Electrical and Electronics Engineers Inc. 2017. p. 4531-4536. 7989526. (Proceedings - IEEE International Conference on Robotics and Automation). doi: 10.1109/ICRA.2017.7989526

Tazhigaliyeva, Nazgul ; Kalidolda, Nazerke ; Imashev, Alfarabi et al. / Cyrillic manual alphabet recognition in RGB and RGB-D data for sign language interpreting robotic system (SLIRS). ICRA 2017 - IEEE International Conference on Robotics and Automation. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 4531-4536 (Proceedings - IEEE International Conference on Robotics and Automation).

@inproceedings{792e1cf2468c42208eb21bdcc7f85ef5,

title = "Cyrillic manual alphabet recognition in RGB and RGB-D data for sign language interpreting robotic system (SLIRS)",

abstract = "Deaf-mute communities around the world experience a need in effective human-robot interaction system that would act as an interpreter in public places such as banks, hospitals, or police stations. The focus of this work is to address the challenges presented to hearing-impaired people by developing an interpreting robotic system required for effective communication in public places. To this end, we utilize a previously developed neural network-based learning architecture to recognize Cyrillic manual alphabet, which is used for fingerspelling in Kazakhstan. In order to train and test the performance of the recognition system, we collected four datasets comprising of static and motion RGB and RGB-D data of 33 manual gestures. After applying them to standard machine learning algorithms as well as to our previously developed learning-based method, we achieved an average accuracy of 93\% for a complete alphabet recognition by modeling motion depth data.",

author = "Nazgul Tazhigaliyeva and Nazerke Kalidolda and Alfarabi Imashev and Shynggys Islam and Kairat Aitpayev and Parisi, \{German I.\} and Anara Sandygulova",

note = "Funding Information: The authors would like to thank the stuff of the community fund “Yunsz Alem” (Silent World) for the provided help with the project. This work was funded by the School of Science and Technology, Nazarbayev University, Kazakhstan. Publisher Copyright: {\textcopyright} 2017 IEEE.; 2017 IEEE International Conference on Robotics and Automation, ICRA 2017 ; Conference date: 29-05-2017 Through 03-06-2017",

year = "2017",

month = jul,

day = "21",

doi = "10.1109/ICRA.2017.7989526",

language = "English",

series = "Proceedings - IEEE International Conference on Robotics and Automation",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "4531--4536",

booktitle = "ICRA 2017 - IEEE International Conference on Robotics and Automation",

address = "United States",

}

TY - GEN

T1 - Cyrillic manual alphabet recognition in RGB and RGB-D data for sign language interpreting robotic system (SLIRS)

AU - Tazhigaliyeva, Nazgul

AU - Kalidolda, Nazerke

AU - Imashev, Alfarabi

AU - Islam, Shynggys

AU - Aitpayev, Kairat

AU - Parisi, German I.

AU - Sandygulova, Anara

N1 - Funding Information: The authors would like to thank the stuff of the community fund “Yunsz Alem” (Silent World) for the provided help with the project. This work was funded by the School of Science and Technology, Nazarbayev University, Kazakhstan. Publisher Copyright: © 2017 IEEE.

PY - 2017/7/21

Y1 - 2017/7/21

N2 - Deaf-mute communities around the world experience a need in effective human-robot interaction system that would act as an interpreter in public places such as banks, hospitals, or police stations. The focus of this work is to address the challenges presented to hearing-impaired people by developing an interpreting robotic system required for effective communication in public places. To this end, we utilize a previously developed neural network-based learning architecture to recognize Cyrillic manual alphabet, which is used for fingerspelling in Kazakhstan. In order to train and test the performance of the recognition system, we collected four datasets comprising of static and motion RGB and RGB-D data of 33 manual gestures. After applying them to standard machine learning algorithms as well as to our previously developed learning-based method, we achieved an average accuracy of 93% for a complete alphabet recognition by modeling motion depth data.

AB - Deaf-mute communities around the world experience a need in effective human-robot interaction system that would act as an interpreter in public places such as banks, hospitals, or police stations. The focus of this work is to address the challenges presented to hearing-impaired people by developing an interpreting robotic system required for effective communication in public places. To this end, we utilize a previously developed neural network-based learning architecture to recognize Cyrillic manual alphabet, which is used for fingerspelling in Kazakhstan. In order to train and test the performance of the recognition system, we collected four datasets comprising of static and motion RGB and RGB-D data of 33 manual gestures. After applying them to standard machine learning algorithms as well as to our previously developed learning-based method, we achieved an average accuracy of 93% for a complete alphabet recognition by modeling motion depth data.

UR - https://www.scopus.com/pages/publications/85027966201

UR - https://www.scopus.com/pages/publications/85027966201#tab=citedBy

U2 - 10.1109/ICRA.2017.7989526

DO - 10.1109/ICRA.2017.7989526

M3 - Conference contribution

AN - SCOPUS:85027966201

T3 - Proceedings - IEEE International Conference on Robotics and Automation

SP - 4531

EP - 4536

BT - ICRA 2017 - IEEE International Conference on Robotics and Automation

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2017 IEEE International Conference on Robotics and Automation, ICRA 2017

Y2 - 29 May 2017 through 3 June 2017

ER -

Cyrillic manual alphabet recognition in RGB and RGB-D data for sign language interpreting robotic system (SLIRS)

Abstract

Publication series

Conference

Funding

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this