Initial Explorations on Chaotic Behaviors of Recurrent Neural Networks

Bagdat Myrzakhmetov; Rustem Takhanov; Zhenisbek Assylbekov

doi:10.1007/978-3-031-24337-0_26

Initial Explorations on Chaotic Behaviors of Recurrent Neural Networks

Bagdat Myrzakhmetov, Rustem Takhanov, Zhenisbek Assylbekov

Nazarbayev University

Nazarbayev University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In this paper we analyzed the dynamics of Recurrent Neural Network architectures. We explored the chaotic nature of state-of-the-art Recurrent Neural Networks: Vanilla Recurrent Network and Recurrent Highway Networks. Our experiments showed that they exhibit chaotic behavior in the absence of input data. We also proposed a way of removing chaos from Recurrent Neural Networks. Our findings show that initialization of the weight matrices during the training plays an important role, as initialization with the matrices whose norm is smaller than one will lead to the non-chaotic behavior of the Recurrent Neural Networks. The advantage of the non-chaotic cells is stable dynamics. At the end, we tested our chaos-free version of the Recurrent Highway Networks (RHN) in a real-world application. In the language modeling task, chaos-free versions of RHN perform on par with the original version.

Original language	English
Title of host publication	Computational Linguistics and Intelligent Text Processing - 20th International Conference, CICLing 2019, Revised Selected Papers
Editors	Alexander Gelbukh
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	351-363
Number of pages	13
ISBN (Print)	9783031243363
DOIs	https://doi.org/10.1007/978-3-031-24337-0_26
Publication status	Published - 2023
Event	20th International Conference on Computational Linguistics and Intelligent Text Processing, CICLing 2019 - La Rochelle, France Duration: Apr 7 2019 → Apr 13 2019

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13451 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	20th International Conference on Computational Linguistics and Intelligent Text Processing, CICLing 2019
Country/Territory	France
City	La Rochelle
Period	4/7/19 → 4/13/19

Funding

Acknowledgement. This work has been funded by the Committee of Science of the Ministry of Education and Science of the Republic of Kazakhstan, IRN AP05133700. The work of Bagdat Myrzakhmetov partially has been funded by the Committee of Science of the Ministry of Education and Science of the Republic of Kazakhstan under the research grant AP05134272. The authors would like to thank Professor Anastasios Bountis for his valuable feedback.

Keywords

Chaos theory
Language modeling
Recurrent highway networks
Recurrent neural networks

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-031-24337-0_26

Cite this

Myrzakhmetov, B., Takhanov, R., & Assylbekov, Z. (2023). Initial Explorations on Chaotic Behaviors of Recurrent Neural Networks. In A. Gelbukh (Ed.), Computational Linguistics and Intelligent Text Processing - 20th International Conference, CICLing 2019, Revised Selected Papers (pp. 351-363). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13451 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-24337-0_26

Initial Explorations on Chaotic Behaviors of Recurrent Neural Networks. / Myrzakhmetov, Bagdat; Takhanov, Rustem; Assylbekov, Zhenisbek.
Computational Linguistics and Intelligent Text Processing - 20th International Conference, CICLing 2019, Revised Selected Papers. ed. / Alexander Gelbukh. Springer Science and Business Media Deutschland GmbH, 2023. p. 351-363 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13451 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Myrzakhmetov, B, Takhanov, R & Assylbekov, Z 2023, Initial Explorations on Chaotic Behaviors of Recurrent Neural Networks. in A Gelbukh (ed.), Computational Linguistics and Intelligent Text Processing - 20th International Conference, CICLing 2019, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13451 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 351-363, 20th International Conference on Computational Linguistics and Intelligent Text Processing, CICLing 2019, La Rochelle, France, 4/7/19. https://doi.org/10.1007/978-3-031-24337-0_26

Myrzakhmetov B, Takhanov R, Assylbekov Z. Initial Explorations on Chaotic Behaviors of Recurrent Neural Networks. In Gelbukh A, editor, Computational Linguistics and Intelligent Text Processing - 20th International Conference, CICLing 2019, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2023. p. 351-363. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-24337-0_26

Myrzakhmetov, Bagdat ; Takhanov, Rustem ; Assylbekov, Zhenisbek. / Initial Explorations on Chaotic Behaviors of Recurrent Neural Networks. Computational Linguistics and Intelligent Text Processing - 20th International Conference, CICLing 2019, Revised Selected Papers. editor / Alexander Gelbukh. Springer Science and Business Media Deutschland GmbH, 2023. pp. 351-363 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{8bb693d458ec468c8479c158a980141a,

title = "Initial Explorations on Chaotic Behaviors of Recurrent Neural Networks",

abstract = "In this paper we analyzed the dynamics of Recurrent Neural Network architectures. We explored the chaotic nature of state-of-the-art Recurrent Neural Networks: Vanilla Recurrent Network and Recurrent Highway Networks. Our experiments showed that they exhibit chaotic behavior in the absence of input data. We also proposed a way of removing chaos from Recurrent Neural Networks. Our findings show that initialization of the weight matrices during the training plays an important role, as initialization with the matrices whose norm is smaller than one will lead to the non-chaotic behavior of the Recurrent Neural Networks. The advantage of the non-chaotic cells is stable dynamics. At the end, we tested our chaos-free version of the Recurrent Highway Networks (RHN) in a real-world application. In the language modeling task, chaos-free versions of RHN perform on par with the original version.",

keywords = "Chaos theory, Language modeling, Recurrent highway networks, Recurrent neural networks",

author = "Bagdat Myrzakhmetov and Rustem Takhanov and Zhenisbek Assylbekov",

note = "Funding Information: Acknowledgement. This work has been funded by the Committee of Science of the Ministry of Education and Science of the Republic of Kazakhstan, IRN AP05133700. The work of Bagdat Myrzakhmetov partially has been funded by the Committee of Science of the Ministry of Education and Science of the Republic of Kazakhstan under the research grant AP05134272. The authors would like to thank Professor Anastasios Bountis for his valuable feedback. Publisher Copyright: {\textcopyright} 2023, Springer Nature Switzerland AG.; 20th International Conference on Computational Linguistics and Intelligent Text Processing, CICLing 2019 ; Conference date: 07-04-2019 Through 13-04-2019",

year = "2023",

doi = "10.1007/978-3-031-24337-0\_26",

language = "English",

isbn = "9783031243363",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "351--363",

editor = "Alexander Gelbukh",

booktitle = "Computational Linguistics and Intelligent Text Processing - 20th International Conference, CICLing 2019, Revised Selected Papers",

address = "Germany",

}

TY - GEN

T1 - Initial Explorations on Chaotic Behaviors of Recurrent Neural Networks

AU - Myrzakhmetov, Bagdat

AU - Takhanov, Rustem

AU - Assylbekov, Zhenisbek

N1 - Funding Information: Acknowledgement. This work has been funded by the Committee of Science of the Ministry of Education and Science of the Republic of Kazakhstan, IRN AP05133700. The work of Bagdat Myrzakhmetov partially has been funded by the Committee of Science of the Ministry of Education and Science of the Republic of Kazakhstan under the research grant AP05134272. The authors would like to thank Professor Anastasios Bountis for his valuable feedback. Publisher Copyright: © 2023, Springer Nature Switzerland AG.

PY - 2023

Y1 - 2023

N2 - In this paper we analyzed the dynamics of Recurrent Neural Network architectures. We explored the chaotic nature of state-of-the-art Recurrent Neural Networks: Vanilla Recurrent Network and Recurrent Highway Networks. Our experiments showed that they exhibit chaotic behavior in the absence of input data. We also proposed a way of removing chaos from Recurrent Neural Networks. Our findings show that initialization of the weight matrices during the training plays an important role, as initialization with the matrices whose norm is smaller than one will lead to the non-chaotic behavior of the Recurrent Neural Networks. The advantage of the non-chaotic cells is stable dynamics. At the end, we tested our chaos-free version of the Recurrent Highway Networks (RHN) in a real-world application. In the language modeling task, chaos-free versions of RHN perform on par with the original version.

AB - In this paper we analyzed the dynamics of Recurrent Neural Network architectures. We explored the chaotic nature of state-of-the-art Recurrent Neural Networks: Vanilla Recurrent Network and Recurrent Highway Networks. Our experiments showed that they exhibit chaotic behavior in the absence of input data. We also proposed a way of removing chaos from Recurrent Neural Networks. Our findings show that initialization of the weight matrices during the training plays an important role, as initialization with the matrices whose norm is smaller than one will lead to the non-chaotic behavior of the Recurrent Neural Networks. The advantage of the non-chaotic cells is stable dynamics. At the end, we tested our chaos-free version of the Recurrent Highway Networks (RHN) in a real-world application. In the language modeling task, chaos-free versions of RHN perform on par with the original version.

KW - Chaos theory

KW - Language modeling

KW - Recurrent highway networks

KW - Recurrent neural networks

UR - http://www.scopus.com/inward/record.url?scp=85149991021&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85149991021&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-24337-0_26

DO - 10.1007/978-3-031-24337-0_26

M3 - Conference contribution

AN - SCOPUS:85149991021

SN - 9783031243363

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 351

EP - 363

BT - Computational Linguistics and Intelligent Text Processing - 20th International Conference, CICLing 2019, Revised Selected Papers

A2 - Gelbukh, Alexander

PB - Springer Science and Business Media Deutschland GmbH

T2 - 20th International Conference on Computational Linguistics and Intelligent Text Processing, CICLing 2019

Y2 - 7 April 2019 through 13 April 2019

ER -