TY - GEN
T1 - Initial Explorations on Chaotic Behaviors of Recurrent Neural Networks
AU - Myrzakhmetov, Bagdat
AU - Takhanov, Rustem
AU - Assylbekov, Zhenisbek
N1 - Funding Information:
Acknowledgement. This work has been funded by the Committee of Science of the Ministry of Education and Science of the Republic of Kazakhstan, IRN AP05133700. The work of Bagdat Myrzakhmetov partially has been funded by the Committee of Science of the Ministry of Education and Science of the Republic of Kazakhstan under the research grant AP05134272. The authors would like to thank Professor Anastasios Bountis for his valuable feedback.
Publisher Copyright:
© 2023, Springer Nature Switzerland AG.
PY - 2023
Y1 - 2023
N2 - In this paper we analyzed the dynamics of Recurrent Neural Network architectures. We explored the chaotic nature of state-of-the-art Recurrent Neural Networks: Vanilla Recurrent Network and Recurrent Highway Networks. Our experiments showed that they exhibit chaotic behavior in the absence of input data. We also proposed a way of removing chaos from Recurrent Neural Networks. Our findings show that initialization of the weight matrices during the training plays an important role, as initialization with the matrices whose norm is smaller than one will lead to the non-chaotic behavior of the Recurrent Neural Networks. The advantage of the non-chaotic cells is stable dynamics. At the end, we tested our chaos-free version of the Recurrent Highway Networks (RHN) in a real-world application. In the language modeling task, chaos-free versions of RHN perform on par with the original version.
AB - In this paper we analyzed the dynamics of Recurrent Neural Network architectures. We explored the chaotic nature of state-of-the-art Recurrent Neural Networks: Vanilla Recurrent Network and Recurrent Highway Networks. Our experiments showed that they exhibit chaotic behavior in the absence of input data. We also proposed a way of removing chaos from Recurrent Neural Networks. Our findings show that initialization of the weight matrices during the training plays an important role, as initialization with the matrices whose norm is smaller than one will lead to the non-chaotic behavior of the Recurrent Neural Networks. The advantage of the non-chaotic cells is stable dynamics. At the end, we tested our chaos-free version of the Recurrent Highway Networks (RHN) in a real-world application. In the language modeling task, chaos-free versions of RHN perform on par with the original version.
KW - Chaos theory
KW - Language modeling
KW - Recurrent highway networks
KW - Recurrent neural networks
UR - http://www.scopus.com/inward/record.url?scp=85149991021&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85149991021&partnerID=8YFLogxK
U2 - 10.1007/978-3-031-24337-0_26
DO - 10.1007/978-3-031-24337-0_26
M3 - Conference contribution
AN - SCOPUS:85149991021
SN - 9783031243363
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 351
EP - 363
BT - Computational Linguistics and Intelligent Text Processing - 20th International Conference, CICLing 2019, Revised Selected Papers
A2 - Gelbukh, Alexander
PB - Springer Science and Business Media Deutschland GmbH
T2 - 20th International Conference on Computational Linguistics and Intelligent Text Processing, CICLing 2019
Y2 - 7 April 2019 through 13 April 2019
ER -