Initial Explorations on Chaotic Behaviors of Recurrent Neural Networks

Bagdat Myrzakhmetov, Rustem Takhanov, Zhenisbek Assylbekov

Research output: Chapter in Book/Report/Conference proceedingConference contribution


In this paper we analyzed the dynamics of Recurrent Neural Network architectures. We explored the chaotic nature of state-of-the-art Recurrent Neural Networks: Vanilla Recurrent Network and Recurrent Highway Networks. Our experiments showed that they exhibit chaotic behavior in the absence of input data. We also proposed a way of removing chaos from Recurrent Neural Networks. Our findings show that initialization of the weight matrices during the training plays an important role, as initialization with the matrices whose norm is smaller than one will lead to the non-chaotic behavior of the Recurrent Neural Networks. The advantage of the non-chaotic cells is stable dynamics. At the end, we tested our chaos-free version of the Recurrent Highway Networks (RHN) in a real-world application. In the language modeling task, chaos-free versions of RHN perform on par with the original version.

Original languageEnglish
Title of host publicationComputational Linguistics and Intelligent Text Processing - 20th International Conference, CICLing 2019, Revised Selected Papers
EditorsAlexander Gelbukh
PublisherSpringer Science and Business Media Deutschland GmbH
Number of pages13
ISBN (Print)9783031243363
Publication statusPublished - 2023
Event20th International Conference on Computational Linguistics and Intelligent Text Processing, CICLing 2019 - La Rochelle, France
Duration: Apr 7 2019Apr 13 2019

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13451 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference20th International Conference on Computational Linguistics and Intelligent Text Processing, CICLing 2019
CityLa Rochelle


  • Chaos theory
  • Language modeling
  • Recurrent highway networks
  • Recurrent neural networks

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science


Dive into the research topics of 'Initial Explorations on Chaotic Behaviors of Recurrent Neural Networks'. Together they form a unique fingerprint.

Cite this