Two-Step Deep Reinforcement Q-Learning based Relay Selection in Cooperative WPCNs

Gulnur Tolebi; Theodoros A. Tsiftsis; Galymzhan Nauryzbayev

doi:10.1109/BalkanCom58402.2023.10167871

Two-Step Deep Reinforcement Q-Learning based Relay Selection in Cooperative WPCNs

Gulnur Tolebi
, Theodoros A. Tsiftsis
, Galymzhan Nauryzbayev

Nazarbayev University
Department of Informatics & Telecommunications
University of Thessaly

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Citations (Scopus)

Abstract

In this paper, we propose an intelligent relay selection scheme employing deep reinforcement learning for a wireless powered cooperative network. We formulate the given problem as a Markov decision process with an unknown transitional probability between states. Therefore, a model-free off-policy relay selection model is proposed. The given model was deployed using a deep Q-network, with an updated relay selection process. Using channel characteristics, we find inaccessible nodes to form a pool of relays available for transmission and encourage the neural network to choose them. In addition, we propose a novel reward policy to train the model that is based on stored energy levels on the relays and promotes the system to expend energy. We numerically quantity the network performance in terms of outage probability and energy outage probability and compare them with the basic Q-learning.

Original language	English
Title of host publication	2023 International Balkan Conference on Communications and Networking, BalkanCom 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9798350339109
DOIs	https://doi.org/10.1109/BalkanCom58402.2023.10167871
Publication status	Published - 2023
Event	2023 International Balkan Conference on Communications and Networking, BalkanCom 2023 - Istanbul, Turkey Duration: Jun 5 2023 → Jun 8 2023

Publication series

Name	2023 International Balkan Conference on Communications and Networking, BalkanCom 2023

Conference

Conference	2023 International Balkan Conference on Communications and Networking, BalkanCom 2023
Country/Territory	Turkey
City	Istanbul
Period	6/5/23 → 6/8/23

Funding

VI. ACKNOWLEDGMENT This research is funded by Nazarbayev University under Collaborative Research Program Grant no. 11022021CRP1513 (PI: Galymzhan Nauryzbayev).

Keywords

outage probability (OP)
Q-learning
reinforcement learning (RL)
Relay selection
wireless powered communication network (WPCN)

ASJC Scopus subject areas

Artificial Intelligence
Computer Networks and Communications
Hardware and Architecture
Information Systems
Safety, Risk, Reliability and Quality
Instrumentation

Access to Document

10.1109/BalkanCom58402.2023.10167871

Cite this

Tolebi, G., Tsiftsis, T. A., & Nauryzbayev, G. (2023). Two-Step Deep Reinforcement Q-Learning based Relay Selection in Cooperative WPCNs. In 2023 International Balkan Conference on Communications and Networking, BalkanCom 2023 (2023 International Balkan Conference on Communications and Networking, BalkanCom 2023). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BalkanCom58402.2023.10167871

Two-Step Deep Reinforcement Q-Learning based Relay Selection in Cooperative WPCNs. / Tolebi, Gulnur; Tsiftsis, Theodoros A.; Nauryzbayev, Galymzhan.
2023 International Balkan Conference on Communications and Networking, BalkanCom 2023. Institute of Electrical and Electronics Engineers Inc., 2023. (2023 International Balkan Conference on Communications and Networking, BalkanCom 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Tolebi, G, Tsiftsis, TA & Nauryzbayev, G 2023, Two-Step Deep Reinforcement Q-Learning based Relay Selection in Cooperative WPCNs. in 2023 International Balkan Conference on Communications and Networking, BalkanCom 2023. 2023 International Balkan Conference on Communications and Networking, BalkanCom 2023, Institute of Electrical and Electronics Engineers Inc., 2023 International Balkan Conference on Communications and Networking, BalkanCom 2023, Istanbul, Turkey, 6/5/23. https://doi.org/10.1109/BalkanCom58402.2023.10167871

Tolebi G, Tsiftsis TA, Nauryzbayev G. Two-Step Deep Reinforcement Q-Learning based Relay Selection in Cooperative WPCNs. In 2023 International Balkan Conference on Communications and Networking, BalkanCom 2023. Institute of Electrical and Electronics Engineers Inc. 2023. (2023 International Balkan Conference on Communications and Networking, BalkanCom 2023). doi: 10.1109/BalkanCom58402.2023.10167871

Tolebi, Gulnur ; Tsiftsis, Theodoros A. ; Nauryzbayev, Galymzhan. / Two-Step Deep Reinforcement Q-Learning based Relay Selection in Cooperative WPCNs. 2023 International Balkan Conference on Communications and Networking, BalkanCom 2023. Institute of Electrical and Electronics Engineers Inc., 2023. (2023 International Balkan Conference on Communications and Networking, BalkanCom 2023).

@inproceedings{e3f0ca0e5a9b4c649b9615a045f90aa5,

title = "Two-Step Deep Reinforcement Q-Learning based Relay Selection in Cooperative WPCNs",

abstract = "In this paper, we propose an intelligent relay selection scheme employing deep reinforcement learning for a wireless powered cooperative network. We formulate the given problem as a Markov decision process with an unknown transitional probability between states. Therefore, a model-free off-policy relay selection model is proposed. The given model was deployed using a deep Q-network, with an updated relay selection process. Using channel characteristics, we find inaccessible nodes to form a pool of relays available for transmission and encourage the neural network to choose them. In addition, we propose a novel reward policy to train the model that is based on stored energy levels on the relays and promotes the system to expend energy. We numerically quantity the network performance in terms of outage probability and energy outage probability and compare them with the basic Q-learning.",

keywords = "outage probability (OP), Q-learning, reinforcement learning (RL), Relay selection, wireless powered communication network (WPCN)",

author = "Gulnur Tolebi and Tsiftsis, \{Theodoros A.\} and Galymzhan Nauryzbayev",

note = "Funding Information: VI. ACKNOWLEDGMENT This research is funded by Nazarbayev University under Collaborative Research Program Grant no. 11022021CRP1513 (PI: Galymzhan Nauryzbayev). Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 International Balkan Conference on Communications and Networking, BalkanCom 2023 ; Conference date: 05-06-2023 Through 08-06-2023",

year = "2023",

doi = "10.1109/BalkanCom58402.2023.10167871",

language = "English",

series = "2023 International Balkan Conference on Communications and Networking, BalkanCom 2023",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2023 International Balkan Conference on Communications and Networking, BalkanCom 2023",

address = "United States",

}

TY - GEN

T1 - Two-Step Deep Reinforcement Q-Learning based Relay Selection in Cooperative WPCNs

AU - Tolebi, Gulnur

AU - Tsiftsis, Theodoros A.

AU - Nauryzbayev, Galymzhan

N1 - Funding Information: VI. ACKNOWLEDGMENT This research is funded by Nazarbayev University under Collaborative Research Program Grant no. 11022021CRP1513 (PI: Galymzhan Nauryzbayev). Publisher Copyright: © 2023 IEEE.

PY - 2023

Y1 - 2023

N2 - In this paper, we propose an intelligent relay selection scheme employing deep reinforcement learning for a wireless powered cooperative network. We formulate the given problem as a Markov decision process with an unknown transitional probability between states. Therefore, a model-free off-policy relay selection model is proposed. The given model was deployed using a deep Q-network, with an updated relay selection process. Using channel characteristics, we find inaccessible nodes to form a pool of relays available for transmission and encourage the neural network to choose them. In addition, we propose a novel reward policy to train the model that is based on stored energy levels on the relays and promotes the system to expend energy. We numerically quantity the network performance in terms of outage probability and energy outage probability and compare them with the basic Q-learning.

AB - In this paper, we propose an intelligent relay selection scheme employing deep reinforcement learning for a wireless powered cooperative network. We formulate the given problem as a Markov decision process with an unknown transitional probability between states. Therefore, a model-free off-policy relay selection model is proposed. The given model was deployed using a deep Q-network, with an updated relay selection process. Using channel characteristics, we find inaccessible nodes to form a pool of relays available for transmission and encourage the neural network to choose them. In addition, we propose a novel reward policy to train the model that is based on stored energy levels on the relays and promotes the system to expend energy. We numerically quantity the network performance in terms of outage probability and energy outage probability and compare them with the basic Q-learning.

KW - outage probability (OP)

KW - Q-learning

KW - reinforcement learning (RL)

KW - Relay selection

KW - wireless powered communication network (WPCN)

UR - https://www.scopus.com/pages/publications/85165637945

UR - https://www.scopus.com/pages/publications/85165637945#tab=citedBy

U2 - 10.1109/BalkanCom58402.2023.10167871

DO - 10.1109/BalkanCom58402.2023.10167871

M3 - Conference contribution

AN - SCOPUS:85165637945

T3 - 2023 International Balkan Conference on Communications and Networking, BalkanCom 2023

BT - 2023 International Balkan Conference on Communications and Networking, BalkanCom 2023

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2023 International Balkan Conference on Communications and Networking, BalkanCom 2023

Y2 - 5 June 2023 through 8 June 2023

ER -

Two-Step Deep Reinforcement Q-Learning based Relay Selection in Cooperative WPCNs

Abstract

Publication series

Conference

Funding

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this