Q-Learning Based Relay Selection Policy for Multi-Relay Cooperative Networks with Energy Harvesting

SUN Baoxi; LI Wei; WENG Kangyi; YU Wenzheng; LIANG Zhonghua

doi:null

您当前的位置：

首页 >

文章列表页 >

Q-Learning Based Relay Selection Policy for Multi-Relay Cooperative Networks with Energy Harvesting

更新时间：2025-12-02

- Q-Learning Based Relay Selection Policy for Multi-Relay Cooperative Networks with Energy Harvesting
- JOURNAL OF XI’AN JIAOTONG UNIVERSITY (2025)
- 作者机构：
  
  长安大学信息工程学院，710018，西安
- 作者简介：
- 基金信息：
- DOI：
  CLC： TN929.5
- Received：30 August 2025，
  
  Revised：2025-12-01，
  
  Accepted：02 December 2025，
- 稿件说明：
移动端阅览
SUN Baoxi, LI Wei, WENG Kangyi, et al. Q-Learning Based Relay Selection Policy for Multi-Relay Cooperative Networks with Energy Harvesting[J/OL]. JOURNAL OF XI’AN JIAOTONG UNIVERSITY, 2025.
DOI：

SUN Baoxi, LI Wei, WENG Kangyi, et al. Q-Learning Based Relay Selection Policy for Multi-Relay Cooperative Networks with Energy Harvesting[J/OL]. JOURNAL OF XI’AN JIAOTONG UNIVERSITY, 2025. DOI：

摘要

针对采用太阳能的能量收集（energy harvesting

EH）多中继协作通信网络中断性能较差的问题，基于强化学习理论提出了一种联合中继选择和功率分配策略的优化方法。首先采用基于太阳辐照度实测数据的随机EH模型用于描述能量收集动态

状况；然后建立了EH多中继协作网络的马尔可夫决策过程模型，定义动作空间、状态空间、收益函数、传输策略和价值函数；最后使用Q学习算法设计了网络传输策略的学习和优化过程，明确了单步学习过程中的动作选择、状态转移和Q值更新方法。该优化方法能够根据太阳能EH状态、无线信道衰落状态和所有EH中继的电池电量，动态选择单个中继并确定其发射功率转发信号，以优化网络信息传输中断概率。此外，进行了计算机仿真实验，分析了网络参数对中断概率性能的影响，结果表明：网络中断概率在极高信噪比下存在饱和特性；本文优化方法相对于DQN算法，在网络中断概率为10

-2

~10

-3

时可获得3~5 dB的信噪比增益；相对于Actor-Critic和SARSA算法在网络中断概率为10

~10

-3

时可分别获得2~3 dB和3~7 dB的信噪比增益。

Abstract

To tackle the issue of poor outage performance in multi-relay cooperative communication networks with solar energy harvesting (EH)

an optimization method for joint relay selection and power allocation policy is proposed based on reinforcement learning theory. First

a stochastic EH model based on measured solar irradiance data is adopted to mimic the dynamic status of solar EH. Then

the Markov decision process model of the multi-relay cooperative networks with solar EH is established

to define the action space

state space

reward function

transmission policy

and value function. Finally

the learning and optimization procedure of the transmission policy is designed by exploiting the Q-learning algorithm

which specifies the action selection

system state transfer and Q-value updating in a single-step of the learning procedure. This optimization method enables the dynamic selection of a single relay and its corresponding transmission power for signal forwarding based on the solar EH status

the wireless channel fading and the battery power of all EH relays

so as to optimize the outage probability of information transmission. Moreover

the impact of the network parameters on the outage performance is analyzed in computer simulation experiments and a saturation structure for the outage probability is revealed. In addition

simulation results demonstrate that

compared with the DQN algorithm

the proposed method can achieve 3~5 dB of signal-to-noise ratio (SNR) gains within the outage probability ranges of 10

-2

~10

-3

. Compare

d with the Actor-Critic and SARSA algorithms

the proposed method can achieve 2~3 dB and 3~7 dB of SNR gains within the outage probability ranges of 10

~10

-3

关键词

Keywords

references

李向阳 , 尚飞 , 闫宇博 , 等 . 智能物联网低功耗感知综述 [J ] . 计算机研究与发展 , 2024 , 61 ( 11 ): 2754 - 2775 .

LI Xiangyang , SHANG Fei , YAN Yubo , et al . Survey on Low Power Sensing of AIoT [J ] . Journal of Computer Research and Development , 2024 , 61 ( 11 ): 2754 - 2775 .

王新奕 , 费泽松 , 周一青 , 等 . 面向物联网的通感算智融合：关键技术与未来展望 [J ] . 电子与信息学报 , 2025 , 47 ( 4 ): 888 - 908 .

WANG Xinyi , FEI Zesong , ZHOU Yiqing , et al . Integrated Sensing, Communication, Computation, and Intelligence Towards IoT: Key Technologies and Future Directions [J ] . Journal of Electronics & Information Technology , 2025 , 47 ( 4 ): 888 - 908 .

WILLIAMS A J , TORQUATO M F , CAMERON I M , et al . Survey of Energy Harvesting Technologies for Wireless Sensor Networks [J ] . IEEE Access , 2021 , 9 : 77493 - 77510 .

郭潇瑶 , 刘超 , 包建荣 , 等 . 低复杂度放大转发多中继选择及功率分配 [J ] . 电子学报 , 2022 , 50 ( 11 ): 2602 - 2609 .

GUO Xiaoyao , LIU Chao , BAO Jianrong , et al . Multi⁃Relay Selection and Power Allocation of Low Complexity Amplify⁃and⁃Forward Cooperation [J ] . Acta Electronica Sinica , 2022 , 50 ( 11 ): 2602 - 2609 .

MA D , LAN G , HASSAN M , et al . Sensing, computing, and communications for energy harvesting IoTs: A survey [J ] . IEEE Communications Surveys and Tutorials , 2020 , 22 ( 2 ): 1222 - 1250 .

周欣彤 , 肖琨 . 综合上下行链路的无线能量收集协作网络资源分配 [J ] . 浙江大学学报(工学版) , 2023 , 57 ( 12 ): 2544 - 2552 .

ZHOU Xintong , XIAO Kun . Resource allocation algorithm for wireless energy harvesting cooperative network integrating uplink and downlink [J ] . Journal of Zhejiang U- niversity (Engineering Science) , 2023 , 57 ( 12 ): 2544 - 2552 .

MAO Bomin , TANG Fengxiao , KAWAMOTO Y , et al . AI Models for Green Communications Towards 6G [J ] . IEEE Communications Surveys and Tutorials , 2022 , 24 ( 1 ): 210 - 247 .

陈珍珍 , 冀保峰 , 陈苏丹 . 基于能量收集的中继协同无线物联传输方案 [J ] . 计算机应用与软件 , 2022 , 39 ( 11 ): 99 - 104 .

CHEN Zhenzhen , JI Baofeng , CHEN Sudan . Transmission scheme in wireless internet of things with relay cooperation based on energy harvesting [J ] . Computer Applications and Software , 2022 , 39 ( 11 ): 99 - 104 .

WU Yuan , QIAN Liping , HUANG Liang , et al . Optimal relay selection and power control for energy-harvesting wireless relay networks [J ] . IEEE Transactions on Green Communications and Networking , 2018 , 2 ( 2 ): 471 - 481 .

张士兵 , 韩刘可 , 张美娟 . 基于能量收集的全双工认知中继网络功率分配算法 [J ] . 通信学报 , 2020 , 41 ( 9 ): 139 - 146 .

ZHANG Shibing , HAN Liuke , ZHANG Meijuan . Power allocation algorithm of full duplex cognitive relay network based on energy harvesting [J ] . Journal on Communica- tions , 2020 , 41 ( 9 ): 139 - 146 .

RAO S . Relay Selection for Energy Harvesting Cooperative NOMA [C ] // 2019 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS) . Goa, India : IEEE , 2019 : 1 - 6 .

LI Wei , KU M L , CHEN Yan , et al . ZHU Shihua. Performance Analysis for Two-Way Network-Coded Dual-Relay Networks With Stochastic Energy Harvesting [J ] . IEEE Transactions on Wireless Communications , 2017 , 16 ( 9 ): 5747 - 5761 .

AHMED I , AHMED I , HOSSAIN J . Optimal stochastic power allocation and relay selection for energy harvesting systems [J ] . IEEE Wireless Communications Letters , 2017 , 6 ( 4 ): 546 - 549 .

HU Yang , QIU Chengrun , CHEN Yan . Lyapunov- Optimized Two-Way Relay Networks With Stochastic Energy Harvesting [J ] . IEEE Transactions on Wireless Communications , 2018 , 17 ( 9 ): 6280 - 6292 .

CHEN Xuehan , LIU Yong , CAI L X , et al . Resource Allocation for Wireless Cooperative IoT Network With Energy Harvesting [J ] . IEEE Transactions on Wireless Communications , 2020 , 19 ( 7 ): 4879 - 4893 .

ORTIZ A , WEBER T , KLEIN A . Multi-Agent Reinforce-ment Learning for Energy Harvesting Two-Hop Communi-cations With a Partially Observable System State [J ] . IEEE Transactions on Green Communications and Networking , 2022 , 5 ( 1 ): 442 - 456 .

ZHAO Bin , ZHAO Xiaohui . Deep Reinforcement Learning Resource Allocation in Wireless Sensor Networks With Energy Harvesting and Relay [J ] . IEEE Internet of Things Journal , 2022 , 9 ( 3 ): 2330 - 2345 .

WANG Jindong , LIU Zhibin , LIU Xiaowu . Relay Selection for Energy-Harvesting Collaborative Communication Systems: A Deep Reinforcement Learning Approach [J ] . IEEE Transactions on Industrial Informatics , 2025 , 21 ( 1 ): 771 - 780 .

WANG Tong , WU Shiting , WANG Zhenyong , et al . A Multi-Featured Actor-Critic Relay Selection Scheme for Large-Scale Energy Harvesting WSNs [J ] . IEEE Wireless Communications Letters , 2022 , 10 ( 1 ): 180 - 184 .

KU M L , CHEN Yan , LIU K J R . Data-driven stochastic transmission policies for energy harvesting sensor communications [J ] . IEEE Journal on Selected Areas in Communications , 2015 , 33 ( 8 ): 1505 - 1520 .

邵彦彰 , 王磊 , 薛奕蕾 . 采用分布式Alamouti编码的虚拟全双工中继通信方案 [J ] . 西安交通大学学报 , 2020 , 54 ( 12 ): 45 - 53 .

SHAO Yanzhang , WANG Lei , XUE Yilei . A Virtual Full-Duplex Relay Communication Scheme Based on Distributed Alamouti Protocol [J ] . Journal of Xi’an Jiaotong University , 2020 , 54 ( 12 ): 45 - 53 .

LOPEZ-SALAMANCA J J , SEMAN L O , BEREJUCK M D , et al . Finite-State Markov Chains Channel Model for CubeSats Communication Uplink [J ] . IEEE Transactions on Aerospace and Electronic Systems , 2020 , 56 ( 1 ): 142 - 154 .

温广辉 , 杨涛 , 周佳玲 , 等 . 强化学习与自适应动态规划:从基础理论到多智能体系统中的应用进展综述 [J ] . 控制与决策 , 2023 , 38 ( 5 ): 1200 - 1230 .

WEN Guanghui , YANG Tao , ZHOU Jialing , et al . Reinforcement learning and adaptive/approximate dynamic programming: A survey from theory to applications in multi-agent systems [J ] . Control and Decision , 2023 , 38 ( 5 ): 1200 - 1230 .

LI Wei , KU M L , CHEN Yan , et al . On Outage Probability for Stochastic Energy Harvesting Communications in Fading Channels [J ] . IEEE Signal Processing Letters , 2015 , 22 ( 11 ): 1893 - 1897 .

丁世飞 , 杜威 , 张健 , 等 . 多智能体深度强化学习研究进展 [J ] . 计算机学报 , 2024 , 47 ( 7 ): 1547 - 1567 .

DING Shifei , DU Wei , ZHANG Jian , et al . Research progress of multi-agent deep reinforcement learning [J ] . CHINESE JOURNAL OF COMPUTERS , 2024 , 47 ( 7 ): 1547 - 1567 .

丁长文 , 杨霖 , 李高祥 . 能量收集双向中继网络的高能效联合中继选择和功率分配算法 [J ] . 电子学报 , 2017 , 45 ( 5 ): 1124 - 1129 .

DING Changwen , YANG Lin , LI Gaoxiang . Energy-Efficient Relay Selection and Power Allocation for Energy Harvesting Two-Way Relay Network [J ] . ACTA ELECTRONICA SINICA , 2017 , 45 ( 5 ): 1124 - 1129 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Q-Learning-Based Relay Selection Policy for Multi-Relay Cooperative Networks with Energy Harvesting

Relay Selection Scheme in Cooperative Networks with Amplify-and-Forward Protocol

Relay Selection Protocol for Cooperative Wireless Networks with Interference

A Scheme of Relay Selection and Transmission Power Allocation for Wireless Energy Transfer

Reliable Delivery Scheme for Wireless Sensor Networks Based on Cooperation among Neighboring Nodes

Related Author

LIANG Zhonghua

YU Wenzheng

WENG Kangyi

LI Wei

SUN Baoxi

李国兵 1

朱世华 1

Related Institution

School of Information Engineering，Chang’an University

School of Electronics and Information Engineering, Xi'an Jiaotong University

School of Automation and Information Engineering, Xi'an University of Technology

State Key Lab of Integrated Services Networks, Xidian University

School of Electronic and Information Engineering, Xi'an Jiaotong University

⁰