面向虚拟数据空间的智能TCP拥塞控制算法

王龙翔; 董凯; 李小轩; 董小社; 张兴军; 朱正东; 王宇菲; 张利平

doi:10.7652/xjtuxb202105010

您当前的位置：

首页 >

文章列表页 >

面向虚拟数据空间的智能TCP拥塞控制算法

更新时间：2025-07-09

- 面向虚拟数据空间的智能TCP拥塞控制算法
- Intelligent TCP Congestion Control Method for Virtual Data Space
- 西安交通大学学报 2021年55卷第5期页码：83-91
- 作者机构：
  
  西安交通大学计算机科学与技术学院,西安,710049
- 作者简介：
- 基金信息：
  
  : 2020-09-29。作者简介: 王龙翔(1988—),男,工程师;董小社(通信作者),男,教授,博士生导师。基金项目: 国家重点研发计划资助项目(2018YFB0203902)
- DOI：10.7652/xjtuxb202105010
  中图分类号： TP319
- 网络首发：2021-05-10，
  
  纸质出版：2021
- 稿件说明：
移动端阅览
王龙翔, 董凯, 李小轩, 等. 面向虚拟数据空间的智能TCP拥塞控制算法[J]. 西安交通大学学报, 2021,55(5):83-91.

Intelligent TCP Congestion Control Method for Virtual Data Space[J]. 2021, 55(5): 83-91.
王龙翔, 董凯, 李小轩, 等. 面向虚拟数据空间的智能TCP拥塞控制算法[J]. 西安交通大学学报, 2021,55(5):83-91. DOI： 10.7652/xjtuxb202105010.

Intelligent TCP Congestion Control Method for Virtual Data Space[J]. 2021, 55(5): 83-91. DOI： 10.7652/xjtuxb202105010.

摘要

为优化虚拟数据空间网络传输性能

提出了基于近端策略优化的智能TCP拥塞控制算法TCP-PPO2。将TCP拥塞控制过程抽象为一个可部分观察的马尔可夫决策过程

在该过程中构建一个智能体

与网络环境进行互动。智能体通过观察网络状态特征对拥塞窗口长度进行调节

网络环境向智能体反馈奖励值

智能体尝试最大化回合内获得奖励期望值。设计了包括吞吐率、网络时延等网络特征的状态空间

使智能体能够观察到足够多的信息进行决策并且降低性能开销。通过加权算法设计奖励函数

使智能体能够平衡优化吞吐率与时延。通过近端策略优化算法更新智能体模型参数

对过大的参数更新进行截断

将参数更新限制在一定范围内

减少梯度下降过程中出现的振荡

实现训练过程的快速收敛。在NS3模拟器上实现了基于近端策略优化的TCP拥塞控制算法

并与Cubic、HighSpeed和NewReno等主流拥塞控制算法进行了对比

结果表明:TCP-PPO2吞吐率性能可达对比算法的2～3倍以上; 80%的采样点时延相比链路最小时延值只增加了4%。

Abstract

To optimize the transmission performance of virtual data space network

an intelligent TCP congestion control algorithm based on proximal policy optimization is proposed(TCP-PPO2). The TCP congestion control process is abstracted as a Markov decision process

which can be partially observed. In this process

an agent is constructed to interact with the network environment. The agent adjusts the size of congestion window by observing the characteristics of network state. The network environment feeds back a reward value to the agent

and the agent tries to maximize the expected reward value in an episode. The state space including throughput

network delay and other network characteristics is designed

so that agents can observe enough information to make decisions and reduce performance overhead. The weighted reward function is designed to balance the throughput and delay. The parameters of the agent model are updated by the proximal policy optimization algorithm

and excessive parameter updates are truncated. The parameter update is limited to a certain range

which reduces the oscillation problem in the process of gradient descent

and realizes quick convergence of the training process. The TCP congestion control algorithm based on proximal policy optimization is implemented on NS3 simulator

and compared with the mainstream congestion control algorithms such as cubic

HighSpeed and NewReno. The results show that the throughput performance of TCP-PPO2 can reach more than 2-3 times of the comparison method

while the delay value of 80% of the sampling points only increases 4% compared with the minimum link delay.

关键词

Keywords

references

FLOYD S, HENDERSON T. The NewReno modification to TCP's fast recovery algorithm: RFC2582 [A/OL]. [2020-07-01]. https:∥dl.acm.org/doi/pdf/10.17487/RFC2582.

HA S, RHEE I, XU Lisong. Cubic: a new TCP-friendly high-speed TCP variant [J]. ACM SIGOPS Operating Systems Review, 2008, 42(5): 64-74.

BRAKMO L S, O'MALLEY S W, PETERSON L L. TCP Vegas: new techniques for congestion detection and avoidance [C]∥Proceedings of the Conference on Communications Architectures, Protocols and Applications. New York, USA: ACM, 1994: 24-35.

MASCOLO S, CASETTI C, GERLA M, et al. TCP westwood: bandwidth estimation for enhanced transport over wireless links [C]∥Proceedings of the 7th Annual International Conference on Mobile Computing and Networking(MOBICOM). New York, USA: ACM. New York, USA: ACM, 2001: 287-297.

TAN K, SONG J, ZHANG Q, et al. A compound TCP approach for high-speed and long distance networks [C]∥Proceedings of the 25th IEEE International Conference on Computer Communications(INFOCOM). Piscataway, NJ, USA: IEEE, 2006: 4146841.

CARDWELL N, CHENG Y, GUNN C S, et al. BBR: congestion-based congestion control [J]. Communications of the ACM, 2017, 60(2): 58-66.

XIAO Liang, LU Xiaozhen, XU Dongjin, et al. UAV relay in VANETs against smart jamming with reinforcement learning [J]. IEEE Transactions on Vehicular Technology, 2018, 67(5): 4087-4097.

NIROUI F, ZHANG Kaicheng, KASHINO Z, et al. Deep reinforcement learning robot for search and rescue applications: exploration in unknown cluttered environments [J]. IEEE Robotics and Automation Letters, 2019, 4(2): 610-617.

HUANG Shanfeng, LV B, WANG Rui, et al. Scheduling for mobile edge computing with random user arrivals: an approximate MDP and reinforcement learning approach [J]. IEEE Transactions on Vehicular Technology, 2020, 69(7): 7735-7750.

CAO Zhengcai, LIN Chengran, ZHOU Mengchu, et al. Scheduling semiconductor testing facility by using cuckoo search algorithm with reinforcement learning and surrogate modeling [J]. IEEE Transactions on Automation Science and Engineering, 2019, 16(2): 825-837.

颛孙少帅, 杨俊安, 刘辉, 等. 采用双层强化学习的干扰决策算法 [J]. 西安交通大学学报, 2018, 52(2): 63-69.

ZHUANSUN Shaoshuai, YANG Junan, LIU Hui, et al. An algorithm for jamming decision using dual reinforcement learning [J]. Journal of Xi'an Jiaotong University, 2018, 52(2): 63-69.

MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning [J]. Nature, 2015, 518(7540): 529-533.

HABACHI O, SHIANG H P, VAN DER SCHAAR M, et al. Online learning based congestion control for adaptive multimedia transmission [J]. IEEE Transactions on Signal Processing, 2013, 61(6): 1460-1469.

VAN DER HOOFT J, PETRANGELI S, CLAEYS M, et al. A learning-based algorithm for improved bandwidth-awareness of adaptive streaming clients [C]∥Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management(IM). Piscataway, NJ, USA: IEEE, 2015: 131-138.

CUI Laizhong, YUAN Zuxian, MING Zhongxing, et al. Improving the congestion control performance for mobile networks in high-speed railway via deep rein-forcement learning [J]. IEEE Transactions on Vehicular Technology, 2020, 69(6): 5864-5875.

GU Lin, ZENG Deze, LI Wei, et al. Intelligent VNF orchestration and flow scheduling via model-assisted deep reinforcement learning [J]. IEEE Journal on Selected Areas in Communications, 2020, 38(2): 279-291.

NA W, BAE B, CHO S, et al. DL-TCP: Deep learning-based transmission control protocol for disaster 5G mmwave networks [J]. IEEE Access, 2019, 7: 145134-145144.

XIE Ruitao, JIA Xiaohua, WU Kaishun. Adaptive online decision method for initial congestion window in 5G mobile edge computing using deep reinforcement learning [J]. IEEE Journal on Selected Areas in Communications, 2020, 38(2): 389-403.

LAN Dehao, TAN Xiaobin, LV Jinyang, et al. A deep reinforcement learning based congestion control mechanism for NDN [C]∥Proceedings of the 2019 IEEE International Conference on Communications(ICC). Piscataway, NJ, USA: IEEE, 2019: 8761737.

XIAO Kefan, MAO Shiwen, TUGNAIT J K. TCP-drinc: smart congestion control based on deep reinforcement learning [J]. IEEE Access, 2019, 7: 11892-11904.

BACHL M, ZSEBY T, FABINI J. Rax: deep reinforcement learning for congestion control [C]∥Proceedings of the 2019 IEEE International Conference on Communications(ICC). Piscataway, NJ, USA: IEEE, 2019: 8761187.

LI Wei, ZHOU Fan, CHOWDHURY K R, et al. QTCP: adaptive congestion control with reinforcement learning [J]. IEEE Transactions on Network Science and Engineering, 2019, 6(3): 445-458.

WATKINS C J C H, DAYAN P. Q-learning [J]. Machine Learning, 1992, 8(3): 279-292.

SCHULMAN J, WOLSKI F, DHARIWAL P, et al. Proximal policy optimization algorithms [EB/OL]. [2020-07-01]. https:∥arxiv.org/pdf/1707.06347. pdf.

SCHULMAN J, LEVINE S, MORITZ P, et al. Trust region policy optimization [C]∥Proceedings of the 32nd International Conference on Machine Learning. Princeton, NJ, USA: International Machine Learning Society(IMLS), 2015: 1889-1897.

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

数据中心网络差分流传输控制协议研究

超临界二氧化碳向心透平轮背间隙内风阻损失及流动特性研究

扇贝仿生密封腔室内超临界二氧化碳泄漏流湍动能耗散机理研究

离子液体改性燃料的电流体动力学效应研究

多工况下离心压气机叶轮-机匣一体联合设计优化