Intelligent TCP Congestion Control Method for Virtual Data Space

doi:10.7652/xjtuxb202105010

您当前的位置：

首页 >

文章列表页 >

Intelligent TCP Congestion Control Method for Virtual Data Space

更新时间：2025-07-09

- Intelligent TCP Congestion Control Method for Virtual Data Space
- Vol. 55, Issue 5, Pages: 83-91(2021)
- 作者机构：
  
  西安交通大学计算机科学与技术学院,西安,710049
- 作者简介：
- 基金信息：
- DOI：10.7652/xjtuxb202105010
  CLC： TP319
- Online First：10 May 2021，
  
  Published：2021
- 稿件说明：
移动端阅览
Intelligent TCP Congestion Control Method for Virtual Data Space[J]. 2021, 55(5): 83-91.
DOI：

Intelligent TCP Congestion Control Method for Virtual Data Space[J]. 2021, 55(5): 83-91. DOI： 10.7652/xjtuxb202105010.

摘要

为优化虚拟数据空间网络传输性能

提出了基于近端策略优化的智能TCP拥塞控制算法TCP-PPO2。将TCP拥塞控制过程抽象为一个可部分观察的马尔可夫决策过程

在该过程中构建一个智能体

与网络环境进行互动。智能体通过观察网络状态特征对拥塞窗口长度进行调节

网络环境向智能体反馈奖励值

智能体尝试最大化回合内获得奖励期望值。设计了包括吞吐率、网络时延等网络特征的状态空间

使智能体能够观察到足够多的信息进行决策并且降低性能开销。通过加权算法设计奖励函数

使智能体能够平衡优化吞吐率与时延。通过近端策略优化算法更新智能体模型参数

对过大的参数更新进行截断

将参数更新限制在一定范围内

减少梯度下降过程中出现的振荡

实现训练过程的快速收敛。在NS3模拟器上实现了基于近端策略优化的TCP拥塞控制算法

并与Cubic、HighSpeed和NewReno等主流拥塞控制算法进行了对比

结果表明:TCP-PPO2吞吐率性能可达对比算法的2～3倍以上; 80%的采样点时延相比链路最小时延值只增加了4%。

Abstract

To optimize the transmission performance of virtual data space network

an intelligent TCP congestion control algorithm based on proximal policy optimization is proposed(TCP-PPO2). The TCP congestion control process is abstracted as a Markov decision process

which can be partially observed. In this process

an agent is constructed to interact with the network environment. The agent adjusts the size of congestion window by observing the characteristics of network state. The network environment feeds back a reward value to the agent

and the agent tries to maximize the expected reward value in an episode. The state space including throughput

network delay and other network characteristics is designed

so that agents can observe enough information to make decisions and reduce performance overhead. The weighted reward function is designed to balance the throughput and delay. The parameters of the agent model are updated by the proximal policy optimization algorithm

and excessive parameter updates are truncated. The parameter update is limited to a certain range

which reduces the oscillation problem in the process of gradient descent

and realizes quick convergence of the training process. The TCP congestion control algorithm based on proximal policy optimization is implemented on NS3 simulator

and compared with the mainstream congestion control algorithms such as cubic

HighSpeed and NewReno. The results show that the throughput performance of TCP-PPO2 can reach more than 2-3 times of the comparison method

while the delay value of 80% of the sampling points only increases 4% compared with the minimum link delay.

关键词

Keywords

references

FLOYD S, HENDERSON T. The NewReno modification to TCP's fast recovery algorithm: RFC2582 [A/OL]. [2020-07-01]. https:∥dl.acm.org/doi/pdf/10.17487/RFC2582.

HA S, RHEE I, XU Lisong. Cubic: a new TCP-friendly high-speed TCP variant [J]. ACM SIGOPS Operating Systems Review, 2008, 42(5): 64-74.

BRAKMO L S, O'MALLEY S W, PETERSON L L. TCP Vegas: new techniques for congestion detection and avoidance [C]∥Proceedings of the Conference on Communications Architectures, Protocols and Applications. New York, USA: ACM, 1994: 24-35.

MASCOLO S, CASETTI C, GERLA M, et al. TCP westwood: bandwidth estimation for enhanced transport over wireless links [C]∥Proceedings of the 7th Annual International Conference on Mobile Computing and Networking(MOBICOM). New York, USA: ACM. New York, USA: ACM, 2001: 287-297.

TAN K, SONG J, ZHANG Q, et al. A compound TCP approach for high-speed and long distance networks [C]∥Proceedings of the 25th IEEE International Conference on Computer Communications(INFOCOM). Piscataway, NJ, USA: IEEE, 2006: 4146841.

CARDWELL N, CHENG Y, GUNN C S, et al. BBR: congestion-based congestion control [J]. Communications of the ACM, 2017, 60(2): 58-66.

XIAO Liang, LU Xiaozhen, XU Dongjin, et al. UAV relay in VANETs against smart jamming with reinforcement learning [J]. IEEE Transactions on Vehicular Technology, 2018, 67(5): 4087-4097.

NIROUI F, ZHANG Kaicheng, KASHINO Z, et al. Deep reinforcement learning robot for search and rescue applications: exploration in unknown cluttered environments [J]. IEEE Robotics and Automation Letters, 2019, 4(2): 610-617.

HUANG Shanfeng, LV B, WANG Rui, et al. Scheduling for mobile edge computing with random user arrivals: an approximate MDP and reinforcement learning approach [J]. IEEE Transactions on Vehicular Technology, 2020, 69(7): 7735-7750.

CAO Zhengcai, LIN Chengran, ZHOU Mengchu, et al. Scheduling semiconductor testing facility by using cuckoo search algorithm with reinforcement learning and surrogate modeling [J]. IEEE Transactions on Automation Science and Engineering, 2019, 16(2): 825-837.

颛孙少帅, 杨俊安, 刘辉, 等. 采用双层强化学习的干扰决策算法 [J]. 西安交通大学学报, 2018, 52(2): 63-69.

ZHUANSUN Shaoshuai, YANG Junan, LIU Hui, et al. An algorithm for jamming decision using dual reinforcement learning [J]. Journal of Xi'an Jiaotong University, 2018, 52(2): 63-69.

MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning [J]. Nature, 2015, 518(7540): 529-533.

HABACHI O, SHIANG H P, VAN DER SCHAAR M, et al. Online learning based congestion control for adaptive multimedia transmission [J]. IEEE Transactions on Signal Processing, 2013, 61(6): 1460-1469.

VAN DER HOOFT J, PETRANGELI S, CLAEYS M, et al. A learning-based algorithm for improved bandwidth-awareness of adaptive streaming clients [C]∥Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management(IM). Piscataway, NJ, USA: IEEE, 2015: 131-138.

CUI Laizhong, YUAN Zuxian, MING Zhongxing, et al. Improving the congestion control performance for mobile networks in high-speed railway via deep rein-forcement learning [J]. IEEE Transactions on Vehicular Technology, 2020, 69(6): 5864-5875.

GU Lin, ZENG Deze, LI Wei, et al. Intelligent VNF orchestration and flow scheduling via model-assisted deep reinforcement learning [J]. IEEE Journal on Selected Areas in Communications, 2020, 38(2): 279-291.

NA W, BAE B, CHO S, et al. DL-TCP: Deep learning-based transmission control protocol for disaster 5G mmwave networks [J]. IEEE Access, 2019, 7: 145134-145144.

XIE Ruitao, JIA Xiaohua, WU Kaishun. Adaptive online decision method for initial congestion window in 5G mobile edge computing using deep reinforcement learning [J]. IEEE Journal on Selected Areas in Communications, 2020, 38(2): 389-403.

LAN Dehao, TAN Xiaobin, LV Jinyang, et al. A deep reinforcement learning based congestion control mechanism for NDN [C]∥Proceedings of the 2019 IEEE International Conference on Communications(ICC). Piscataway, NJ, USA: IEEE, 2019: 8761737.

XIAO Kefan, MAO Shiwen, TUGNAIT J K. TCP-drinc: smart congestion control based on deep reinforcement learning [J]. IEEE Access, 2019, 7: 11892-11904.

BACHL M, ZSEBY T, FABINI J. Rax: deep reinforcement learning for congestion control [C]∥Proceedings of the 2019 IEEE International Conference on Communications(ICC). Piscataway, NJ, USA: IEEE, 2019: 8761187.

LI Wei, ZHOU Fan, CHOWDHURY K R, et al. QTCP: adaptive congestion control with reinforcement learning [J]. IEEE Transactions on Network Science and Engineering, 2019, 6(3): 445-458.

WATKINS C J C H, DAYAN P. Q-learning [J]. Machine Learning, 1992, 8(3): 279-292.

SCHULMAN J, WOLSKI F, DHARIWAL P, et al. Proximal policy optimization algorithms [EB/OL]. [2020-07-01]. https:∥arxiv.org/pdf/1707.06347. pdf.

SCHULMAN J, LEVINE S, MORITZ P, et al. Trust region policy optimization [C]∥Proceedings of the 32nd International Conference on Machine Learning. Princeton, NJ, USA: International Machine Learning Society(IMLS), 2015: 1889-1897.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Differentiated Flow Transmission Control Protocol in Data Center Networks

Research on the Electrohydrodynamic Effects of Ionic Liquid-Modified Fuels

Bi-Level Multi-Obj ective Optimization of a Wind-Solar Coupled Hydrogen Production System Based on the Synergy of Hybrid Electrolyzer Arrays

Sensitivity Analysis of Foil Geometrical Parameters for the Third-Generation CO2 Gas Foil Bearings Based on Design of Experiments Method

A Structure-Semantics-Residual Collaborative Image Compression Framework：Towards Privacy Protection and High-Fidelity Reconstruction

Related Author

蔡岳平

张文鹏

罗森

Related Institution

College of Communication Engineering, Chongqing University

⁰