Collaborative optimization method for computing and storage oriented to parallel training of multimodal models

JU Tao; GUO Dongyu; WU Chengyuan; ZHU Chenxi

doi:null

您当前的位置：

首页 >

文章列表页 >

Collaborative optimization method for computing and storage oriented to parallel training of multimodal models

更新时间：2026-10-11

- Collaborative optimization method for computing and storage oriented to parallel training of multimodal models
- JOURNAL OF XI’AN JIAOTONG UNIVERSITY (2026)
- 作者机构：
  
  兰州交通大学电子与信息工程学院，兰州 730070
- 作者简介：
- 基金信息：
- DOI：
  CLC： TP311
- Received：22 June 2026，
  
  Revised：2026-07-27，
  
  Accepted：21 September 2026，
- 稿件说明：
移动端阅览
JU Tao, GUO Dongyu, WU Chengyuan, et al. Collaborative optimization method for computing and storage oriented to parallel training of multimodal models[J/OL]. JOURNAL OF XI’AN JIAOTONG UNIVERSITY,2026-10-11. http://zkxb.xjtu.edu.cn/en/article/173714125/.
DOI：

JU Tao, GUO Dongyu, WU Chengyuan, et al. Collaborative optimization method for computing and storage oriented to parallel training of multimodal models[J/OL]. JOURNAL OF XI’AN JIAOTONG UNIVERSITY,2026-10-11. http://zkxb.xjtu.edu.cn/en/article/173714125/. DOI：

摘要

针对多模态模型并行训练中，因模型参数量大、模态异构、计算资源受限导致的模型无法加载、计算负载不均衡、通信时延过大等问题，提出了一种计算存储协同优化方法。对多模态模型进行细粒度模态剖析，获取多模态模型异构模态特征，设计多模态模型流水线并行切分策略，解决模型无法加载问题；通过构建融合不同层次存储空间的统一虚拟化内存池，将参数、梯度与激活值按访问频率与生命周期分布于不同层级的存储空间中，实现显存容量的有效扩充，使多模态模型训练不再受限于显存大小；最后设计模态感知的调度优化方法，依据不同模态编码器的计算密度与时延差异分配GPU计算资源，同时设计异步传输与数据预取机制，进一步减小通信开销，实现计算任务的细粒度调度，均衡计算设备负载。实验结果表明，本文方法和已有方法相比，可在保证模型稳定收敛的同时，大幅降低存储和通信开销、充分利用计算资源提升模型整体训练效率。

Abstract

To address the issues of model loading failure

unbalanced computing load

and excessive communication delay in the parallel training of multimodal models due to large model parameters

heterogeneous modalities

and limited computing resources

a computing and storage collaborative optimization method is proposed. A fine-grained modality analysis of the multimodal model is conducted to obtain the heterogeneous modality features of the multimodal model

and a multimodal model pipeline parallel splitting strategy is designed to solve the problem of model loading failure. A unified virtualized memory pool integrating different levels of storage space is constructed

and parameters

gradients

and activation values are distributed in different levels of storage space according to their access frequency and life cycle

effectively expanding the capacity of the graphics memory and eliminating the limitation of the size of the graphics memory on the training of multimodal models. Finally

a modality-aware scheduling optimization method is designed to allocate GPU computing resources based on the computing density and delay differences of different modality encoders

and an asynchronous transmission and data prefetching mechanism is designed to further reduce communication overhead and achieve fine-grained scheduling of computing tasks

balancing the load of computing devices. Experimental results show that compared with existing methods

the proposed method can significantly reduce storage and communication overhead

fully utilize computing resources

and improve the overall training efficiency of the model while ensuring stable convergence of the model.

关键词

Keywords

references

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [J ] . Advances in Neural Information Processing Systems , 2017 , 30 : 5998 - 6008 .

KOROTEEV M V . BERT: a review of applications in natural language processing and understanding [J/OL ] . arXiv: 2103.11943 , 2021 .

ZHAO W X , ZHOU K , LI J , et al . A survey of large language models [J/OL ] . arXiv: 2303.18223 , 2023 .

朱虎明 , 李佩 , 焦李成 , 等 . 深度神经网络并行化研究综述 [J/OL ] . 计算机学报 , 2018 , 41 ( 8 ): 1861 - 1881 .

ZHU H M , LI P , JIAO L C , et al . Review of parallel deep neural network [J ] . Chinese Journal of Computers , 2018 , 41 ( 8 ): 1861 - 1881 . (in Chinese)

李东升 , 唐宇 , 乔林波 , 等 . 面向国产平台的大模型训练显存优化技术 [J ] . 国防科技大学学报 , 2026 , 48 ( 02 ): 284 - 295 .

LI D S , TANG Y , QIAO L B , et al . Technologies for memory optimization for large model training on domestic platforms [J ] . Journal of National University of Defense Technology , 2026 , 48 ( 2 ): 284 - 295 . (in Chinese)

王恩东 , 闫瑞栋 , 郭振华 , 等 . 分布式训练系统及其优化算法综述 [J ] . 计算机学报 , 2024 , 47 ( 01 ): 1 - 28 .

WANG E D , YAN R D , GUO Z H , et al . A survey of distributed training system and its optimization algorithms [J ] . Chinese Journal of Computers , 2024 , 47 ( 01 ): 1 - 28 . (in Chinese)

卢凯 , 赖志权 , 李笙维 , 等 . 并行智能训练技术：挑战与发展 [J ] . 中国科学：信息科学， 2023 , 53 ( 8 ): 1441 – 1468 .

LU K , LAI Z Q , LI S W , et al . Parallel intelligent computing: development and challenges [J ] . Science China Information Sciences , 2023 , 53 ( 8 ): 1441 - 1468 . (in Chinese)

GAO H R , LUO D H , WU Y W , et al . vTensor-based GPU memory management for edge deep learning training [J ] . Journal of Computer Science and Technology , 2025 , 40 ( 6 ): 1608 - 1625 .

CHEN X , CHEN D Z , HAN Y , et al . moDNN: Memory optimal deep neural network training on graphics processing units [J ] . IEEE Transactions on Parallel and Distributed Systems , 2018 , 30 ( 3 ): 646 - 661 .

HUANG C C , JIN G , LI J . SwapAdvisor: Pushing deep learning beyond the GPU memory limit via smart swapping [C ] // Proceedings of the 25th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) . New York : ACM , 2020 : 1341 - 1355 .

RAJBHANDARI S , RASLEY J , RUWASE O , et al . ZeRO: Memory optimizations toward training trillion parameter models [C ] // Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC) . New York : IEEE , 2020 : 1 - 16 .

RAJBHANDARI S , RUWASE O , RASLEY J , et al . ZeRO-Infinity: Breaking the GPU memory wall for extreme scale deep learning [C ] // Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC) . New York : IEEE , 2021 : 1 - 14 .

REN J , RAJBHANDARI S , AMINABADI R Y , et al . ZeRO-Offload: Democratizing billion-scale model training [C ] // Proceedings of the USENIX Annual Technical Conference (USENIX ATC) . Berkeley : USENIX Association , 2021 : 551 - 564 .

RYU J , YU J , et al . SPipe: Hybrid GPU and CPU pipeline for training LLMs under memory pressure [C ] // Proceedings of the 34th International Conference on Parallel Architectures and Compilation Techniques . Piscataway : IEEE , 2025 : 14 - 29 .

SERGEEV A , DEL BALSO M . Horovod: Fast and easy distributed deep learning in TensorFlow [J/OL ] . arXiv: 1802.05799 , 2018 .

LI M , ANDERSEN D G , SMOLA A , et al . Communication efficient distributed machine learning with the parameter server [J ] . Advances in Neural Information Processing Systems , 2014 , 27 : 19 - 27 .

巨涛 , 康贺廷 , 刘帅 , 等 . 一种深度神经网络多步延迟参数更新并行优化方法 [J ] . 哈尔滨工业大学学报 , 2025 , 57 ( 09 ): 95 - 108 .

JU T , KANG H T , LIU S , et al . A multi-step delayed parameter update parallel optimization method for deep neural networks [J ] . Journal of Harbin Institute of Technology , 2025 , 57 ( 9 ): 95 - 108 . (in Chinese)

HUANG Y , CHENG Y , BAPNA A , et al . GPipe: Efficient training of giant neural networks using pipeline parallelism [J ] . Advances in Neural Information Processing Systems , 2019 , 32 : 103 - 112 .

NARAYANAN D , HARLAP A , PHANISHAYEE A , et al . PipeDream: Generalized pipeline parallelism for DNN training [C ] // Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP) . New York : ACM , 2019 : 1 - 15 .

XUE Z , HU H , CHEN X , et al . PipeWeaver: Addressing data dynamicity in large multimodal model training with dynamic interleaved pipeline [J/OL ] . arXiv: 2504.14145 , 2025 .

WU H , CHEN L . TawPipe: Topology-aware weight pipeline parallelism for accelerating long-context large models training [C ] // Proceedings of the AAAI Conference on Artificial Intelligence . 2026 , 40 ( 32 ): 26894 - 26902 .

ZHANG Z , ZHONG Y , JIANG Y , et al . Disttrain: Addressing model and data heterogeneity with disaggregated training for multimodal large language models [C ] // Proceedings of the ACM SIGCOMM 2025 Conference . 2025 : 24 - 38 .

SHOEYBI M , PATWARY M , PURI R , et al . Megatron-LM: Training multi-billion parameter language models using model parallelism [J/OL ] . arXiv: 1909.08053 , 2019 .

LIN J , LIU Z , YOU Y , et al . WeiPipe: Weight pipeline parallelism for communication-effective long-context large model training [C ] // Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming . 2025 : 225 - 238 .

巨涛 , 丁肖健 , 郭东雨 , 等 . 一种面向多模态模型的分区混合并行优化方法 [J ] . 西安交通大学学报 , 2026 , 60 ( 02 ): 229 - 240 .

JU T , DING X J , GUO D Y , et al . A partitioned hybrid parallel optimization method for multimodal models [J ] . Journal of Xi'an Jiaotong University , 2026 , 60 ( 2 ): 229 - 240 . (in Chinese)

AHMADITESHNIZI A , GAO W , UDELL M . Optimus: Scalable optimization modeling with (MI)LP solvers and large language models [J/OL ] . arXiv: 2402.10172 , 2024 .

HUANG J , ZHANG Z , ZHENG S , et al . DistMM: Accelerating distributed multimodal model training [C ] // Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI) . Berkeley : USENIX Association , 2024 : 1157 - 1171 .

Feng Y , Xie M , Tian Z , et al . Mobius: Fine tuning large-scale models on commodity GPU servers [C ] // Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2023) . New York : ACM , 2023 : 480 - 494 .

GUO C , ZHANG R , XU J , et al . GMLake: Efficient and transparent GPU memory defragmentation for large-scale DNN training with virtual memory stitching [C ] // Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems . New York : ACM , 2024 : 450 - 466 .

贺巩山 , 赵传磊 , 蒋金虎 , 等 . 面向深度学习的数据存储技术综述 [J/OL ] . 计算机学报， 2025 ( 2025-02-17 )[ 2024-07-07 ] .

HE G S , ZHAO C L , JIANG J H , et al . A survey of data storage technologies for deep learning [J/OL ] . Chinese Journal of Computers , 2025-02-17 [ 2024-07-07 ] . (in Chinese)

巨涛 , 康贺廷 , 刘帅 , 等 . 深度神经网络动态分层梯度稀疏化及梯度合并优化方法 [J ] . 西安交通大学学报 , 2024 , 58 ( 9 ): 105 - 116 .

JU T , KANG H T , LIU S , et al . Dynamic hierarchical gradient sparsification and gradient merging optimization method for deep neural networks [J ] . Journal of Xi'an Jiaotong University , 2024 , 58 ( 9 ): 105 - 116 . (in Chinese)

LIU S , JU T . APapo: An asynchronous parallel optimization method for DNN models [J ] . Future Generation Computer Systems , 2024 , 152 : 317 - 330 .

RADFORD A , KIM J W , HALLACY C , et al . Learning transferable visual models from natural language supervision [C ] // Proceedings of the 38th International Conference on Machine Learning (ICML) . New York : PMLR , 2021 : 8748 - 8763 .

LI J , LI D , SAVARESE S , et al . BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation [C ] // Proceedings of the 39th International Conference on Machine Learning . New York : PMLR , 2022 : 12888 - 12900 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

An Object Distribution Algorithm Based on Arc and Two-Layer Mapping

Experimental Study on Combined Impingement/Film Cooling Characteristics at the Leading Edge of a Turbine Vane

Application and Optimization of Anaerobic Digestion Model No.1 for Photo-Fermentative Hydrogen Production

Dynamic Sensitivity Analysis of Turbine-Generator Shaft Train Based on Dynamic Load Correction

Drag Reduction Characteristics of Longitudinal Grooves under Limited Gap Height Conditions

Related Author

聂世强

伍卫国

崔金华

薛尚山

刘钊华

Related Institution

School of the Electronic and Information Engineering, Xi'an Jiaotong University, Xi'an 710049, Chian

⁰