RGB-Thermal Object Tracking Network Based on Attention Mechanism and Online Template Update

HAN Xiangdong; ZHONG Ao; LIU Chongao; SUN Yanxin; ZHANG Xiangyong; XU Linzhi

doi:10.7652/xjtuxb202508018

您当前的位置：

首页 >

文章列表页 >

RGB-Thermal Object Tracking Network Based on Attention Mechanism and Online Template Update

更新时间：2025-07-30

- RGB-Thermal Object Tracking Network Based on Attention Mechanism and Online Template Update
- Journal of Xi’an Jiaotong University Vol. 59, Issue 8, Pages: 187-198(2025)
- 作者机构：
  
  1.中国兵器工业试验测试研究院智能技术中心， 710065，西安
  2.西安电子科技大学电子工程学院， 710071，西安
- 作者简介：
- 基金信息：
- DOI：10.7652/xjtuxb202508018
  CLC： TP391.4
- Received：21 December 2024，
  
  Online First：07 April 2025，
  
  Published：10 August 2025
- 稿件说明：
移动端阅览
HAN Xiangdong, ZHONG Ao, LIU Chongao, et al. RGB-Thermal Object Tracking Network Based on Attention Mechanism and Online Template Update[J]. Journal of Xi’an Jiaotong University, 2025, 59(8): 187-198.
DOI：

HAN Xiangdong, ZHONG Ao, LIU Chongao, et al. RGB-Thermal Object Tracking Network Based on Attention Mechanism and Online Template Update[J]. Journal of Xi’an Jiaotong University, 2025, 59(8): 187-198. DOI： 10.7652/xjtuxb202508018.

摘要

针对当前可见光-红外目标跟踪算法对红外可见光双模态特征交互与目标动态变化建模不足的问题，结合卷积掩码自编码器模型，提出一种基于注意力与模板在线更新的可见光-红外双模态目标跟踪网络。以卷积掩码自编码器模型为骨干网络，通过采用双嵌入层配合共享权重的骨干网络结构提取可见光与红外特征，深入挖掘可见光与红外数据间的内在联系。通过强化模板与搜索图像的关联性，引入通道空间自注意力机制以增强模板和搜索图像间的交互，来提取模态间可区分的异质互补特征。提出模板在线更新模块，通过在线更新模板与设计模板分数头，利用置信度评分机制融合初始模板的稳定性与在线模板的适应性，解决目标随时间变化导致的模型漂移问题。实验结果表明，所提算法在GTOT和RGBT234公开数据集上的精确率和成功率分别达到93.3%/75.6%和87.2%/63.8%，可在目标不断变化情况下实现精确跟踪。可视化分析表明，所提算法在双模态热力图上可自适应互补，单一模态失效时仍能精准定位目标。

Abstract

To address the insufficient feature interaction between RGB and thermal modalities and the inadequate modeling of dynamic target variations in current RGB-thermal object tracking algorithms

a dual-modal tracking network based on attention and online template updates is proposed

incorporating a convolutional masked autoencoder model. Using the convolutional masked autoencoder as the backbone network

the model extracts RGB and thermal features through a dual-embedding layer with shared-weight backbone architecture

deeply exploring the intrinsic relationships between RGB and thermal data. To enhance the correlation between the template and search images

a channel-spatial self-attention mechanism is introduced to strengthen their interaction and extract discriminative heterogeneous complementary features across modalities. An online template update module is proposed

which dynamically updates the template and incorporates a template scoring head. By leveraging a confidence-based fusion mechanism

it balances the stability of the initial template and the adaptability of the online template

mitigating model drift caused by target variations over time. Experimental results demonstrate that the proposed algorithm achieves precision and success rates of 93.3%/75.6% and 87.2%/63.8% on the GTOT and RGBT234 datasets

respectively

enabling accurate tracking under dynamic target conditions. Visualization analysis shows that the algorithm adaptively complements dual-modal heatmaps and maintains precise target localization even when one modality fails.

关键词

Keywords

references

闵志方 , 杜虎 , 朱雪琼 , 等 . 单目标跟踪研究综述 [J ] . 光学与光电技术 , 2023 , 21 ( 4 ): 1 - 14 .

MIN Zhifang , DU Hu , ZHU Xueqiong , et al . Survey of single target tracking research [J ] . Optics & Optoelectronic Technology , 2023 , 21 ( 4 ): 1 - 14 .

马玉民 , 钱育蓉 , 周伟航 , 等 . 基于孪生网络的目标跟踪算法综述 [J ] . 计算机工程与科学 , 2023 , 45 ( 9 ): 1578 - 1592 .

MA Yumin , QIAN Yurong , ZHOU Weihang , et al . A survey of target tracking algorithms based on Siamese network [J ] . Computer Engineering & Science , 2023 , 45 ( 9 ): 1578 - 1592 .

韩瑞泽 , 冯伟 , 郭青 , 等 . 视频单目标跟踪研究进展综述 [J ] . 计算机学报 , 2022 , 45 ( 9 ): 1877 - 1907 .

HAN Ruize , FENG Wei , GUO Qing , et al . Single object tracking research: a survey [J ] . Chinese Journal of Computers , 2022 , 45 ( 9 ): 1877 - 1907 .

ZHANG Xingchen , YE Ping , LEUNG H , et al . Object fusion tracking based on visible and infrared images: a comprehensive review [J ] . Information Fusion , 2020 , 63 : 166 - 187 .

YE Botao , CHANG Hong , MA Bingpeng , et al . Joint feature learning and relation modeling for tracking: a one-stream framework [C ] // Computer Vision-ECCV 2022 . Cham, Switzerland : Springer Nature Switzerland , 2022 : 341 - 357 .

朱杰 , 翟素兰 , 刘磊 . 基于自适应上下文感知相关滤波的RGB-T目标跟踪 [J ] . 阜阳师范大学学报(自然科学版) , 2022 , 39 ( 3 ): 65 - 72 .

ZHU Jie , ZHAI Sulan , LIU Lei . RGB-T object tracking based on adaptive context-aware correlation filter [J ] . Journal of Fuyang Normal University (Natural Science) , 2022 , 39 ( 3 ): 65 - 72 .

张天路 , 张强 . 基于深度学习的RGB-T目标跟踪技术综述 [J ] . 模式识别与人工智能 , 2023 , 36 ( 4 ): 327 - 353 .

ZHANG Tianlu , ZHANG Qiang . A survey of RGB-T object tracking technologies based on deep learning [J ] . Pattern Recognition and Artificial Intelligence , 2023 , 36 ( 4 ): 327 - 353 .

李成龙 , 鹿安东 , 刘磊 , 等 . 多模态视觉跟踪方法综述 [J ] . 中国图象图形学报 , 2023 , 28 ( 1 ): 37 - 56 .

LI Chenglong , LU Andong , LIU Lei , et al . Multi-modal visual tracking: a survey [J ] . Journal of Image and Graphics , 2023 , 28 ( 1 ): 37 - 56 .

张兆宇 , 田春娜 , 周恒 , 等 . 联合在线分类的双注意力RGBT孪生网络跟踪 [J ] . 西安电子科技大学学报 , 2022 , 49 ( 6 ): 76 - 85 .

ZHANG Zhaoyu , TIAN Chunna , ZHOU Heng , et al . Online classification jointed RGBT tracking based on the dual attention Siamese network [J ] . Journal of Xidian University , 2022 , 49 ( 6 ): 76 - 85 .

ZHANG Pengyu , WANG Dong , LU Huchuan . Multi-modal visual tracking: review and experimental comparison [J ] . Computational Visual Media , 2024 , 10 ( 2 ): 193 - 214 .

ZHANG Xingming , ZHANG Xuehan , DU Xuedan , et al . Learning multi-domain convolutional network for RGB-T visual tracking [C ] // 2018 11th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI) . Piscataway, NJ, USA : IEEE , 2018 : 1 - 6 .

LI Chenglong , LU Andong , ZHENG Aihua , et al . Multi-adapter RGBT tracking [C ] // 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) . Piscataway, NJ, USA : IEEE , 2019 : 2262 - 2270 .

WANG Chaoqun , XU Chunyan , CUI Zhen , et al . Cross-modal pattern-propagation for RGB-T tracking [C ] // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway, NJ, USA : IEEE , 2020 : 7062 - 7071 .

ZHANG Xingchen , YE Ping , PENG Shengyun , et al . SiamFT: an RGB-infrared fusion tracking method via fully convolutional Siamese networks [J ] . IEEE Access , 2019 , 7 : 122122 - 122133 .

PENG Jingchao , ZHAO Haitao , HU Zhengwei , et al . Siamese infrared and visible light fusion network for RGB-T tracking [J ] . International Journal of Machine Learning and Cybernetics , 2023 , 14 ( 9 ): 3281 - 3293 .

FENG Mingzheng , SU Jianbo . Learning reliable modal weight with transformer for robust RGBT tracking [J ] . Knowledge-Based Systems , 2022 , 249 : 108945 .

WOO S , PARK J , LEE J Y , et al . CBAM: convolutional block attention module [C ] // Computer Vision-ECCV 2018 . Cham, Switzerland : Springer International Publishing , 2018 : 3 - 19 .

陈建明 , 李定鲣 , 曾祥津 , 等 . 一种跨模态光学信息交互和模板动态更新的RGBT目标跟踪方法 [J ] . 光学学报 , 2024 , 44 ( 7 ): 101 - 115 .

CHEN Jianming , LI Dingjian , ZENG Xiangjin , et al . Cross-modal optical information interaction and template dynamic update for RGBT target tracking method [J ] . Acta Optica Sinica , 2024 , 44 ( 7 ): 101 - 115 .

HU Weiming , WANG Qiang , ZHANG Li , et al . SiamMask: a framework for fast online object tracking and segmentation [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023 , 45 ( 3 ): 3072 - 3089 .

LIN T Y , GOYAL P , GIRSHICK R , et al . Focal loss for dense object detection [C ] // 2017 IEEE International Conference on Computer Vision (ICCV) . Piscataway, NJ, USA : IEEE , 2017 : 2999 - 3007 .

LI Xiang , WANG Wenhai , WU Lijun , et al . Generalized focal loss: learning qualified and distributed bounding boxes for dense object detection [C ] // Proceedings of the 34th International Conference on Neural Information Processing Systems . Red Hook, NY, USA : Curran Associates Inc. , 2020 : 21002 - 21012 .

REZATOFIGHI H , TSOI N , GWAK J Y , et al . Generalized intersection over union: a metric and a loss for bounding box regression [C ] // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway, NJ, USA : IEEE , 2019 : 658 - 666 .

JADON S . A survey of loss functions for semantic segmentation [C ] // 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB) . Piscataway, NJ, USA : IEEE , 2020 : 1 - 7 .

DENG Jia , DONG Wei , SOCHER R , et al . ImageNet: a large-scale hierarchical image database [C ] // 2009 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway, NJ, USA : IEEE , 2009 : 248 - 255 .

LI Chenglong , HUI Cheng , HU Shiyi , et al . Learning collaborative sparse representation for grayscale-thermal tracking [J ] . IEEE Transactions on Image Processing , 2016 , 25 ( 12 ): 5743 - 5756 .

LI Chenglong , LIANG Xinyan , LU Yijuan , et al . RGB-T object tracking: benchmark and baseline [J ] . Pattern Recognition , 2019 , 96 : 106977 .

VALMADRE J , BERTINETTO L , HENRIQUES J , et al . End-to-end representation learning for correlation filter based tracking [C ] // 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway, NJ, USA : IEEE , 2017 : 5000 - 5008 .

JUNG I , SON J , BAEK M , et al . Real-time MDNet [C ] // Computer Vision-ECCV 2018 . Cham, Switzerland : Springer International Publishing , 2018 : 89 - 104 .

NAM H , HAN B . Learning multi-domain convolutional neural networks for visual tracking [C ] // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway, NJ, USA : IEEE , 2016 : 4293 - 4302 .

ZHU Yabin , LI Chenglong , LUO Bin , et al . Dense feature aggregation and pruning for RGBT tracking [C ] // Proceedings of the 27th ACM International Conference on Multimedia . New York, NY, USA : ACM , 2019 : 465 - 472 .

ZHANG Hui , ZHANG Lei , ZHUO Li , et al . Object tracking in RGB-T videos using modal-aware attention network and competitive learning [J ] . Sensors , 2020 , 20 ( 2 ): 393 .

GAO Yuan , LI Chenglong , ZHU Yabin , et al . Deep adaptive fusion network for high performance RGBT tracking [C ] // 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) . Piscataway, NJ, USA : IEEE , 2019 : 91 - 99 .

LU Andong , LI Chenglong , YAN Yuqing , et al . RGBT tracking via multi-adapter network with hierarchical divergence loss [J ] . IEEE Transactions on Image Processing , 2021 , 30 : 5613 - 5625 .

XIAO Yun , YANG Mengmeng , LI Chenglong , et al . Attribute-based progressive fusion network for rgbt tracking [C ] // Proceedings of the AAAI Conference on Artificial Intelligence . Palo Alto, CA, USA : AAAI Press , 2022 : 2831 - 2838 .

LU Andong , QIAN Cun , LI Chenglong , et al . Duality-gated mutual condition network for RGBT tracking [J ] . IEEE Transactions on Neural Networks and Learning Systems , 2025 , 36 ( 3 ): 4118 - 4131 .

LI Chenglong , ZHAO Nan , LU Nan , et al . Weighted sparse representation regularized graph learning for RGB-T object tracking [C ] // Proceedings of the 25th ACM International Conference on Multimedia . New York, NY, USA : ACM , 2017 : 1856 - 1864 .

TANG Ming , FENG Jiayi . Multi-kernel correlation filter for visual tracking [C ] // 2015 IEEE International Conference on Computer Vision (ICCV) . Piscataway, NJ, USA : IEEE , 2015 : 3038 - 3046 .

HOU Xiaojun , XING Jiazheng , QIAN Yijie , et al . SDSTrack: self-distillation symmetric adapter learning for multi-modal visual object tracking [C ] // 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway, NJ, USA : IEEE , 2024 : 26541 - 26551 .

SUN Dengdi , PAN Yajie , LU Andong , et al . Transformer RGBT tracking with spatio-temporal multimodal tokens [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2024 , 34 ( 11 ): 12059 - 12072 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Early Identification of Unknown Applications Based on Sequence Features and Knowledge Guidance

Integrated and Joint Design Optimization of Centrifugal Compressor Impeller and Casing Treatment at Multiple Working Points

Energy Efficiency Optimization of the Cryogenic Air Separation Process for Electronic-Grade Ultra Pure Nitrogen

Bi-Level Multi-Obj ective Optimization of a Wind-Solar Coupled Hydrogen Production System Based on the Synergy of Hybrid Electrolyzer Arrays

I/O Performance Modeling and Prediction Method for Large-Scale Parallel Applications Using Machine Learning

Related Author

LIU Yitong

WANG Pinghui

ZHAO Junzhou

Related Institution

School of Automation and Engineering, Xi'an Jiaotong University

Ministry of Education Key Lab for Intelligent Networks and Network Security, Xi'an Jiaotong University

⁰