采用注意力与模板在线更新的可见光-红外目标跟踪网络

韩向东; 钟傲; 刘冲澳; 孙延鑫; 张向永; 徐淋智

doi:10.7652/xjtuxb202508018

您当前的位置：

首页 >

文章列表页 >

采用注意力与模板在线更新的可见光-红外目标跟踪网络

—————————————————————— | 更新时间：2025-07-30

- 采用注意力与模板在线更新的可见光-红外目标跟踪网络
- RGB-Thermal Object Tracking Network Based on Attention Mechanism and Online Template Update
- 西安交通大学学报 2025年59卷第8期页码：187-198
- 作者机构：
  
  1.中国兵器工业试验测试研究院智能技术中心， 710065，西安
  2.西安电子科技大学电子工程学院， 710071，西安
- 作者简介：
  
  韩向东（1989—），男，副研究员。
- 基金信息：
  
  国家自然科学基金资助项目(62173265);航空科学基金资助项目(F024020021);国防科工局资助项目
- DOI：10.7652/xjtuxb202508018
  中图分类号： TP391.4
- 收稿：2024-12-21，
  
  网络首发：2025-04-07，
  
  纸质出版：2025-08-10
- 稿件说明：
移动端阅览
韩向东, 钟傲, 刘冲澳, 等. 采用注意力与模板在线更新的可见光-红外目标跟踪网络[J]. 西安交通大学学报, 2025,59(8):187-198.

HAN Xiangdong, ZHONG Ao, LIU Chongao, et al. RGB-Thermal Object Tracking Network Based on Attention Mechanism and Online Template Update[J]. Journal of Xi’an Jiaotong University, 2025, 59(8): 187-198.
韩向东, 钟傲, 刘冲澳, 等. 采用注意力与模板在线更新的可见光-红外目标跟踪网络[J]. 西安交通大学学报, 2025,59(8):187-198. DOI： 10.7652/xjtuxb202508018.

HAN Xiangdong, ZHONG Ao, LIU Chongao, et al. RGB-Thermal Object Tracking Network Based on Attention Mechanism and Online Template Update[J]. Journal of Xi’an Jiaotong University, 2025, 59(8): 187-198. DOI： 10.7652/xjtuxb202508018.

摘要

针对当前可见光-红外目标跟踪算法对红外可见光双模态特征交互与目标动态变化建模不足的问题，结合卷积掩码自编码器模型，提出一种基于注意力与模板在线更新的可见光-红外双模态目标跟踪网络。以卷积掩码自编码器模型为骨干网络，通过采用双嵌入层配合共享权重的骨干网络结构提取可见光与红外特征，深入挖掘可见光与红外数据间的内在联系。通过强化模板与搜索图像的关联性，引入通道空间自注意力机制以增强模板和搜索图像间的交互，来提取模态间可区分的异质互补特征。提出模板在线更新模块，通过在线更新模板与设计模板分数头，利用置信度评分机制融合初始模板的稳定性与在线模板的适应性，解决目标随时间变化导致的模型漂移问题。实验结果表明，所提算法在GTOT和RGBT234公开数据集上的精确率和成功率分别达到93.3%/75.6%和87.2%/63.8%，可在目标不断变化情况下实现精确跟踪。可视化分析表明，所提算法在双模态热力图上可自适应互补，单一模态失效时仍能精准定位目标。

Abstract

To address the insufficient feature interaction between RGB and thermal modalities and the inadequate modeling of dynamic target variations in current RGB-thermal object tracking algorithms

a dual-modal tracking network based on attention and online template updates is proposed

incorporating a convolutional masked autoencoder model. Using the convolutional masked autoencoder as the backbone network

the model extracts RGB and thermal features through a dual-embedding layer with shared-weight backbone architecture

deeply exploring the intrinsic relationships between RGB and thermal data. To enhance the correlation between the template and search images

a channel-spatial self-attention mechanism is introduced to strengthen their interaction and extract discriminative heterogeneous complementary features across modalities. An online template update module is proposed

which dynamically updates the template and incorporates a template scoring head. By leveraging a confidence-based fusion mechanism

it balances the stability of the initial template and the adaptability of the online template

mitigating model drift caused by target variations over time. Experimental results demonstrate that the proposed algorithm achieves precision and success rates of 93.3%/75.6% and 87.2%/63.8% on the GTOT and RGBT234 datasets

respectively

enabling accurate tracking under dynamic target conditions. Visualization analysis shows that the algorithm adaptively complements dual-modal heatmaps and maintains precise target localization even when one modality fails.

关键词

Keywords

references

闵志方 , 杜虎 , 朱雪琼 , 等 . 单目标跟踪研究综述 [J ] . 光学与光电技术 , 2023 , 21 ( 4 ): 1 - 14 .

MIN Zhifang , DU Hu , ZHU Xueqiong , et al . Survey of single target tracking research [J ] . Optics & Optoelectronic Technology , 2023 , 21 ( 4 ): 1 - 14 .

马玉民 , 钱育蓉 , 周伟航 , 等 . 基于孪生网络的目标跟踪算法综述 [J ] . 计算机工程与科学 , 2023 , 45 ( 9 ): 1578 - 1592 .

MA Yumin , QIAN Yurong , ZHOU Weihang , et al . A survey of target tracking algorithms based on Siamese network [J ] . Computer Engineering & Science , 2023 , 45 ( 9 ): 1578 - 1592 .

韩瑞泽 , 冯伟 , 郭青 , 等 . 视频单目标跟踪研究进展综述 [J ] . 计算机学报 , 2022 , 45 ( 9 ): 1877 - 1907 .

HAN Ruize , FENG Wei , GUO Qing , et al . Single object tracking research: a survey [J ] . Chinese Journal of Computers , 2022 , 45 ( 9 ): 1877 - 1907 .

ZHANG Xingchen , YE Ping , LEUNG H , et al . Object fusion tracking based on visible and infrared images: a comprehensive review [J ] . Information Fusion , 2020 , 63 : 166 - 187 .

YE Botao , CHANG Hong , MA Bingpeng , et al . Joint feature learning and relation modeling for tracking: a one-stream framework [C ] // Computer Vision-ECCV 2022 . Cham, Switzerland : Springer Nature Switzerland , 2022 : 341 - 357 .

朱杰 , 翟素兰 , 刘磊 . 基于自适应上下文感知相关滤波的RGB-T目标跟踪 [J ] . 阜阳师范大学学报(自然科学版) , 2022 , 39 ( 3 ): 65 - 72 .

ZHU Jie , ZHAI Sulan , LIU Lei . RGB-T object tracking based on adaptive context-aware correlation filter [J ] . Journal of Fuyang Normal University (Natural Science) , 2022 , 39 ( 3 ): 65 - 72 .

张天路 , 张强 . 基于深度学习的RGB-T目标跟踪技术综述 [J ] . 模式识别与人工智能 , 2023 , 36 ( 4 ): 327 - 353 .

ZHANG Tianlu , ZHANG Qiang . A survey of RGB-T object tracking technologies based on deep learning [J ] . Pattern Recognition and Artificial Intelligence , 2023 , 36 ( 4 ): 327 - 353 .

李成龙 , 鹿安东 , 刘磊 , 等 . 多模态视觉跟踪方法综述 [J ] . 中国图象图形学报 , 2023 , 28 ( 1 ): 37 - 56 .

LI Chenglong , LU Andong , LIU Lei , et al . Multi-modal visual tracking: a survey [J ] . Journal of Image and Graphics , 2023 , 28 ( 1 ): 37 - 56 .

张兆宇 , 田春娜 , 周恒 , 等 . 联合在线分类的双注意力RGBT孪生网络跟踪 [J ] . 西安电子科技大学学报 , 2022 , 49 ( 6 ): 76 - 85 .

ZHANG Zhaoyu , TIAN Chunna , ZHOU Heng , et al . Online classification jointed RGBT tracking based on the dual attention Siamese network [J ] . Journal of Xidian University , 2022 , 49 ( 6 ): 76 - 85 .

ZHANG Pengyu , WANG Dong , LU Huchuan . Multi-modal visual tracking: review and experimental comparison [J ] . Computational Visual Media , 2024 , 10 ( 2 ): 193 - 214 .

ZHANG Xingming , ZHANG Xuehan , DU Xuedan , et al . Learning multi-domain convolutional network for RGB-T visual tracking [C ] // 2018 11th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI) . Piscataway, NJ, USA : IEEE , 2018 : 1 - 6 .

LI Chenglong , LU Andong , ZHENG Aihua , et al . Multi-adapter RGBT tracking [C ] // 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) . Piscataway, NJ, USA : IEEE , 2019 : 2262 - 2270 .

WANG Chaoqun , XU Chunyan , CUI Zhen , et al . Cross-modal pattern-propagation for RGB-T tracking [C ] // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway, NJ, USA : IEEE , 2020 : 7062 - 7071 .

ZHANG Xingchen , YE Ping , PENG Shengyun , et al . SiamFT: an RGB-infrared fusion tracking method via fully convolutional Siamese networks [J ] . IEEE Access , 2019 , 7 : 122122 - 122133 .

PENG Jingchao , ZHAO Haitao , HU Zhengwei , et al . Siamese infrared and visible light fusion network for RGB-T tracking [J ] . International Journal of Machine Learning and Cybernetics , 2023 , 14 ( 9 ): 3281 - 3293 .

FENG Mingzheng , SU Jianbo . Learning reliable modal weight with transformer for robust RGBT tracking [J ] . Knowledge-Based Systems , 2022 , 249 : 108945 .

WOO S , PARK J , LEE J Y , et al . CBAM: convolutional block attention module [C ] // Computer Vision-ECCV 2018 . Cham, Switzerland : Springer International Publishing , 2018 : 3 - 19 .

陈建明 , 李定鲣 , 曾祥津 , 等 . 一种跨模态光学信息交互和模板动态更新的RGBT目标跟踪方法 [J ] . 光学学报 , 2024 , 44 ( 7 ): 101 - 115 .

CHEN Jianming , LI Dingjian , ZENG Xiangjin , et al . Cross-modal optical information interaction and template dynamic update for RGBT target tracking method [J ] . Acta Optica Sinica , 2024 , 44 ( 7 ): 101 - 115 .

HU Weiming , WANG Qiang , ZHANG Li , et al . SiamMask: a framework for fast online object tracking and segmentation [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023 , 45 ( 3 ): 3072 - 3089 .

LIN T Y , GOYAL P , GIRSHICK R , et al . Focal loss for dense object detection [C ] // 2017 IEEE International Conference on Computer Vision (ICCV) . Piscataway, NJ, USA : IEEE , 2017 : 2999 - 3007 .

LI Xiang , WANG Wenhai , WU Lijun , et al . Generalized focal loss: learning qualified and distributed bounding boxes for dense object detection [C ] // Proceedings of the 34th International Conference on Neural Information Processing Systems . Red Hook, NY, USA : Curran Associates Inc. , 2020 : 21002 - 21012 .

REZATOFIGHI H , TSOI N , GWAK J Y , et al . Generalized intersection over union: a metric and a loss for bounding box regression [C ] // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway, NJ, USA : IEEE , 2019 : 658 - 666 .

JADON S . A survey of loss functions for semantic segmentation [C ] // 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB) . Piscataway, NJ, USA : IEEE , 2020 : 1 - 7 .

DENG Jia , DONG Wei , SOCHER R , et al . ImageNet: a large-scale hierarchical image database [C ] // 2009 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway, NJ, USA : IEEE , 2009 : 248 - 255 .

LI Chenglong , HUI Cheng , HU Shiyi , et al . Learning collaborative sparse representation for grayscale-thermal tracking [J ] . IEEE Transactions on Image Processing , 2016 , 25 ( 12 ): 5743 - 5756 .

LI Chenglong , LIANG Xinyan , LU Yijuan , et al . RGB-T object tracking: benchmark and baseline [J ] . Pattern Recognition , 2019 , 96 : 106977 .

VALMADRE J , BERTINETTO L , HENRIQUES J , et al . End-to-end representation learning for correlation filter based tracking [C ] // 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway, NJ, USA : IEEE , 2017 : 5000 - 5008 .

JUNG I , SON J , BAEK M , et al . Real-time MDNet [C ] // Computer Vision-ECCV 2018 . Cham, Switzerland : Springer International Publishing , 2018 : 89 - 104 .

NAM H , HAN B . Learning multi-domain convolutional neural networks for visual tracking [C ] // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway, NJ, USA : IEEE , 2016 : 4293 - 4302 .

ZHU Yabin , LI Chenglong , LUO Bin , et al . Dense feature aggregation and pruning for RGBT tracking [C ] // Proceedings of the 27th ACM International Conference on Multimedia . New York, NY, USA : ACM , 2019 : 465 - 472 .

ZHANG Hui , ZHANG Lei , ZHUO Li , et al . Object tracking in RGB-T videos using modal-aware attention network and competitive learning [J ] . Sensors , 2020 , 20 ( 2 ): 393 .

GAO Yuan , LI Chenglong , ZHU Yabin , et al . Deep adaptive fusion network for high performance RGBT tracking [C ] // 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) . Piscataway, NJ, USA : IEEE , 2019 : 91 - 99 .

LU Andong , LI Chenglong , YAN Yuqing , et al . RGBT tracking via multi-adapter network with hierarchical divergence loss [J ] . IEEE Transactions on Image Processing , 2021 , 30 : 5613 - 5625 .

XIAO Yun , YANG Mengmeng , LI Chenglong , et al . Attribute-based progressive fusion network for rgbt tracking [C ] // Proceedings of the AAAI Conference on Artificial Intelligence . Palo Alto, CA, USA : AAAI Press , 2022 : 2831 - 2838 .

LU Andong , QIAN Cun , LI Chenglong , et al . Duality-gated mutual condition network for RGBT tracking [J ] . IEEE Transactions on Neural Networks and Learning Systems , 2025 , 36 ( 3 ): 4118 - 4131 .

LI Chenglong , ZHAO Nan , LU Nan , et al . Weighted sparse representation regularized graph learning for RGB-T object tracking [C ] // Proceedings of the 25th ACM International Conference on Multimedia . New York, NY, USA : ACM , 2017 : 1856 - 1864 .

TANG Ming , FENG Jiayi . Multi-kernel correlation filter for visual tracking [C ] // 2015 IEEE International Conference on Computer Vision (ICCV) . Piscataway, NJ, USA : IEEE , 2015 : 3038 - 3046 .

HOU Xiaojun , XING Jiazheng , QIAN Yijie , et al . SDSTrack: self-distillation symmetric adapter learning for multi-modal visual object tracking [C ] // 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway, NJ, USA : IEEE , 2024 : 26541 - 26551 .

SUN Dengdi , PAN Yajie , LU Andong , et al . Transformer RGBT tracking with spatio-temporal multimodal tokens [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2024 , 34 ( 11 ): 12059 - 12072 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

采用序列特征与知识引导的未知网络应用早期识别方法

采用机器学习的大规模并行程序I/O性能建模与预测方法

超临界水异常区域热力学特性研究

隧道内燃料电池汽车氢气泄漏扩散特性及影响因素

水蒸气掺混对氨裂解气旋流燃烧与排放特性的影响研究