Steady-State Motion Visual Evoked Potential Decoding Method Based on Integration of Transformer Network and Convolutional Neural Network

ZHANG Huanqing; XIE Jun; YANG Hanlin; DU Fangzhao; JIN Zhiwei; CHEN Yujie

doi:10.7652/xjtuxb202505005

您当前的位置：

首页 >

文章列表页 >

Steady-State Motion Visual Evoked Potential Decoding Method Based on Integration of Transformer Network and Convolutional Neural Network

Special Topic　Medical-Engineering Intersection and Human-Computer Interaction | 更新时间：2025-08-26

- Steady-State Motion Visual Evoked Potential Decoding Method Based on Integration of Transformer Network and Convolutional Neural Network
- Journal of Xi’an Jiaotong University Vol. 59, Issue 5, Pages: 44-53(2025)
- 作者机构：
  
  西安交通大学机械工程学院， 710049，西安
- 作者简介：
- 基金信息：
- DOI：10.7652/xjtuxb202505005
  CLC： R318
- Received：25 August 2024，
  
  Online First：13 January 2025，
  
  Published：10 May 2025
- 稿件说明：
移动端阅览
ZHANG Huanqing, XIE Jun, YANG Hanlin, et al. Steady-State Motion Visual Evoked Potential Decoding Method Based on Integration of Transformer Network and Convolutional Neural Network[J]. Journal of Xi’an Jiaotong University, 2025, 59(5): 44-53.
DOI：

ZHANG Huanqing, XIE Jun, YANG Hanlin, et al. Steady-State Motion Visual Evoked Potential Decoding Method Based on Integration of Transformer Network and Convolutional Neural Network[J]. Journal of Xi’an Jiaotong University, 2025, 59(5): 44-53. DOI： 10.7652/xjtuxb202505005.

摘要

针对卷积神经网络（CNN）在感受野有限、缺乏对全局信息的有效感知，以及在处理短时稳态运动视觉诱发电位（SSMVEP）信号时分类效果欠佳的问题，提出了一种紧凑EEGNet-Transformer（即EEGNetformer）网络。EEGNetformer网络融合了为脑电（EEG）信号识别任务而设计的通用的卷积神经网络EEGNet网络和Transformer网络的优势，有效地捕捉与处理脑电信号中的局部和全局信息，增强网络对SSMVEP特征的学习，进而实现良好的解码性能。EEGNet网络用于提取SSMVEP的局部时间和空间特征，而Transformer网络用于捕捉脑电时间序列的全局信息。在基于SSMVEP-BCI范式采集的数据基础上，开展了实验以评估EEGNetformer网络的性能。实验结果显示，当在2 s SSMVEP数据条件下，EEGNetformer网络在基于被试者内情况的平均准确率为88.9%±6.6%，在基于跨被试者情况的平均准确率为69.1%±4.3%。与传统的CNN算法相比，EEGNetformer网络的分类性能提升了4.2%~17.4%。研究内容说明，EEGNetformer网络在有效提高SSMVEP-BCI识别准确率方面具有显著优势，为进一步提升SSMVEP-BCI解码性能提供了新的研究思路。

Abstract

Addressing the issues of limited receptive fields and ineffective perception of global information in convolutional neural networks (CNNs)

as well as their suboptimal classification performance when processing short-duration steady-state motion visual evoked potentials (SSMVEP) signals

a compact EEGNet-Transformer (i.e

EEGNetformer) network is proposed. EEGNetformer network integrates the strengths of the EEGNet network (a general-purpose CNN designed for EEG recognition tasks) and the Transformer network. It efficiently captures and processes both local and global information in the electroencephalogram (EEG) signals

enhancing the network's learning of the SSMVEP features

thereby achieving an excellent decoding performance. The EEGNet network is utilized to extracts local temporal and spatial features of SSMVEP

while the Transformer network is utilized to capture global information in EEG time series. Based on data collected using the SSMVEP-BCI paradigm

experiments were conducted to evaluate the performance of the EEGNetformer network. The results showed that under the condition of 2 s SSMVEP data

EEGNetformer achieved an average accuracy of 88.9%±6.6% for within-subject conditions and 69.1%±4.3% for cross-subject conditions. Compared with traditional CNN algorithms

the classification performance of the EEGNetformer network was improved by 4.2%—17.4%. These findings demonstrate the significant advantage of EEGNetformer network in enhancing the recognition accuracy of SSMVEP-BCI systems

providing a new research direction for further enhancing SSMVEP-BCI decoding performance.

关键词

Keywords

references

WOLPAW J R . Brain-computer interfaces [M ] . Amsterdam, Holland : Elsevier , 2013 : 67 - 74 .

姜耿 , 赵春临 . 基于EEG的脑机接口发展综述 [J ] . 计算机测量与控制 , 2022 , 30 ( 7 ): 1 - 8 .

JIANG Geng , ZHAO Chunlin . A review of the development of brain-computer interface based on EEG [J ] . Computer Measurement & Control , 2022 , 30 ( 7 ): 1 - 8 .

VÄRBU K , MUHAMMAD N , MUHAMMAD Y . Past, present, and future of EEG-based BCI applications [J ] . Sensors , 2022 , 22 ( 9 ): 3331 .

BONDRE C , KAPGATE D . Brain computer interfaces using SSVEP: an overview [J ] . International Journal of Engineering & Technology , 2014 , 1 : 9 - 17 .

邱爽 , 杨帮华 , 陈小刚 , 等 . 非侵入式脑-机接口编解码技术研究进展 [J ] . 中国图象图形学报 , 2023 , 28 ( 6 ): 1543 - 1566 .

QIU Shuang , YANG Banghua , CHEN Xiaogang , et al . A survey on encoding and decoding technology of non-invasive brain-computer interface [J ] . Journal of Image and Graphics , 2023 , 28 ( 6 ): 1543 - 1566 .

KAPGATE D , KALBANDE D . A review on visual brain computer interface [C ] // Advancements of Medical Electronics . New Delhi : Springer India , 2015 : 193 - 206 .

梁栗炎 , 孔姝懿 , 张倩 , 等 . 脑机接口的范式、算法与编解码概念探讨与研究 [J ] . 信息通信技术与政策 , 2024 , 50 ( 5 ): 61 - 70 .

LIANG Liyan , KONG Shuyi , ZHANG Qian , et al . Exploration and research on paradigm, algorithm, as well as encoding and decoding concepts of brain-computer interface [J ] . Information and Communications Technology and Policy , 2024 , 50 ( 5 ): 61 - 70 .

李丽 , 陈枭宇 , 随力 . 稳态运动视觉诱发电位的诱发及在脑机接口中的应用进展 [J ] . 上海理工大学学报 , 2022 , 44 ( 1 ): 27 - 33 .

LI Li , CHEN Xiaoyu , SUI Li . Induction of steady-state motion visual evoked potential and its application in brain-computer interface [J ] . Journal of University of Shanghai for Science and Technology , 2022 , 44 ( 1 ): 27 - 33 .

XIE Jun , XU Guanghua , WANG Jing , et al . Steady-state motion visual evoked potentials produced by oscillating Newton's rings: implications for brain-computer interfaces [J ] . PLoS One , 2012 , 7 ( 6 ): e39707 .

刘贵彤 , 张志敏 , 柴晓珂 , 等 . 基于SSMVEP的脑-机接口视觉刺激探究 [J ] . 中国医疗器械杂志 , 2018 , 42 ( 5 ): 313 - 316 .

LIU Guitong , ZHANG Zhimin , CHAI Xiaoke , et al . Study of steady state motion visual evoked potential-based visual stimulation of BCI system [J ] . Chinese Journal of Medical Instrumentation , 2018 , 42 ( 5 ): 313 - 316 .

LIN Zhonglin , ZHANG Changshui , WU Wei , et al . Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs [J ] . IEEE Transactions on Biomedical Engineering , 2006 , 53 ( 12 ): 2610 - 2614 .

CHEN Xiaogang , WANG Yijun , GAO Shangkai , et al . Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain-computer interface [J ] . Journal of Neural Engineering , 2015 , 12 ( 4 ): 046008 .

ZHANG Yue , XIE S Q , WANG He , et al . Data analytics in steady-state visual evoked potential-based brain-computer interface: a review [J ] . IEEE Sensors Journal , 2021 , 21 ( 2 ): 1124 - 1138 .

CRAIK A , HE Yongtian , CONTRERAS-VIDAL J L . Deep learning for electroencephalogram (EEG) classification tasks: a review [J ] . Journal of Neural Engineering , 2019 , 16 ( 3 ): 031001 .

XU Dongcen , TANG Fengzhen , LI Yiping , et al . An analysis of deep learning models in SSVEP-based BCI: a survey [J ] . Brain Sciences , 2023 , 13 ( 3 ): 483 .

LAWHERN V J , SOLON A J , WAYTOWICH N R , et al . EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces [J ] . Journal of Neural Engineering , 2018 , 15 ( 5 ): 056013 .

RAVI A , BENI N H , MANUEL J , et al . Comparing user-dependent and user-independent training of CNN for SSVEP BCI [J ] . Journal of Neural Engineering , 2020 , 17 ( 2 ): 026028 .

DING Wenlong , SHAN Jianhua , FANG Bin , et al . Filter bank convolutional neural network for short time-window steady-state visual evoked potential classification [J ] . IEEE Transactions on Neural Systems and Rehabilitation Engineering , 2021 , 29 : 2615 - 2624 .

PENG Zhiliang , HUANG Wei , GU Shanzhi , et al . Conformer: local features coupling global representations for visual recognition [C ] // 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway, NJ, USA : IEEE , 2021 : 357 - 366 .

HAN Kai , WANG Yunhe , CHEN Hanting , et al . A survey on vision transformer [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023 , 45 ( 1 ): 87 - 110 .

HAN Kai , XIAO An , WU Enhua , et al . Transformer in transformer [C ] // Advances in Neural Information Processing Systems . Red Hook, NY, USA : Curran Associates, Inc , 2021 : 15908 - 15919 .

MAURÍCIO J , DOMINGUES I , BERNARDINO J . Comparing vision transformers and convolutional neural networks for image classification: a literature review [J ] . Applied Sciences , 2023 , 13 ( 9 ): 5521 .

XIE Jin , ZHANG Jie , SUN Jiayao , et al . A transformer-based approach combining deep learning network and spatial-temporal information for raw EEG classification [J ] . IEEE Transactions on Neural Systems and Rehabilitation Engineering , 2022 , 30 : 2126 - 2136 .

SONG Yonghao , JIA Xueyu , YANG Lie , et al . Transformer-based spatial-temporal feature learning for EEG decoding [EB/OL ] . ( 2021-06-11 ) [ 2024-08-02 ] . https://arxiv.org/abs/2106.11170 https://arxiv.org/abs/2106.11170 .

GUO Jiayi , CAI Qing , AN Jianpeng , et al . A Transformer based neural network for emotion recognition and visualizations of crucial EEG channels [J ] . Physica: A Statistical Mechanics and its Applications , 2022 , 603 : 127700 .

戴洋毅 , 何康 , 瑚琦 , 等 . CNN-Transformer混合模型在计算机视觉领域的研究综述 [J ] . 建模与仿真 , 2023 , 12 ( 4 ): 3657 - 3672 .

DAI Yangyi , HE Kang , HU Qi , et al . Review of CNN-Transformer hybrid model in computer vision [J ] . Modeling and Simulation , 2023 , 12 ( 4 ): 3657 - 3672 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Research on the Electrohydrodynamic Effects of Ionic Liquid-Modified Fuels

Integrated and Joint Design Optimization of Centrifugal Compressor Impeller and Casing Treatment at Multiple Working Points

Diffusion Characteristics and Influencing Factors in Case of Hydrogen Leakage from Fuel Cell Vehicles Running in Tunnels

Effects of Expansion Angle Crater Film Hole on the Film Cooling Characteristics of a Double-Wall Structure

Numerical Simulation of the Energy Conversion Process in Thermoacoustic Engines with Different Stack Materials

Related Author

No data

Related Institution

No data

⁰