[1]张富强,曾 夏,白筠妍,等.多模态数据融合的加工作业动态手势识别方法[J].郑州大学学报(工学版),2024,45(05):30-36.[doi:10.13705/j.issn.1671-6833.2024.02.007]
　ZHANG Fuqiang,ZENG Xia,BAI Junyan,et al.Dynamic Gesture Recognition Method for Machining Operations Based on Multi-modalData Fusion[J].Journal of Zhengzhou University (Engineering Science),2024,45(05):30-36.[doi:10.13705/j.issn.1671-6833.2024.02.007]
点击复制

多模态数据融合的加工作业动态手势识别方法()
分享到：

《郑州大学学报(工学版)》[ISSN:1671-6833/CN:41-1339/T]

卷:
45
期数:
2024年05期
页码:
30-36
栏目:
出版日期:
2024-08-08

文章信息/Info

Title:
Dynamic Gesture Recognition Method for Machining Operations Based on Multi-modalData Fusion
文章编号:
1671-6833(2024)05-0030-07
作者:
张富强1; 2;  曾 夏1; 2;  白筠妍1; 2;  丁 凯1; 2
1. 长安大学 道路施工技术与装备教育部重点实验室,陕西 西安 710064;2. 长安大学 智能制造系统研究所,陕西西安 710064
Author(s):
ZHANG Fuqiang1; 2;  ZENG Xia1; 2;  BAI Junyan1; 2;  DING Kai1; 2
1. Key Laboratory of Road Construction Technology and Equipment of MOE, Changan University, Xian 710064, China; 2. Instituteof Smart Manufacturing Systems, Changan University, Xian 710064, China
关键词:
多模态数据融合;  加工作业;  动态手势识别;  C3D;  Mish 激活函数;  人机交互
Keywords:
multi-modal data fusion;  machining operation;  dynamic gesture recognition;  C3D;  Mish activationfunction;  human-computer interaction
分类号:
TH166
DOI:
10.13705/j.issn.1671-6833.2024.02.007
文献标志码:
A
摘要:
为了解决单模态数据所提供的特征信息缺乏而导致的识别准确率难以提高、模型鲁棒性较低等问题,提出了面向人机交互的加工作业多模态数据融合动态手势识别策略。 首先,采用 C3D 网络模型并在视频的空间维度和时间维度对深度图像和彩色图像两种模态数据进行特征提取;其次,将两种模态数据识别结果在决策层按最大值规则进行融合,同时,将原模型使用的 Relu 激活函数替换为 Mish 激活函数优化梯度特性;最后,通过 3 组对比实验得到 6 种动态手势的平均识别准确率为 96. 8%。 结果表明:所提方法实现了加工作业中动态手势识别的高准确率和高鲁棒性的目标,对人机交互技术在实际生产场景中的应用起到推动作用。
Abstract:
In order to solve the problem of difficulty in improving the recognition accuray and the low robustness ofthe model caused by the lack of feature information provided by single mode data, a dynamic gesture recognitionstrategy based on multi-modal data fusion of machining operations for human-computer interaction was proposed.Firstly, the C3D network model was used to extract features from the depth image and color image modal data basedon the spatial and temporal dimensions of videos. Secondly, the recognition results of the two modal data were fusedaccording to the maximum principle at the decision-making level. Meanwhile, the Relu activation function used inthe original model was replaced by Mish activation function to optimize the gradient update effect. Finally, throughthree sets of comparative experiments, it was found that the average recognition accuracy of six dynamic gesturesreached 96. 8%. The results showed that the proposed method achieved the goal of high accuracy and high robustness of dynamic gesture recognition in machining operation, which would play a role in promoting the application ofhuman-computer interaction technology in actual production scenes.

参考文献/References:

[1] 李浩, 刘根, 文笑雨, 等. 面向人机交互的数字孪生系统工业安全控制体系与关键技术[ J] . 计算机集成制造系统, 2021, 27(2) : 374-389.

LI H, LIU G, WEN X Y, et al. Industrial safety controlsystem and key technologies of digital twin system oriented to human-machine interaction[ J] . Computer Integrated Manufacturing Systems, 2021, 27(2) : 374-389.
[2] LIU N J, LOVELL B C, KOOTSOOKOS P J, et al.Model structure selection & training algorithms for anHMM gesture recognition system[C]∥Ninth InternationalWorkshop on Frontiers in Handwriting Recognition. Piscataway:IEEE, 2004: 100-105.
[3] HARTMANN B, LINK N. Gesture recognition with inertial sensors and optimized DTW prototypes [ C ] ∥2010IEEE International Conference on Systems, Man and Cybernetics. Piscataway: IEEE, 2010: 2102-2109.
[4] 李浩, 杨森林, 张晓丽. 基于机器视觉的火车驾驶员动态手势识别方法[ J] . 传感器与微系统, 2021, 40(2) : 34-37, 43.
LI H, YANG S L, ZHANG X L. Dynamic gesture recognition method of train driver based on machine vision[ J] .Transducer and Microsystem Technologies, 2021, 40(2) : 34-37, 43.
[5] JIA L S, ZHOU X Z, XUE C Q. Non-trajectory-basedgesture recognition in human-computer interaction basedon hand skeleton data[ J] . Multimedia Tools and Applications, 2022, 81(15) : 20509-20539.
[6] 彭金柱, 董梦超, 杨扬. 基于视觉和肌电信息融合的手势识别方法 [ J] . 郑 州 大 学 学 报 ( 工 学 版) , 2021,42(2) : 67-73.
PENG J Z, DONG M C, YANG Y. Human gesture recognition method based on vision and EMG signal information [ J] . Journal of Zhengzhou University ( EngineeringScience) , 2021, 42(2) : 67-73.
[7] CHAKRAVARTHI S S, RAO B N K, CHALLA A P, etal. Gesture recognition for enhancing human computer interaction[ J] . Journal of Scientific & Industrial Research,2023, 82(4) : 438-443.
[8] CHEOK M J, OMAR Z, JAWARD M H. A review ofhand gesture and sign language recognition techniques[ J] . International Journal of Machine Learning and Cybernetics, 2019, 10(1) : 131-153.
[9] MUJAHID A, AWAN M J, YASIN A, et al. Real-timehand gesture recognition based on deep learning YOLOv3model[ J] . Applied Sciences, 2021, 11(9) : 4164.
[10] ADITHYA A, RAJESH R. A deep convolutional neuralnetwork approach for static hand gesture recognition[ J] .Procedia Computer Science, 2020, 171: 2353-2361.
[11] SHARMA S, SINGH S. Vision-based hand gesture recognition using deep learning for the interpretation of signlanguage[ J] . Expert Systems with Applications, 2021,182: 115657.
[12] 刘杰, 王月, 田明. 多尺度时空特征融合的动态手势识别网络[ J] . 电子与信息学报, 2023, 45(7) : 2614-2622.
LIU J, WANG Y, TIAN M. Dynamic gesture recognitionnetwork based on multiscale spatiotemporal feature fusion[ J] . Journal of Electronics & Information Technology,2023, 45(7) : 2614-2622.
[13] SINGH D K. 3D-CNN based dynamic gesture recognitionfor Indian sign language modeling[ J] . Procedia Computer Science, 2021, 189: 76-83.
[14] ZHANG W J, WANG J C. Dynamic hand gesture recognition based on 3D convolutional neural network models[C]∥2019 IEEE 16th International Conference on Networking, Sensing and Control ( ICNSC ) . Piscataway:IEEE, 2019: 224-229.
[15] YANG L C, CHEN J, ZHU W H. Dynamic hand gesturerecognition based on a leap motion controller and two-layer bidirectional recurrent neural network [ J ] . Sensors,2020, 20(7) : 2106.
[16] KHODABANDELOU G, JUNG P G, AMIRAT Y, et al.Attention-based gated recurrent unit for gesture recognition[ J] . IEEE Transactions on Automation Science andEngineering, 2021, 18(2) : 495-507.
[17] LI C K, LI S, GAO Y B, et al. A two-stream neural network for pose-based hand gesture recognition [ J] . IEEETransactions on Cognitive and Developmental Systems,2022, 14(4) : 1594-1603.
[18] PALANISAMY G, THANGASWAMY S S. An efficienthand gesture recognition based on optimal deep embeddedhybrid convolutional neural network-long short term memory network model [ J] . Concurrency and Computation:Practice and Experience, 2022, 34(21) : 1-15.
[19] 谷学静, 周自朋, 郭宇承, 等. 基于 CNN-LSTM 混合模型的动态手势识别方法[ J] . 计算机应用与软件,2021, 38(11) : 205-209.
GU X J, ZHOU Z P, GUO Y C, et al. Dynamic gesturerecognition method based on CNN-LSTM hybrid model[ J ] . Computer Applications and Software, 2021, 38(11) : 205-209.
[20] REHMAN M, AHMED F, KHAN M, et al. Dynamichand gesture recognition using 3D-CNN and LSTM networks[ J] . Computers Materials & Continua, 2021, 70(3) : 4675-4690.
[21] MO D H, TIEN C L, YEH Y L, et al. Design of digitaltwin human-machine interface sensor with intelligent fingergesture recognition[J]. Sensors, 2023, 23(7): 3509.
[22] JI S W, XU W, YANG M, et al. 3D convolutional neuralnetworks for human action recognition [ J] . IEEE Transactions on Pattern Analysis and Machine Intelligence,2013, 35(1) : 221-231.
[23] TRAN D, BOURDEV L, FERGUS R, et al. Learningspatiotemporal features with 3D convolutional networks[C]∥2015 IEEE International Conference on ComputerVision ( ICCV) . Piscataway:IEEE, 2015: 4489-4497.
[24] MERCIONI M A, HOLBAN S. Soft Clipping Mish—a novel activation function for deep learning[C]∥2021 4th International Conference on Information and Computer Technologies (ICICT). Piscataway: IEEE, 2021: 13-17.
[25] 杨婷. 基于深度学习的多模态情感识别[ D] . 南昌:南昌大学, 2021.
YANG T. Multimodal emotion recognition based on deeplearning[D] . Nanchang: Nanchang University, 2021.

更新日期/Last Update: 2024-09-02