A Novel Action Transformer Network for Hybrid Multimodal Sign Language Recognition

作     者：Sameena Javaid Safdar Rizvi 

作者机构：Department of Computer SciencesSchool of Engineering and Applied SciencesBahria UniversityKarachi CampusKarachiPakistan 

出 版 物：《Computers, Materials & Continua》 (计算机、材料和连续体（英文）)

年 卷 期：2023年第74卷第1期

页      面：523-537页

核心收录：

学科分类：0710[理学-生物学] 080903[工学-微电子学与固体电子学] 0809[工学-电子科学与技术（可授工学、理学学位）] 08[工学] 0701[理学-数学] 0702[理学-物理学] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

主　　题：Sign language gesture recognition manual signs non-manual signs action transformer network 

摘      要：Sign language fills the communication gap for people with hearing and speaking *** includes both visual modalities,manual gestures consisting of movements of hands,and non-manual gestures incorporating body movements including head,facial expressions,eyes,shoulder shrugging,*** both gestures have been detected;identifying separately may have better accuracy,butmuch communicational information is *** sign language mechanism is needed to detect manual and non-manual gestures to convey the appropriate detailed message to *** novel proposed system contributes as Sign LanguageAction Transformer Network(SLATN),localizing hand,body,and facial gestures in video *** we are expending a Transformer-style structural design as a“base networkto extract features from a spatiotemporal *** impulsively learns to track individual persons and their action context inmultiple ***,a“head networkemphasizes hand movement and facial expression simultaneously,which is often crucial to understanding sign language,using its attention mechanism for creating tight bounding boxes around classified *** model’s work is later compared with the traditional identification methods of activity *** not only works faster but achieves better accuracy as *** achieves overall 82.66%testing accuracy with a very considerable performance of computation with 94.13 Giga-Floating Point Operations per Second(G-FLOPS).Another contribution is a newly created dataset of Pakistan Sign Language forManual and Non-Manual(PkSLMNM)gestures.