An approach based on deep learning for Indian sign language translation

作     者：Kinjal Bhargavkumar Mistree Devendra Thakor Brijesh Bhatt 

作者机构：Chhotubhai Gopalbhai Patel Institute of TechnologyUka Tarsadia UniversityBardoliIndia Dharmsinh Desai Institute of TechnologyDharmsinh Desai UniversityNadiadIndia 

出 版 物：《International Journal of Intelligent Computing and Cybernetics》 (智能计算与控制论国际期刊（英文）)

年 卷 期：2023年第16卷第3期

页      面：397-419页

核心收录：

学科分类：0502[文学-外国语言文学] 050201[文学-英语语言文学] 05[文学] 

基　　金：Ethical statement: All participants (parent and legal guardian in the case of children under 16) gave written informed consent to participate in the study. Consent was given for publication by all participants (parent and legal guardian in the case of children under 16). Plain language summary: The work described in this paper is an important step forward in this domain as we created (1) a dataset of 47 880 ISL videos of 13 720 ISL sentences and (2) an ISL gloss corpus from the Brown corpus to promote research in ISL gloss to translation in text and speech of natural languages 

主　　题：Indian sign language Neural machine translation ISL corpus Pretrained models Sign language recognition Sign language translation Paper type Research paper 

摘      要：Purpose–According to the Indian Sign Language Research and Training Centre(ISLRTC),India has approximately 300 certified human interpreters to help people with hearing *** paper aims to address the issue of Indian Sign Language(ISL)sentence recognition and translation into semantically equivalent English text in a signer-independent ***/methodology/approach–This study presents an approach that translates ISL sentences into English text using the MobileNetV2 model and Neural Machine Translation(NMT).The authors have created an ISL corpus from the Brown corpus using ISL grammar rules to perform machine *** authors’approach converts ISL videos of the newly created dataset into ISL gloss sequences using the MobileNetV2 model and the recognized ISL gloss sequence is then fed to a machine translation module that generates an English sentence for each ISL ***–As per the experimental results,pretrained MobileNetV2 model was proven the best-suited model for the recognition of ISL sentences and NMT provided better results than Statistical Machine Translation(SMT)to convert ISL text into English *** automatic and human evaluation of the proposed approach yielded accuracies of 83.3 and 86.1%,*** limitations/implications–It can be seen that the neural machine translation systems produced translations with repetitions of other translated words,strange translations when the total number of words per sentence is increased and one or more unexpected terms that had no relation to the source text on *** most common type of error is the mistranslation of places,numbers and *** this has little effect on the overall structure of the translated sentence,it indicates that the embedding learned for these few words could be ***/value–Sign language recognition and translation is a crucial step toward improving communication between the deaf and the rest of *** of the shortage of hum