Audiovisual speech recognition based on a deep convolutional neural network

作     者：Shashidhar Rudregowda Sudarshan Patilkulkarni Vinayakumar Ravi Gururaj H.L. Moez Krichen 

作者机构：Department of Electronics and Communication EngineeringJSS Science and Technology UniversityMysuru570006India Center for Artificial IntelligencePrince Mohammad Bin Fahd UniversityKhobar34754Saudi Arabia Department of Information TechnologyManipal Institute of Technology BengaluruManipal Academy of Higher EducationManipal560064India Department of Information TechnologyFaculty of Computer Science and Information Technology(FCSIT)Al-Baha UniversityAlaqiq65779-7738Saudi Arabia ReDCAD LaboratoryUniversity of SfaxSfax3038Tunisia 

出 版 物：《Data Science and Management》 (数据科学与管理（英文）)

年 卷 期：2024年第7卷第1期

页      面：25-34页

核心收录：

学科分类：08[工学] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

主　　题：Audiovisual speech recognition,Custom dataset 1D Convolution neural network(CNN) Deep CNN(DCNN) Long short-term memory(LSTM) Lipreading Dlib Mel-frequency cepstral coefficient(MFCC) 

摘      要：Audiovisual speech recognition is an emerging research *** is the recognition of what someone is saying using visual information,primarily lip *** this study,we created a custom dataset for Indian English linguistics and categorized it into three main categories:(1)audio recognition,(2)visual feature extraction,and(3)combined audio and visual *** features were extracted using the mel-frequency cepstral coefficient,and classification was performed using a one-dimension convolutional neural *** feature extraction uses Dlib and then classifies visual speech using a long short-term memory type of recurrent neural ***,integration was performed using a deep convolutional *** audio speech of Indian English was successfully recognized with accuracies of 93.67%and 91.53%,respectively,using testing data from 200 *** training accuracy for visual speech recognition using the Indian English dataset was 77.48%and the test accuracy was 76.19%using 60 *** integration,the accuracies of audiovisual speech recognition using the Indian English dataset for training and testing were 94.67%and 91.75%,respectively.