Research on Facial Expression Capture Based on Two-Stage Neural Network

作     者：Zhenzhou Wang Shao Cui Xiang Wang JiaFeng Tian 

作者机构：School of Information Science and EngineeringHebei University of Science and TechnologyShijiazhuang050000China School of EngineeringNewcastle UniversityNewcastle Upon TyneNE98United Kingdom 

出 版 物：《Computers, Materials & Continua》 (计算机、材料和连续体（英文）)

年 卷 期：2022年第72卷第9期

页      面：4709-4725页

核心收录：

学科分类：08[工学] 0805[工学-材料科学与工程（可授工学、理学学位）] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：This research was funded by College Student Innovation and Entrepreneurship Training Program,grant number 2021055Z and S202110082031 the Special Project for Cultivating Scientific and Technological Innovation Ability of College and Middle School Students in Hebei Province,Grant Number 2021H011404. 

主　　题：Facial expression capture facial landmarks multi-task cascaded convolutional networks high-resolution network animation generation 

摘      要：To generate realistic three-dimensional animation of virtual character,capturing real facial expression is the primary task.Due to diverse facial expressions and complex background,facial landmarks recognized by existing strategies have the problem of deviations and low accuracy.Therefore,a method for facial expression capture based on two-stage neural network is proposed in this paper which takes advantage of improved multi-task cascaded convolutional networks(MTCNN)and high-resolution network.Firstly,the convolution operation of traditional MTCNN is improved.The face information in the input image is quickly filtered by feature fusion in the first stage and Octave Convolution instead of the original ones is introduced into in the second stage to enhance the feature extraction ability of the network,which further rejects a large number of false candidates.The model outputs more accurate facial candidate windows for better landmarks recognition and locates the faces.Then the images cropped after face detection are input into high-resolution network.Multi-scale feature fusion is realized by parallel connection of multi-resolution streams,and rich high-resolution heatmaps of facial landmarks are obtained.Finally,the changes of facial landmarks recognized are tracked in real-time.The expression parameters are extracted and transmitted to Unity3D engine to drive the virtual character’s face,which can realize facial expression synchronous animation.Extensive experimental results obtained on the WFLW database demonstrate the superiority of the proposed method in terms of accuracy and robustness,especially for diverse expressions and complex background.The method can accurately capture facial expression and generate three-dimensional animation effects,making online entertainment and social interaction more immersive in shared virtual space.