Construction of Human Digital Twin Model Based on Multimodal Data and Its Application in Locomotion Mode Identifcation

作     者：Ruirui Zhong Bingtao Hu Yixiong Feng Hao Zheng Zhaoxi Hong Shanhe Lou Jianrong Tan Ruirui Zhong;Bingtao Hu;Yixiong Feng;Hao Zheng;Zhaoxi Hong;Shanhe Lou;Jianrong Tan

作者机构：State Key Laboratory of Fluid Power and Mechatronic SystemsZhejiang UniversityHangzhou 310027China Engineering Research Center for Design Engineering and Digital Twin of Zhejiang ProvinceHangzhou 310027China Hangzhou Innovation InstituteBeihang UniversityHangzhou 310052China School of Mechanical and Aerospace EngineeringNanyang Technological UniversitySingapore 637460Singapore 

出 版 物：《Chinese Journal of Mechanical Engineering》 (中国机械工程学报（英文版）)

年 卷 期：2023年第36卷第5期

页      面：7-19页

核心收录：

学科分类：12[管理学] 1201[管理学-管理科学与工程(可授管理学、工学学位)] 081104[工学-模式识别与智能系统] 08[工学] 0835[工学-软件工程] 0811[工学-控制科学与工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：Supported by National Natural Science Foundation of China(Grant Nos.52205288,52130501,52075479) Zhejiang Provincial Key Research&Development Program(Grant No.2021C01110) 

主　　题：Human digital twin Human-cyber-physical system Bidirectional long short-term memory Convolutional neural network Multimodal data 

摘      要：With the increasing attention to the state and role of people in intelligent manufacturing, there is a strong demand for human-cyber-physical systems (HCPS) that focus on human-robot interaction. The existing intelligent manufacturing system cannot satisfy efcient human-robot collaborative work. However, unlike machines equipped with sensors, human characteristic information is difcult to be perceived and digitized instantly. In view of the high complexity and uncertainty of the human body, this paper proposes a framework for building a human digital twin (HDT) model based on multimodal data and expounds on the key technologies. Data acquisition system is built to dynamically acquire and update the body state data and physiological data of the human body and realize the digital expression of multi-source heterogeneous human body information. A bidirectional long short-term memory and convolutional neural network (BiLSTM-CNN) based network is devised to fuse multimodal human data and extract the spatiotemporal features, and the human locomotion mode identifcation is taken as an application case. A series of optimization experiments are carried out to improve the performance of the proposed BiLSTM-CNN-based network model. The proposed model is compared with traditional locomotion mode identifcation models. The experimental results proved the superiority of the HDT framework for human locomotion mode identifcation.