A multi-modal clustering method for traditonal Chinese medicine clinical data via media convergence

作     者：Jingna Si Ziwei Tian Dongmei Li Lei Zhang Lei Yao Wenjuan Jiang Jia Liu Runshun Zhang Xiaoping Zhang 

作者机构：State Key Laboratory of Tree Genetics and BreedingCollege of Biological Sciences and TechnologyBeijing Forestry UniversityBeijingChina Department of Comparative Plant and Fungal BiologyWellcome Trust Millennium BuildingArdinglyUK School of Information Science and TechnologyBeijing Forestry UniversityBeijingChina Engineering Research Center for Forestry‐oriented Intelligent Information ProcessingNational Forestry and Grassland AdministrationBeijingChina National Data Center of Traditional Chinese MedicineChina Academy of Chinese Medical SciencesBeijingChina University of Wisconsin‐MilwaukeeMilwaukeeWisconsinUSA Experimental Research CenterChina Academy of Chinese Medical SciencesBeijingChina Guang'anmen HospitalChina Academy of Chinese Medical SciencesBeijingChina 

出 版 物：《CAAI Transactions on Intelligence Technology》 (智能技术学报（英文）)

年 卷 期：2023年第8卷第2期

页      面：390-400页

核心收录：

学科分类：0710[理学-生物学] 1002[医学-临床医学] 1001[医学-基础医学(可授医学、理学学位)] 08[工学] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：China Academy of Chinese Medical Sciences Grant/Award Number:CI2021A00512。 

主　　题：graph convolutional encoder media convergence multi-modal clustering traditional Chinese medicine 

摘      要：Media convergence is a media change led by technological innovation.Applying media convergence technology to the study of clustering in Chinese medicine can significantly exploit the advantages of media fusion.Obtaining consistent and complementary information among multiple modalities through media convergence can provide technical support for clustering.This article presents an approach based on Media Convergence and Graph convolution Encoder Clustering(MCGEC)for traditonal Chinese medicine(TCM)clinical data.It feeds modal information and graph structure from media information into a multi-modal graph convolution encoder to obtain the media feature representation learnt from multiple modalities.MCGEC captures latent information from various modalities by fusion and optimises the feature representations and network architecture with learnt clustering labels.The experiment is conducted on real-world multimodal TCM clinical data,including information like images and text.MCGEC has improved clustering results compared to the generic single-modal clustering methods and the current more advanced multi-modal clustering methods.MCGEC applied to TCM clinical datasets can achieve better results.Integrating multimedia features into clustering algorithms offers significant benefits compared to single-modal clustering approaches that simply concatenate features from different modalities.It provides practical technical support for multi-modal clustering in the TCM field incorporating multimedia features.