检索结果-南通市图书馆

multimodal fusion of Brain Imaging Data: Methods and Applications

Machine Intelligence Research 2024年第1期21卷 136-152页

作者： Na Luo Weiyang Shi Zhengyi Yang Ming Song Tianzi Jiang Brainnetome Center and National Laboratory of Pattern Recognition Institute of AutomationChinese Academy of SciencesBeijing 100190China School of Artificial Intelligence University of Chinese Academy of SciencesBeijing 100049China Center for Excellence in Brain Science and Intelligence Technology Institute of AutomationChinese Academy of SciencesBeijing 100190China Research Center for Augmented Intelligence Zhejiang LaboratoryHangzhou 311100China

Neuroimaging data typically include multiple modalities,such as structural or functional magnetic resonance imaging,dif-fusion tensor imaging,and positron emission tomography,which provide multiple views for observing and analyzing the brain.To lever-age the complementary representations of different modalities,multimodal fusion is consequently needed to dig out both inter-modality and intra-modality information.With the exploited rich information,it is becoming popular to combine multiple modality data to ex-plore the structural and functional characteristics of the brain in both health and disease status.In this paper,we first review a wide spectrum of advanced machine learning methodologies for fusing multimodal brain imaging data,broadly categorized into unsupervised and supervised learning strategies.Followed by this,some representative applications are discussed,including how they help to under-stand the brain arealization,how they improve the prediction of behavioral phenotypes and brain aging,and how they accelerate the biomarker exploration of brain diseases.Finally,we discuss some exciting emerging trends and important future directions.Collectively,we intend to offer a comprehensive overview of brain imaging fusion methods and their successful applications,along with the chal-lenges imposed by multi-scale and big data,which arises an urgent demand on developing new models and platforms.

关键词： multimodal fusion supervised learning unsupervised learning brain atlas cognition brain disorders

来源：

维普期刊数据库评论

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

multimodal fusion recognition for digital twin

引用

Digital Communications and Networks 2024年第2期10卷 337-346页

作者： Tianzhe Zhou Xuguang Zhang Bing Kang Mingkai Chen Key Laboratory of Broadband Wireless Communication and Sensor Network Technology Ministry of EducationNanjing University of Posts and TelecommunicationsNanjing210003Chin

The digital twin is the concept of transcending reality,which is the reverse feedback from the real physical space to the virtual digital space.People hold great prospects for this emerging technology.In order to realize the upgrading of the digital twin industrial chain,it is urgent to introduce more modalities,such as vision,haptics,hearing and smell,into the virtual digital space,which assists physical entities and virtual objects in creating a closer connection.Therefore,perceptual understanding and object recognition have become an urgent hot topic in the digital twin.Existing surface material classification schemes often achieve recognition through machine learning or deep learning in a single modality,ignoring the complementarity between multiple modalities.In order to overcome this dilemma,we propose a multimodal fusion network in our article that combines two modalities,visual and haptic,for surface material recognition.On the one hand,the network makes full use of the potential correlations between multiple modalities to deeply mine the modal semantics and complete the data mapping.On the other hand,the network is extensible and can be used as a universal architecture to include more modalities.Experiments show that the constructed multimodal fusion network can achieve 99.42%classification accuracy while reducing complexity.

关键词： Digital twin multimodal fusion Object recognition Deep learning Transfer learning

来源：

维普期刊数据库评论

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

MCSTransWnet:A new deep learning process for postoperative corneal topography prediction based on raw multimodal data from the Pentacam HR system

引用

Medicine in Novel Technology and Devices 2024年第1期21卷 53-63页

作者： Nan Cheng Zhe Zhang Jing Pan Xiao-Na Li Wei-Yi Chen Guang-Hua Zhang Wei-Hua Yang College of Biomedical Engineering Taiyuan University of TechnologyTaiyuan 030600ShanxiChina Shenzhen Eye Hospital Jinan UniversityShenzhen Eye InstituteShenzhen 518040GuangdongChina Department of Intelligence and Automation Taiyuan UniversityTaiyuan 030032ShanxiChina Graphics and Imaging Laboratory University of GironaGirona 17004Spain

This work provides a new multimodal fusion generative adversarial net(GAN)model,Multiple Conditions Transform W-net(MCSTransWnet),which primarily uses femtosecond laser arcuate keratotomy surgical parameters and preoperative corneal topography to predict postoperative corneal topography in astigmatism-corrected patients.The MCSTransWnet model comprises a generator and a discriminator,and the generator is composed of two sub-generators.The first sub-generator extracts features using the U-net model,vision transform(ViT)and a multi-parameter conditional module branch.The second sub-generator uses a U-net network for further image denoising.The discriminator uses the pixel discriminator in Pix2Pix.Currently,most GAN models are convolutional neural networks;however,due to their feature extraction locality,it is difficult to comprehend the relationships among global features.Thus,we added a vision Transform network as the model branch to extract the global features.It is normally difficult to train the transformer,and image noise and geometric information loss are likely.Hence,we adopted the standard U-net fusion scheme and transform network as the generator,so that global features,local features,and rich image details could be obtained simultaneously.Our experimental results clearly demonstrate that MCSTransWnet successfully predicts postoperative corneal topographies(structural similarity=0.765,peak signal-to-noise ratio=16.012,and Fréchet inception distance=9.264).Using this technique to obtain the rough shape of the postoperative corneal topography in advance gives clinicians more references and guides changes to surgical planning and improves the success rate of surgery.

关键词： Deep learning Generative adversarial networks Corneal topography Transformer W-net U-net Medical imaging multimodal fusion

来源：

维普期刊数据库评论

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

Cross-Modal Complementary Network with Hierarchical fusion for multimodal Sentiment Classification

引用

Tsinghua Science and Technology 2022年第4期27卷 664-679页

作者： Cheng Peng Chunxia Zhang Xiaojun Xue Jiameng Gao Hongjian Liang Zhengdong Niu School of Computer Science and Technology Beijing Institute of TechnologyBeijing 100081China School of Information Production and SystemsWaseda UniversityFukuoka 808-0135Japan

multimodal Sentiment Classification(MSC)uses multimodal data,such as images and texts,to identify the users'sentiment polarities from the information posted by users on the Internet.MSC has attracted considerable attention because of its wide applications in social computing and opinion mining.However,improper correlation strategies can cause erroneous fusion as the texts and the images that are unrelated to each other may integrate.Moreover,simply concatenating them modal by modal,even with true correlation,cannot fully capture the features within and between modals.To solve these problems,this paper proposes a Cross-Modal Complementary Network(CMCN)with hierarchical fusion for MSC.The CMCN is designed as a hierarchical structure with three key modules,namely,the feature extraction module to extract features from texts and images,the feature attention module to learn both text and image attention features generated by an image-text correlation generator,and the cross-modal hierarchical fusion module to fuse features within and between modals.Such a CMCN provides a hierarchical fusion framework that can fully integrate different modal features and helps reduce the risk of integrating unrelated modal features.Extensive experimental results on three public datasets show that the proposed approach significantly outperforms the state-of-the-art methods.

关键词： multimodal sentiment analysis multimodal fusion Cross-Modal Complementary Network(CMCN) hierarchical fusion joint optimization

来源：

维普期刊数据库

同方期刊数据库评论

在线全文

学校读者我要写书评

暂无评论

MFF-Net: multimodal Feature fusion Network for 3D Object Detection

引用

Computers, Materials & Continua 2023年第6期75卷 5615-5637页

作者： Peicheng Shi Zhiqiang Liu Heng Qi Aixi Yang School of Mechanical Engineering Anhui Polytechnic UniversityWuhu241000Anhui ProvinceChina School of Mechanical Polytechnic Institute of Zhejiang UniversityHangzhou310000Zhejiang ProvinceChina

In complex traffic environment scenarios,it is very important for autonomous vehicles to accurately perceive the dynamic information of other vehicles around the vehicle in advance.The accuracy of 3D object detection will be affected by problems such as illumination changes,object occlusion,and object detection distance.To this purpose,we face these challenges by proposing a multimodal feature fusion network for 3D object detection(MFF-Net).In this research,this paper first uses the spatial transformation projection algorithm to map the image features into the feature space,so that the image features are in the same spatial dimension when fused with the point cloud features.Then,feature channel weighting is performed using an adaptive expression augmentation fusion network to enhance important network features,suppress useless features,and increase the directionality of the network to features.Finally,this paper increases the probability of false detection and missed detection in the non-maximum suppression algo-rithm by increasing the one-dimensional threshold.So far,this paper has constructed a complete 3D target detection network based on multimodal feature fusion.The experimental results show that the proposed achieves an average accuracy of 82.60%on the Karlsruhe Institute of Technology and Toyota Technological Institute(KITTI)dataset,outperforming previous state-of-the-art multimodal fusion networks.In Easy,Moderate,and hard evaluation indicators,the accuracy rate of this paper reaches 90.96%,81.46%,and 75.39%.This shows that the MFF-Net network has good performance in 3D object detection.

关键词： 3D object detection multimodal fusion neural network autonomous driving attention mechanism

来源：

维普期刊数据库评论

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

Multi-level fusion with deep neural networks for multimodal sentiment classification

引用

The Journal of China Universities of Posts and Telecommunications 2022年第3期29卷 25-33页

作者： Zhang Guangwei Zhao Bing Li Ruifan School of Computer Seciences Beijing University of Posts and CommunicationsBeijing 100876China School of Science Yanshan UniversityQinhuangdao 066004China School of Artificial Intelligence Beijing University of Posts and CommunicationsBeijing 100876China

The task of multimodal sentiment classification aims to associate multimodal information, such as images and texts with appropriate sentiment polarities. There are various levels that can affect human sentiment in visual and textual modalities. However, most existing methods treat various levels of features independently without having effective method for feature fusion. In this paper, we propose a multi-level fusion classification(MFC) model to predict the sentiment polarity based on the fusing features from different levels by exploiting the dependency among them. The proposed architecture leverages convolutional neural networks(CNNs) with multiple layers to extract levels of features in image and text modalities. Considering the dependencies within the low-level and high-level features, a bi-directional(Bi) recurrent neural network(RNN) is adopted to integrate the learned features from different layers in CNNs. In addition, a conflict detection module is incorporated to address the conflict between modalities. Experiments on the Flickr dataset demonstrate that the MFC method achieves comparable performance compared with strong baseline methods.

关键词： multimodal fusion sentiment analysis deep learning

来源：

维普期刊数据库

同方期刊数据库评论

在线全文

学校读者我要写书评

暂无评论

Deep multimodal learning for municipal solid waste sorting

引用

Science China(Technological Sciences) 2022年第2期65卷 324-335页

作者： LU Gang WANG YuanBin XU HuXiu YANG HuaYong ZOU Jun State Key Laboratory of Fluid Power and Mechatronic Systems Zhejiang UniversityHangzhou 310027China

Automated waste sorting can dramatically increase waste sorting efficiency and reduce its regulation cost. Most of the current methods only use a single modality such as image data or acoustic data for waste classification, which makes it difficult to classify mixed and confusable wastes. In these complex situations, using multiple modalities becomes necessary to achieve a high classification accuracy. Traditionally, the fusion of multiple modalities has been limited by fixed handcrafted features. In this study, the deep-learning approach was applied to the multimodal fusion at the feature level for municipal solid-waste sorting.More specifically, the pre-trained VGG16 and one-dimensional convolutional neural networks(1 D CNNs) were utilized to extract features from visual data and acoustic data, respectively. These deeply learned features were then fused in the fully connected layers for classification. The results of comparative experiments proved that the proposed method was superior to the single-modality methods. Additionally, the feature-based fusion strategy performed better than the decision-based strategy with deeply learned features.

关键词： deep multimodal learning municipal waste sorting multimodal fusion convolutional neural networks

来源：

维普期刊数据库

同方期刊数据库评论

在线全文

学校读者我要写书评

暂无评论

Developing a Physiological Signal-Based, Mean Threshold and Decision-Level fusion Algorithm (PMD) for Emotion Recognition

引用

清华大学学报:自然科学版(英文版) 2023年第4期28卷 673-685页

作者： Qiuju Zhang Hongtao Zhang Keming Zhou Le Zhang College of Computer Science Sichuan UniversityChengdu 610065China August International Ltd. HoddesdonEN110EEUK Key Laboratory of Systems Health Science of Zhejiang Province Hangzhou Institute for Advanced StudyUniversity of Chinese Academy of SciencesHangzhou 310024China

With the development of computers,artificial intelligence,and cognitive science,engagement in deep communication between humans and computers has become increasingly important.Therefore,affective computing is a current hot research topic.Thus,this study constructs a Physiological signal-based,Mean-threshold,and Decision-level fusion algorithm(PMD)to identify human emotional states.First,we select key features from electroencephalogram and peripheral physiological signals,and use the mean-value method to obtain the classification threshold of each participant and distinguish individual differences.Then,we employ Gaussian Naive Bayes(GNB),Linear Regression(LR),Support Vector Machine(SVM),and other classification methods to perform emotion recognition.Finally,we improve the classification accuracy by developing an ensemble model.The experimental results reveal that physiological signals are more suitable for emotion recognition than classical facial and speech signals.Our proposed mean-threshold method can solve the problem of individual differences to a certain extent,and the ensemble learning model we developed significantly outperforms other classification models,such as GNB and LR.

关键词： electroencephalogram(EEG) peripheral physiological signals machine learning emotion recognition multimodal fusion

来源：

维普期刊数据库评论

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

3D Vehicle Detection Algorithm Based onmultimodal Decision-Level fusion

引用

Computer Modeling in Engineering & Sciences 2023年第6期135卷 2007-2023页

作者： Peicheng Shi Heng Qi Zhiqiang Liu Aixi Yang School of Mechanical Engineering Anhui Polytechnic UniversityWuhu241000China Department Polytechnic Institute of Zhejiang University Hangzhou310000China

3D vehicle detection based on LiDAR-camera fusion is becoming an emerging research topic in autonomous driving.The algorithm based on the Camera-LiDAR object candidate fusion method(CLOCs)is currently considered to be a more effective decision-level fusion algorithm,but it does not fully utilize the extracted features of 3D and 2D.Therefore,we proposed a 3D vehicle detection algorithm based onmultimodal decision-level fusion.First,project the anchor point of the 3D detection bounding box into the 2D image,calculate the distance between 2D and 3D anchor points,and use this distance as a new fusion feature to enhance the feature redundancy of the network.Subsequently,add an attention module:squeeze-and-excitation networks,weight each feature channel to enhance the important features of the network,and suppress useless features.The experimental results show that the mean average precision of the algorithm in the KITTI dataset is 82.96%,which outperforms previous state-ofthe-art multimodal fusion-based methods,and the average accuracy in the Easy,Moderate and Hard evaluation indicators reaches 88.96%,82.60%,and 77.31%,respectively,which are higher compared to the original CLOCs model by 1.02%,2.29%,and 0.41%,respectively.Compared with the original CLOCs algorithm,our algorithm has higher accuracy and better performance in 3D vehicle detection.

关键词： 3D vehicle detection multimodal fusion CLOCs network structure optimization attention module

来源：

维普期刊数据库评论

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

fusion of Hash-Based Hard and Soft Biometrics for Enhancing Face Image Database Search and Retrieval

引用

Computers, Materials & Continua 2023年第12期77卷 3489-3509页

作者： Ameerah Abdullah Alshahrani Emad Sami Jaha Nahed Alowidi Department of Computer Science Faculty of Computing and Information TechnologyKing Abdulaziz UniversityJeddah21589Saudi Arabia

The utilization of digital picture search and retrieval has grown substantially in numerous fields for different purposes during the last decade,owing to the continuing advances in image processing and computer vision approaches.In multiple real-life applications,for example,social media,content-based face picture retrieval is a well-invested technique for large-scale databases,where there is a significant necessity for reliable retrieval capabilities enabling quick search in a vast number of pictures.Humans widely employ faces for recognizing and identifying people.Thus,face recognition through formal or personal pictures is increasingly used in various real-life applications,such as helping crime investigators retrieve matching images from face image databases to identify victims and criminals.However,such face image retrieval becomes more challenging in large-scale databases,where traditional vision-based face analysis requires ample additional storage space than the raw face images already occupied to store extracted lengthy feature vectors and takes much longer to process and match thousands of face images.This work mainly contributes to enhancing face image retrieval performance in large-scale databases using hash codes inferred by locality-sensitive hashing(LSH)for facial hard and soft biometrics as(Hard BioHash)and(Soft BioHash),respectively,to be used as a search input for retrieving the top-k matching faces.Moreover,we propose the multi-biometric score-level fusion of both face hard and soft BioHashes(Hard-Soft BioHash fusion)for further augmented face image retrieval.The experimental outcomes applied on the Labeled Faces in the Wild(LFW)dataset and the related attributes dataset(LFW-attributes),demonstrate that the retrieval performance of the suggested fusion approach(Hard-Soft BioHash fusion)significantly improved the retrieval performance compared to solely using Hard BioHash or Soft BioHash in isolation,where the suggested method provides an augmented accuracy of 87%when executed on 1000 specimens

关键词： Face image retrieval soft biometrics similar pictures hashing database search large databases score-level fusion multimodal fusion

来源：

维普期刊数据库评论

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

欢迎您,

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

在线全文

在线全文

在线全文

在线全文

在线全文

在线全文

在线全文

在线全文

在线全文

在线全文

请选择保存的检索档案：

请选择收藏分类：

通借通还

欢迎您,

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

在线全文

在线全文

在线全文

在线全文

在线全文

在线全文

在线全文

在线全文

在线全文

在线全文

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：