检索结果-南通市图书馆

Contextual and selective attention networks for image captioning

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

Science China(Information Sciences) 2022年第12期65卷 142-156页

作者： Jing WANG Yehao LI Yingwei PAN Ting YAO Jinhui TANG Tao MEI School of Computer Science and Engineering Nanjing University of Science and Technology JD Explore Academy

The steady momentum of innovations has convincingly demonstrated the high capability of attention mechanisms for the sequence to sequence learning. Nevertheless, the computation of attention across a sequence is often independent in either hard or soft mode, thereby resulting in undesired effects such as repeated modeling. In this paper, we introduce a new design to holistically explore the interdependencies between attention histories and locally emphasize the strong focus of each attention on image captioning. Specifically, we present a contextual and selective attention network(namely Co SA-Net) that novelly memorizes contextual attention and brings out the principal components from each attention. Technically,Co SA-Net writes/updates the attended image region features into memory and reads from memory when measuring attention in the next time step to leverage contextual knowledge. Only the regions with the top-k highest attention scores are selected, and each region feature is individually employed to compute an output distribution. The final output is an attention-weighted mixture of all k distributions. In turn, the attention is then upgraded by the posterior distribution conditioned on the output. Our Co SA-Net is appealing given that it is pluggable to the sentence decoder in any neural captioning model. Extensive experiments on the COCO image captioning dataset demonstrate the superiority of Co SA-Net. More remarkably, integrating Co SA-Net to a one-layer long short-term memory(LSTM) decoder increases CIDEr-D performance from 125.2% to 128.5% on the COCO Karpathy test split. When further endowing a two-layer LSTM decoder with Co SA-Net, the CIDEr-D score is boosted to 129.5%.

关键词： image captioning hybrid attention contextual attention

Low Complexity Encoder with Multilabel Classification and image captioning Model

同方期刊数据库评论

在线全文

同方期刊数据库

学校读者我要写书评

暂无评论

Computers, Materials & Continua 2022年第9期72卷 4323-4337页

作者： Mahmoud Ragab Abdullah Addas Information Technology Department Faculty of Computing and Information TechnologyKing Abdulaziz UniversityJeddah21589Saudi Arabia Centre of Artificial Intelligence for Precision Medicines King Abdulaziz UniversityJeddah21589Saudi Arabia Mathematics Department Faculty of ScienceAl-Azhar UniversityNaser City11884CairoEgypt Landscape Architecture Department Faculty of Architecture&PlanningKing Abdulaziz UniversityJeddah21589Saudi Arabia

Due to the advanced development in the multimedia-on-demandtraffic in different forms of audio, video, and images, has extremely movedon the vision of the Internet of Things (IoT) from scalar to Internet ofMultimedia Things (IoMT). Since Unmanned Aerial Vehicles (UAVs) generates a massive quantity of the multimedia data, it becomes a part of IoMT,which are commonly employed in diverse application areas, especially forcapturing remote sensing (RS) images. At the same time, the interpretationof the captured RS image also plays a crucial issue, which can be addressedby the multi-label classification and Computational Linguistics based imagecaptioning techniques. To achieve this, this paper presents an efficient lowcomplexity encoding technique with multi-label classification and image captioning for UAV based RS images. The presented model primarily involves thelow complexity encoder using the Neighborhood Correlation Sequence (NCS)with a burrows wheeler transform (BWT) technique called LCE-BWT forencoding the RS images captured by the UAV. The application of NCS greatlyreduces the computation complexity and requires fewer resources for imagetransmission. Secondly, deep learning (DL) based shallow convolutional neural network for RS image classification (SCNN-RSIC) technique is presentedto determine the multiple class labels of the RS image, shows the novelty ofthe work. Finally, the Computational Linguistics based Bidirectional EncoderRepresentations from Transformers (BERT) technique is applied for imagecaptioning, to provide a proficient textual description of the RS image. Theperformance of the presented technique is tested using the UCM dataset. Thesimulation outcome implied that the presented model has obtained effectivecompression performance, reconstructed image quality, classification results,and image captioning outcome.

关键词： image captioning unmanned aerial vehicle low complexity encoder remote sensing images image classification

Visuals to Text:A Comprehensive Review on Automatic image captioning

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

IEEE/CAA Journal of Automatica Sinica 2022年第8期9卷 1339-1365页

作者： Yue Ming Nannan Hu Chunxiao Fan Fan Feng Jiangwan Zhou Hui Yu Beijing University of Posts and Telecommunications Beijing 100876China School of Creative Technologies University of PortsmouthPortsmouth PO12DJUK

image captioning refers to automatic generation of descriptive texts according to the visual content of *** is a technique integrating multiple disciplines including the computer vision(CV),natural language processing(NLP)and artificial *** recent years,substantial research efforts have been devoted to generate image caption with impressive *** summarize the recent advances in image captioning,we present a comprehensive review on image captioning,covering both traditional methods and recent deep learning-based ***,we first briefly review the early traditional works based on the retrieval and *** deep learning-based image captioning researches are focused,which is categorized into the encoder-decoder framework,attention mechanism and training strategies on the basis of model structures and training manners for a detailed *** that,we summarize the publicly available datasets,evaluation metrics and those proposed for specific requirements,and then compare the state of the art methods on the MS COCO ***,we provide some discussions on open challenges and future research directions.

关键词： Artificial intelligence attention mechanism encoder-decoder framework image captioning multi-modal understanding training strategies

Natural Language Processing with Optimal Deep Learning-Enabled Intelligent image captioning System

维普期刊数据库

同方期刊数据库评论

在线全文

学校读者我要写书评

暂无评论

Computers, Materials & Continua 2023年第2期74卷 4435-4451页

作者： Radwa Marzouk Eatedal Alabdulkreem Mohamed KNour Mesfer Al Duhayyim Mahmoud Othman Abu Sarwar Zamani Ishfaq Yaseen Abdelwahed Motwakel Department of Information Systems College of Computer and Information SciencesPrincess Nourah bint Abdulrahman UniversityP.O.Box 84428Riyadh11671Saudi Arabia Department of Computer Sciences College of Computer and Information SciencesPrincess Nourah bint Abdulrahman UniversityP.O.Box 84428Riyadh11671Saudi Arabia Department of Computer Sciences College of Computing and Information SystemUmm Al-Qura UniversitySaudi Arabia Department of Computer Science College of Sciences and Humanities-AflajPrince Sattam bin Abdulaziz UniversitySaudi Arabia Department of Computer Science Faculty of Computers and Information TechnologyFuture University in EgyptNew Cairo11835Egypt Department of Computer and Self Development Preparatory Year DeanshipPrince Sattam bin Abdulaziz UniversityAlKharjSaudi Arabia

The recent developments in Multimedia Internet of Things(MIoT)devices,empowered with Natural Language Processing(NLP)model,seem to be a promising future of smart *** plays an important role in industrial models such as speech understanding,emotion detection,home automation,and so *** an image needs to be captioned,then the objects in that image,its actions and connections,and any silent feature that remains under-projected or missing from the images should be *** aim of the image captioning process is to generate a caption for *** next step,the image should be provided with one of the most significant and detailed descriptions that is syntactically as well as semantically *** this scenario,computer vision model is used to identify the objects and NLP approaches are followed to describe the *** current study develops aNatural Language Processing with Optimal Deep Learning Enabled Intelligent image captioning System(NLPODL-IICS).The aim of the presented NLPODL-IICS model is to produce a proper description for input *** attain this,the proposed NLPODL-IICS follows two stages such as encoding and decoding ***,at the encoding side,the proposed NLPODL-IICS model makes use of Hunger Games Search(HGS)with Neural Search Architecture Network(NASNet)*** model represents the input data appropriately by inserting it into a predefined length ***,during decoding phase,Chimp Optimization Algorithm(COA)with deeper Long Short Term Memory(LSTM)approach is followed to concatenate the description sentences 4436 CMC,2023,vol.74,no.2 produced by the *** application of HGS and COA algorithms helps in accomplishing proper parameter tuning for NASNet and LSTM models *** proposed NLPODL-IICS model was experimentally validated with the help of two benchmark *** comparative analysis confirmed the superior performance of NLPODL-IICS model over other models.

关键词： Natural language processing information retrieval image captioning deep learning metaheuristics

Efficient image captioning Based on Vision Transformer Models

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

Computers, Materials & Continua 2022年第10期73卷 1483-1500页

作者： Samar Elbedwehy T.Medhat Taher Hamza Mohammed F.Alrahmawy Department of Data Science Faculty of Artificial IntelligenceKafrelsheikh UniversityEgypt Department of Electrical Engineering Faculty of EngineeringKafrelsheikh UniversityEgypt Department of Computer Science Faculty of Computer and Information ScienceMansouraEgypt

image captioning is an emerging field in machine *** refers to the ability to automatically generate a syntactically and semantically meaningful sentence that describes the content of an *** captioning requires a complex machine learning process as it involves two sub models:a vision sub-model for extracting object features and a language sub-model that use the extracted features to generate meaningful ***-based vision transformers models have a great impact in vision field *** this paper,we studied the effect of using the vision transformers on the image captioning process by evaluating the use of four different vision transformer models for the vision sub-models of the image captioning The first vision transformers used is DINO(self-distillation with no labels).The second is PVT(Pyramid Vision Transformer)which is a vision transformer that is not using convolutional *** third is XCIT(cross-Covariance image Transformer)which changes the operation in self-attention by focusing on feature dimension instead of token *** last one is SWIN(Shifted windows),it is a vision transformer which,unlike the other transformers,uses shifted-window in splitting the *** a deeper evaluation,the four mentioned vision transformers have been tested with their different versions and different configuration,we evaluate the use of DINO model with five different backbones,PVT with two versions:PVT_v1and PVT_v2,one model of XCIT,SWIN *** results show the high effectiveness of using SWIN-transformer within the proposed image captioning model with regard to the other models.

关键词： image captioning sequence-to-sequence self-distillation transformer convolutional layer

Oppositional Harris Hawks Optimization with Deep Learning-Based image captioning

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

Computer Systems Science & Engineering 2023年第1期44卷 579-593页

作者： V.R.Kavitha K.Nimala A.Beno K.C.Ramya Seifedine Kadry Byeong-Gwon Kang Yunyoung Nam Department of Computer Science and Engineering Prathyusha Engineering CollegeThiruvallur602025India Department of Networking and Communications SRM Institute of Science and TechnologyChennaiIndia Department of Electronics and Communication Engineering Dr.Sivanthi Aditanar College of EngineeringTiruchendur628215India Department of Electrical and Electronics Engineering Sri Krishna College of Engineering and TechnologyCoimbatore641008India Deparmtent of Applied Data Science Noroff University CollegeKristiansandNorway Department of Information and Communication Engineering Soonchunhyang UniversityAsanKorea Department of Computer Science and Engineering Soonchunhyang UniversityAsanKorea

image captioning is an emergent topic of research in the domain of artificial intelligence(AI).It utilizes an integration of Computer Vision(CV)and Natural Language Processing(NLP)for generating the image *** use in several application areas namely recommendation in editing applications,utilization in virtual assistance,*** development of NLP and deep learning(DL)modelsfind useful to derive a bridge among the visual details and textual *** this view,this paper introduces an Oppositional Harris Hawks Optimization with Deep Learning based image captioning(OHHO-DLIC)*** OHHO-DLIC technique involves the design of distinct levels of ***,the feature extraction of the images is carried out by the use of EfficientNet ***,the image captioning is performed by bidirectional long short term memory(BiLSTM)model,comprising encoder as well as *** last,the oppositional Harris Hawks optimization(OHHO)based hyperparameter tuning process is performed for effectively adjusting the hyperparameter of the EfficientNet and BiLSTM *** experimental analysis of the OHHO-DLIC technique is carried out on the Flickr 8k Dataset and a comprehensive comparative analysis highlighted the better performance over the recent approaches.

关键词： image captioning natural language processing artificial intelligence machine learning deep learning

PCATNet: Position-Class Awareness Transformer for image captioning

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

Computers, Materials & Continua 2023年第6期75卷 6007-6022页

作者： Ziwei Tang Yaohua Yi Changhui Yu Aiguo Yin Research Center of Graphic Communication Printing and PackagingWuhan UniversityWuhan430072China School of Remote Sensing and Information Engineering Wuhan UniversityWuhan430072China Zhuhai Pantum Electronics Co. Ltd.Zhuhai519060China

Existing image captioning models usually build the relation between visual information and words to generate captions,which lack spatial infor-mation and object *** address the issue,we propose a novel Position-Class Awareness Transformer(PCAT)network which can serve as a bridge between the visual features and captions by embedding spatial information and awareness of object *** our proposal,we construct our PCAT network by proposing a novel Grid Mapping Position Encoding(GMPE)method and refining the encoder-decoder ***,GMPE includes mapping the regions of objects to grids,calculating the relative distance among objects and ***,we also improve the Self-attention to adapt the ***,we propose a Classes Semantic Quantization strategy to extract semantic information from the object classes,which is employed to facilitate embedding features and refining the encoder-decoder *** capture the interaction between multi-modal features,we propose Object Classes Awareness(OCA)to refine the encoder and decoder,namely OCAE and OCAD,***,we apply GMPE,OCAE and OCAD to form various combinations and to complete the entire *** utilize the MSCOCO dataset to evaluate the performance of our *** results demonstrate that PCAT outperforms the other competitive methods.

关键词： image captioning relative position encoding object classes awareness

Fine-Grained Features for image captioning

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

Computers, Materials & Continua 2023年第6期75卷 4697-4712页

作者： Mengyue Shao Jie Feng Jie Wu Haixiang Zhang Yayu Zheng Zhejiang Sci-Tech University Hangzhou310020China Zhejiang University of Technology Hangzhou310020China

image captioning involves two different major modalities(image and sentence)that convert a given image into a language that adheres to visual *** all methods first extract image features to reduce the difficulty of visual semantic embedding and then use the caption model to generate fluent *** Convolutional Neural Network(CNN)is often used to extract image features in image captioning,and the use of object detection networks to extract region features has achieved great ***,the region features retrieved by this method are object-level and do not pay attention to fine-grained details because of the detection model’s *** offer an approach to address this issue that more properly generates captions by fusing fine-grained features and region ***,we extract fine-grained features using a panoramic segmentation ***,we suggest two fusion methods and contrast their fusion *** X-linear Attention Network(X-LAN)serves as the foundation for both fusion *** to experimental findings on the COCO dataset,the two-branch fusion approach is *** is important to note that on the COCO Karpathy test split,CIDEr is increased up to 134.3%in comparison to the baseline,highlighting the potency and viability of our method.

关键词： image captioning region features fine-grained features fusion

A Position-Aware Transformer for image captioning

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

Computers, Materials & Continua 2022年第1期70卷 2065-2081页

作者： Zelin Deng Bo Zhou Pei He Jianfeng Huang Osama Alfarraj Amr Tolba School of Computer and Communication Engineering Changsha University of Science and TechnologyChangsha410114China School of Computer Science and Cyber Engineering Guangzhou UniversityGuangzhou510006China Advanced Forming Research Centre University of StrathclydeRenfrewshirePA49LJGlasgowUnited Kingdom Department of Computer Science Community CollegeKing Saud UniversityRiyadh11437Saudi Arabia Department of Mathematics and Computer Science Faculty of ScienceMenoufia UniversityEgypt

image captioning aims to generate a corresponding description of an *** recent years,neural encoder-decodermodels have been the dominant approaches,in which the Convolutional Neural Network(CNN)and Long Short TermMemory(LSTM)are used to translate an image into a natural language *** these approaches,the visual attention mechanisms are widely used to enable deeper image understanding through fine-grained analysis and even multiple steps of ***,most conventional visual attention mechanisms are based on high-level image features,ignoring the effects of other image features,and giving insufficient consideration to the relative positions between image *** this work,we propose a Position-Aware Transformer model with image-feature attention and position-aware attention mechanisms for the above *** image-feature attention firstly extracts multi-level features by using Feature Pyramid Network(FPN),then utilizes the scaled-dot-product to fuse these features,which enables our model to detect objects of different scales in the image more effectivelywithout increasing *** the position-aware attentionmechanism,the relative positions between image features are obtained at first,afterwards the relative positions are incorporated into the original image features to generate captions more *** are carried out on the MSCOCO dataset and our approach achieves competitive BLEU-4,METEOR,ROUGE-L,CIDEr scores compared with some state-of-the-art approaches,demonstrating the effectiveness of our approach.

关键词： Deep learning image captioning transformer attention position-aware