检索结果-南通市图书馆

language model Using Differentiable Neural Computer Based on Forget Gate-Based Memory Deallocation

同方期刊数据库评论

在线全文

同方期刊数据库

学校读者我要写书评

暂无评论

Computers, Materials & Continua 2021年第7期68卷 537-551页

作者： Donghyun Lee Hosung Park Soonshin Seo Changmin Kim Hyunsoo Son Gyujin Kim Ji-Hwan Kim Department of Computer Science and Engineering Sogang UniversitySeoul04107Korea

A differentiable neural computer(DNC)is analogous to the Von Neumann machine with a neural network controller that interacts with an external memory through an attention mechanism.Such DNC’s offer a generalized method for task-specific deep learning models and have demonstrated reliability with reasoning problems.In this study,we apply a DNC to a language model(LM)task.The LM task is one of the reasoning problems,because it can predict the next word using the previous word sequence.However,memory deallocation is a problem in DNCs as some information unrelated to the input sequence is not allocated and remains in the external memory,which degrades performance.Therefore,we propose a forget gatebased memory deallocation(FMD)method,which searches for the minimum value of elements in a forget gate-based retention vector.The forget gatebased retention vector indicates the retention degree of information stored in each external memory address.In experiments,we applied our proposed NTM architecture to LM tasks as a task-specific example and to rescoring for speech recognition as a general-purpose example.For LM tasks,we evaluated DNC using the Penn Treebank and enwik8 LM tasks.Although it does not yield SOTA results in LM tasks,the FMD method exhibits relatively improved performance compared with DNC in terms of bits-per-character.For the speech recognition rescoring tasks,FMD again showed a relative improvement using the LibriSpeech data in terms of word error rate.

关键词： Forget gate-based memory deallocation differentiable neural computer language model forget gate-based retention vector

A Method to Build a Super Small but Practically Accurate language model for Handheld Devices

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

Journal of Computer Science & Technology 2003年第6期18卷 747-755页

作者：吴根清郑方 CenterofSpeechTechnology StateKeyLaboratoryofIntelligentTechnologyandSystemsDepartmentofComputerScienceandTechnologyTsinghuaUniversityBeijing100084P.R.China TheauthorcurrentlyisalsowithBeijingd-EarTechnologiesCo. Ltd.

In this paper, an important question, whether a small language model can be practically accurate enough, is raised. Afterwards, the purpose of a language model, the problems that a language model faces, and the factors that affect the performance of a language model,are analyzed. Finally, a novel method for language model compression is proposed, which makes the large language model usable for applications in handheld devices, such as mobiles, smart phones, personal digital assistants (PDAs), and handheld personal computers (HPCs). In the proposed language model compression method, three aspects are included. First, the language model parameters are analyzed and a criterion based on the importance measure of n-grams is used to determine which n-grams should be kept and which removed. Second, a piecewise linear warping method is proposed to be used to compress the uni-gram count values in the full language model. And third, a rank-based quantization method is adopted to quantize the bi-gram probability values. Experiments show that by using this compression method the language model can be reduced dramatically to only about 1M bytes while the performance almost does not decrease. This provides good evidence that a language model compressed by means of a well-designed compression technique is practically accurate enough, and it makes the language model usable in handheld devices.

关键词： language model language model compression piecewise linear warping tank-based quantization

A Bit Progress on Word-Based language model

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

Journal of Shanghai University(English Edition) 2003年第2期7卷 148-155页

作者：陈勇陈国评 DepartmentofComputerScienceandInformationSystem

A good language model is essential to a postprocessing algorithm for recognition systems. In the past, researchers have presented various language models, such as character based language models, word based language model, syntactical rules language model, hybrid models, etc . The word N gram model is by far an effective and efficient model, but one has to address the problem of data sparseness in establishing the model. Katz and Kneser et al. respectively presented effective remedies to solve this challenging problem. In this study, we proposed an improvement to their methods by incorporating Chinese language specific information or Chinese word class information into the system.

关键词： language model pattern recognition Chinese character recognition.

Protein Structure Prediction:Challenges,Advances,and the Shift of Research Paradigms

维普期刊数据库

同方期刊数据库评论

在线全文

学校读者我要写书评

暂无评论

Genomics, Proteomics & Bioinformatics 2023年第5期21卷 913-925页

作者： Bin Huang Lupeng Kong Chao Wang Fusong Ju Qi Zhang Jianwei Zhu Tiansu Gong Haicang Zhang Chungong Yu Wei-Mou Zheng Dongbo Bu Key Laboratory of Intelligent Information Processing Institute of Computing TechnologyChinese Academy of SciencesBeijing 100190China University of Chinese Academy of Sciences Beijing 100049China Changping Laboratory Beijing 102206China Microsoft Research AI4Science Beijing 100080China Huawei Noah’s Ark Lab Wuhan 430206China Zhongke Big Data Academy Zhengzhou 450046China Institute of Theoretical Physics Chinese Academy of SciencesBeijing 100190China

Protein structure prediction is an interdisciplinary research topic that has attracted researchers from multiple fields,including biochemistry,medicine,physics,mathematics,and computer science.These researchers adopt various research paradigms to attack the same structure prediction problem:biochemists and physicists attempt to reveal the principles governing protein folding;mathematicians,especially statisticians,usually start from assuming a probability distribution of protein structures given a target sequence and then find the most likely structure,while computer scientists formulate protein structure prediction as an optimization problem-finding the structural conformation with the lowest energy or minimizing the difference between predicted structure and native structure.These research paradigms fall into the two statistical modeling cultures proposed by Leo Breiman,namely,data modeling and algorithmic modeling.Recently,we have also witnessed the great success of deep learning in protein structure prediction.In this review,we present a survey of the efforts for protein structure prediction.We compare the research paradigms adopted by researchers from different fields,with an emphasis on the shift of research paradigms in the era of deep learning.In short,the algorithmic modeling techniques,especially deep neural networks,have considerably improved the accuracy of protein structure prediction;however,theories interpreting the neural networks and knowledge on protein folding are still highly desired.

关键词： Protein folding Protein structure prediction Deep learning Transformer language model

Research on Tibetan Speech Recognition Based on the Am-do Dialect

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

Computers, Materials & Continua 2022年第12期73卷 4897-4907页

作者： Kuntharrgyal Khysru Jianguo Wei Jianwu Dang Key Laboratory of Artificial Intelligence Application Technology State Ethnic Affairs Commission Qinghai Minzu UniversityXining810007China Tianjin Key Laboratory of Cognitive Computing and Application Tianjin UniversityTianjin300072China Japan Advanced Institute of Science and Technology IshikawaJapan

In China,Tibetan is usually divided into three major dialects:the Am-do,Khams and Lhasa dialects.The Am-do dialect evolved from ancient Tibetan and is a local variant of modern Tibetan.Although this dialect has its own specific historical and social conditions and development,there have been different degrees of communication with other ethnic groups,but all the abovementioned dialects developed from the same language:Tibetan.This paper uses the particularity of Tibetan suffixes in pronunciation and proposes a lexicon for the Am-do language,which optimizes the problems existing in previous research.Audio data of the Am-do dialect are expanded by data augmentation technology combining noise and reverberation,and the morphological characteristics and characteristics of the Tibetan language are further considered.According to the particularity of Tibetan grammar,grammatical features are used to optimize grammatical relationships and are combined with a language model,and the Am-do dialect is scored and rescored.Experimental results show that compared with the baseline,our proposed new lexicon and data augmentation technology yields a relative increase of approximately 3%in character error rates(CERs)and a relative increase of 3%-19%in the recognition rate of acoustic models and language models.

关键词： Am-do dialect acoustic model language model rescoring

Summary-based model of Information Retrieval in language model Framework

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

Summary-based Model of Information Retrieval in Language mod...

Investigation of Long Short-Term Memory for Tibetan language model

第四届全国信息检索与内容安全学术会议

作者： Li Wei-jiang~(1,+) Zhao Tie-Jun~2 ~1 Computer Application Key Laboratory of Yunnan Province,Kunmmg University of Science and Technology,Kunming 650051,China ~2 School of Computer Science and Technology,Harbin Institute of Technology,Harbin 150001,China

Standard IR treats the whole document as relevant,the implication of this being that using terms from non-relevant sections of these documents for IR may also cause query drift.This paper investigates a novel retrieval model that aims to concentrate more user’s attention on the parts of the text that possess a high density of relevant information,while,at the same time,to provide enough information to support their retrieval decisions.It is proposed that an automatically generated summary of each document,biased to a user’s query,can provide such a function.Based on the experimental results on TREC collections,it is showed the retrieval model based on summary is more effective than standard one based on document.

关键词： Information Retrieval Summary language model smoothing methods

来源： cnki会议评论

在线全文

cnki会议

学校读者我要写书评

暂无评论

Investigation of Long Short-Term Memory for Tibetan Language...

Relation Extraction Based on Prompt Information and Feature Reuse

第十四届全国人机语音通讯学术会议

作者： Tongtong Shen Longbiao Wang Xie Chen Kuntharrgyal Khysru Jianwu Dang Tianjin Key Lab.of Cognitive Computing and Application Tianjin University University of Cambridge Japan Advanced Institute of Science and Technology

Nowadays,recurrent neural network language model（RNNLM） has achieved excellent performance in many speech and natural language processing（NLP） tasks.However,it is hard to train due to the gradient vanishing problem.The problem can be relieved by long short-term memory language model（LSTMLM）.But training of LSTMLM poses great difficulties for low-resource language such as Tibetan since it requires a large quantities of data to guarantee its good performance.This paper mainly explores the performance of LSTMLM for Tibetan by utilizing its characteristics.In addition,domain adaptation is applied to solve the low resource issue to obtain a better performance.Furthermore,an interpolated language model is proposed to get further improvements,which better exploits the complementary attributes between N-gram LM and LSTMLM.Experiments are conducted with two data sets with different sizes,small Tibetan data set（STD） and large Tibetan data set（LTD） respectively.The results show that the proposed LMs show better performance.Compared to Knerser-Ney 3-gram trained with STD,our domain adapted LM,interpolated LM and interpolated LM with domain adaptation yield consistent gains,28.8%,18.0%and 34.2%in relative perplexity reduction respectively.

关键词： Long short-term memory language model Tibetan perplexity low resource

来源： cnki会议评论

在线全文

cnki会议

学校读者我要写书评

暂无评论

Data Intelligence 2023年第3期5卷 824-840页

作者： Ping Feng Xin Zhang Jian Zhao Yingying Wang Biao Huang Jilin University Changchun Jilin 130012China Changchun Universiy Changchun Jilin 130022China Jilin Provincial Key Laboratory of Human Health State Identification and Function Enhancement Changchun Jjilin 130022China

To alleviate the problem of under-utilization features of sentence-level relation extraction,which leads to insufficient performance of the pre-trained language model and underutilization of the feature vector,a sentence-level relation extraction method based on adding prompt information and feature reuse is proposed.At first,in addition to the pair of nominals and sentence information,a piece of prompt information is added,and the overall feature information consists of sentence information,entity pair information,and prompt information,and then the features are encoded by the pre-trained language model ROBERTA.Moreover,in the pre-trained language model,BIGRU is also introduced in the composition of the neural network to extract information,and the feature information is passed through the neural network to form several sets of feature vectors.After that,these feature vectors are reused in different combinations to form multiple outputs,and the outputs are aggregated using ensemble-learning soft voting to perform relation classification.In addition to this,the sum of cross-entropy,KL divergence,and negative log-likelihood loss is used as the final loss function in this paper.In the comparison experiments,the model based on adding prompt information and feature reuse achieved higher results of the SemEval-2010 task 8 relational dataset.

关键词： relation extraction language model prompt information feature reuse loss function

Evaluation on ChatGPT for Chinese language Understanding

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

Data Intelligence 2023年第4期5卷 885-903页

作者： Linhan Li Huaping Zhang Chunjin Li Haowen You Wenyao Cui School of Computer Science Beijing Institute of TechnologyBeijingChina

ChatGPT has attracted extension attention of academia and industry.This paper aims to evaluate ChatGPT in Chinese language understanding capability on 6 tasks using 11 datasets.Experiments indicate that ChatGPT achieved competitive results in sentiment analysis,summary,and reading comprehension in Chinese,while it is prone to factual errors in closed-book QA.Further,on two more difficult Chinese understanding tasks,that is,idiom fill-in-the-blank and cants understanding,we found that a simple chain-of-thought prompt can improve the accuracy of ChatGPT in complex reasoning.This paper further analyses the possible risks of using ChatGPT based on the results.Finally,we briefly describe the research and development progress of our ChatBIT.

关键词： language model ChatGPT ChatBIT Chinese language Understanding Artificial intelligence