检索结果-南通市图书馆

CPT: a pre-trained unbalanced transformer for both Chinese language understanding and generation

在线全文

学校读者我要写书评

暂无评论

Science China(Information Sciences) 2024年第5期67卷 43-55页

作者： Yunfan SHAO Zhichao GENG Yitao LIU Junqi DAI Hang YAN Fei YANG Zhe LI Hujun BAO xipeng qiu School of Computer Science Fudan University Shanghai Key Laboratory of Intelligent Information Processing Fudan University Zhejiang Lab

In this paper, we take the advantage of previous pre-trained models(PTMs) and propose a novel Chinese pre-trained unbalanced transformer(CPT). Different from previous Chinese PTMs, CPT is designed to utilize the shared knowledge between natural language understanding(NLU) and natural language generation(NLG) to boost the performance. CPT consists of three parts: a shared encoder, an understanding decoder, and a generation decoder. Two specific decoders with a shared encoder are pretrained with masked language modeling(MLM) and denoising auto-encoding(DAE) tasks, *** the partially shared architecture and multi-task pre-training, CPT can(1) learn specific knowledge of both NLU or NLG tasks with two decoders and(2) be fine-tuned flexibly that fully exploits the potential of the model. Moreover, the unbalanced transformer saves the computational and storage cost, which makes CPT competitive and greatly accelerates the inference of text generation. Experimental results on a wide range of Chinese NLU and NLG tasks show the effectiveness of CPT.

关键词： pre-trained model transformer language model generation unified model

探索中文预训练模型的混合粒度编码和IDF遮蔽

在线全文

同方期刊数据库

学校读者我要写书评

暂无评论

中文信息学报 2024年第1期38卷 57-64页

作者：邵云帆孙天祥邱锡鹏复旦大学计算机科学技术学院上海200433

目前大多数中文预训练语言模型采用字级别编码,因为字符级编码序列长而产生大量计算开销。词级别编码尽管能够缓解这一问题,但也会带来其他问题,如词典外词、数据稀疏等。针对中文不同粒度的编码,该文提出使用混合粒度编码的中文预训练... 详细信息

目前大多数中文预训练语言模型采用字级别编码,因为字符级编码序列长而产生大量计算开销。词级别编码尽管能够缓解这一问题,但也会带来其他问题,如词典外词、数据稀疏等。针对中文不同粒度的编码,该文提出使用混合粒度编码的中文预训练模型。这一编码所用的词表在大规模预训练语料上得到,因此缓解了词典外词和数据稀疏问题。为了更进一步增强模型性能,该文提出了一种选择性的遮蔽语言建模训练策略——IDF遮蔽。这一策略基于词在大规模预训练语料上统计的逆文档频率。实验表明,与之前的中文预训练语言模型相比,该文所提出方法预训练的模型在多个中文自然语言数据集上取得了更好或相当的性能,并且能更高效地编码文本。

关键词：中文预训练混合粒度编码 IDF遮蔽

在线全文

学校读者我要写书评

暂无评论

大型语言模型:原理、实现与发展

计算机研究与发展 2024年第2期61卷 351-361页

作者：舒文韬李睿潇孙天祥黄萱菁邱锡鹏复旦大学计算机科学技术学院上海200433

近年来,大型语言模型的出现和发展对自然语言处理和人工智能领域产生了变革性影响.随着不断增大模型参数量和训练数据量,语言模型的文本建模困惑度以可预测的形式降低,在各类自然语言处理任务上的表现也持续提升.因此,增加语言模型的参... 详细信息

近年来,大型语言模型的出现和发展对自然语言处理和人工智能领域产生了变革性影响.随着不断增大模型参数量和训练数据量,语言模型的文本建模困惑度以可预测的形式降低,在各类自然语言处理任务上的表现也持续提升.因此,增加语言模型的参数和数据规模成为提升系统智能水平富有前景的途径.首先回顾了大型语言模型的基本定义,从模型表现和算力需求的角度给出了“大型”语言模型的界定标准.其次,从数据、算法、模型3个维度梳理了大型语言模型的发展历程及规律,展示了不同阶段各个维度的规模化如何推动语言模型的发展.接着,考察了大型语言模型所表现出的涌现能力,介绍了思维链、情景学习和指令遵循等关键涌现能力的相关研究和应用现状.最后,展望了大型语言模型的未来发展和技术挑战.

关键词：自然语言处理神经网络大型语言模型预训练对齐

InstructGPT在命名实体识别任务中的表现和挑战

在线全文

学校读者我要写书评

暂无评论

中文信息学报 2024年第1期38卷 74-85页

作者：孙瑜颜航邱锡鹏王定牟小峰黄萱菁复旦大学计算机科学技术学院上海200433 美的集团AI创新中心广东佛山528311

当前,关于大规模语言模型,例如,InstructGPT的研究主要聚焦在自由形式生成任务上,而忽略了在结构化抽取任务上的探索。为了让未来的工作在结构化抽取任务上有一个全面的认知,该文在零样本和少样本设定下,全面分析了InstructGPT在基础的... 详细信息

当前,关于大规模语言模型,例如,InstructGPT的研究主要聚焦在自由形式生成任务上,而忽略了在结构化抽取任务上的探索。为了让未来的工作在结构化抽取任务上有一个全面的认知,该文在零样本和少样本设定下,全面分析了InstructGPT在基础的结构化抽取任务,命名实体识别上的表现。为了让结论更加可靠,该文的实验同时包含了生物医学领域和通用领域的常规和嵌套数据集。实验结果表明,InstructGPT在零样本学习上的性能只能达到微调小规模语言模型的11%~56%,增加少量样本也最多只能提升至72%。为了探究InstructGPT在命名实体识别上表现性能不佳的原因,该文通过分析模型的输出,发现接近50%的句子都存在无效生成的问题。另外,由于无效生成会同时导致“虚假错误预测”和“虚假正确预测”,解决生成无效问题并不能保证性能的提升。此外,InstructGPT抽取嵌套实体的能力还是有待提高,抽取嵌套实体的比例也偏低。因此,用InstructGPT解决命名实体识别任务,除了要保证生成的有效性,还需要更加深入地研究才能找到行之有效的方法。

关键词：大规模语言模型命名实体识别上下文学习思维链

利用两样本孟德尔随机化方法探究复方口服避孕药与乳腺癌发病风险的关系

在线全文

学校读者我要写书评

暂无评论

现代肿瘤医学 2024年第11期32卷 2020-2024页

作者：邓佳琳江涛石玉琳邱锡鹏许家佗上海中医药大学中医学院上海201203 复旦大学计算机学院上海200433

目的:利用两样本孟德尔随机化方法探究评估抗复方口服避孕药(combined oral contraceptive,COC)对乳腺癌风险的影响。方法:对基于与COC高度相关的单核苷酸多态性的全基因组关联研究的合并数据进行二次数据分析,并选择雌激素和孕激素水... 详细信息

目的:利用两样本孟德尔随机化方法探究评估抗复方口服避孕药(combined oral contraceptive,COC)对乳腺癌风险的影响。方法:对基于与COC高度相关的单核苷酸多态性的全基因组关联研究的合并数据进行二次数据分析,并选择雌激素和孕激素水平作为工具变量,分别采用逆方差加权、MR-Egger回归和加权中位数方法进行两样本孟德尔随机化分析,以评估COC、雌激素及孕激素与乳腺癌和ER^(+)/ER^(-)乳腺癌之间的关联。我们分别采用逆方差加权、MR-Egger回归和加权中位数方法进行两样本孟德尔随机化分析。最后进行敏感性分析来评估结果的稳定性和可靠性。结果:逆方差加权、MR-Egger回归和加权中位数方法三种方法均显示COC的使用与乳腺癌(包括ER^(+)/ER^(-)乳腺癌)的发生无显著风险关联(P均>0.05)。结论:孟德尔随机化结果显示COC的使用与乳腺癌发病风险之间无关。

关键词：复方口服避孕药雌激素乳腺癌孟德尔随机化

Competition for gradient-free tuning of large language models:approaches, results, current challenges and future directions

在线全文

学校读者我要写书评

暂无评论

National Science Review 2023年第6期10卷 16-19页

作者： Tingfeng Cao Liang Chen Dixiang Zhang Tianxiang Sun Zhengfu He xipeng qiu Xing Xu Hai Zhang School of Software Engineering South China University of Technology School of Computer Science Fudan University School of Computer Science and Engineering University of Electronic Science and Technology of China Pazhou Laboratory (Huangpu) School of Mathematics Northwest University

PROBLEM Recent years have witnessed the rapid progress of self-supervised language models (LMs)[1],especially large language models (LLMs)[2].LLMs not only achieved state-of-the-art performance on many natural language processing tasks,but also captured widespread attention from the public due to their great potential in a variety of real-world applications (***,search engines,writing assistants,etc.)through providing general-purpose intelligent services.A few of the LLMs are becoming foundation models,an analogy to infrastructure,that empower hundreds of downstream applications.

关键词：

Syntax-guided text generation via graph neural network

在线全文

同方期刊数据库

学校读者我要写书评

暂无评论

Science China(Information Sciences) 2021年第5期64卷 67-76页

作者： Qipeng GUO xipeng qiu Xiangyang XUE Zheng ZHANG Shanghai Key Laboratory of Intelligent Information Processing School of Computer ScienceFudan University NYU Shanghai and AWS Shanghai AI Lab

Text generation is a fundamental and important task in natural language processing. Most of the existing models generate text in a sequential manner and have difficulty modeling complex dependency structures. In this paper, we treat the text generation task as a graph generation problem exploiting both syntactic and word-ordering relationships. Leveraging the framework of the graph neural network, we propose the word graph model. During the process, the model builds a sentence incrementally and maintains syntactic integrity via a syntax-driven, top-down, breadth-first generation process. Experimental results on both synthetic and real text generation tasks show the efficacy of our approach.

关键词： text generation deep learning graph neural network dependency parsing

Dual-axial self-attention network for text classification

在线全文

同方期刊数据库

学校读者我要写书评

暂无评论

Science China(Information Sciences) 2021年第12期64卷 80-90页

作者： Xiaochuan ZHANG xipeng qiu Jianmin PANG Fudong LIU Xingwei LI State Key Laboratory of Mathematical Engineering and Advanced Computing Shanghai Key Laboratory of Intelligent Information Processing Fudan University School of Computer Science Fudan University

Text classification is an important task in natural language processing and numerous studies aim to improve the accuracy and efficiency of text classification models. In this study, we propose an effective and efficient text classification model which is based on self-attention solely. The recently proposed multidimensional self-attention significantly improved the performance of self-attention. However, existing models suffer from two major limitations:(1) the previous multi-dimensional self-attention models are quite timeconsuming;(2) the dependencies of elements along the feature axis are not taken into account. To overcome these problems, in this paper, a much more computational efficient multi-dimensional self-attention model is proposed, and two parallel self-attention modules, called dual-axial self-attention, are applied to capture rich dependencies along the feature axis as well as the text axis. A text classification model is then *** experimental results on eight representative datasets show that the proposed text classification model can obtain state-of-the-art results and the proposed self-attention outperforms conventional self-attention models.

关键词： text classification dual-axial self-attention feature-axial dependency

Pre-trained models for natural language processing: A survey

在线全文

同方期刊数据库

学校读者我要写书评

暂无评论

Science China(Technological Sciences) 2020年第10期63卷 1872-1897页

作者： qiu xipeng SUN TianXiang XU YiGe SHAO YunFan DAI Ning HUANG XuanJing School of Computer Science Fudan UniversityShanghai 200433China Shanghai Key Laboratory of Intelligent Information Processing Shanghai 200433China

Recently, the emergence of pre-trained models(PTMs) has brought natural language processing(NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy from four different perspectives. Next,we describe how to adapt the knowledge of PTMs to downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.

关键词： deep learning neural network natural language processing pre-trained model distributed representation word embedding self-supervised learning language modelling