检索结果-南通市图书馆

Learnware:small models do big

在线全文

同方期刊数据库

学校读者我要写书评

暂无评论

Science China(Information Sciences) 2024年第1期67卷 39-50页

作者： Zhi-Hua ZHOU Zhi-Hao TAN National Key Laboratory for Novel Software Technology Nanjing University

There are complaints about current machine learning techniques such as the requirement of a huge amount of training data and proficient training skills,the difficulty of continual learning,the risk of catastrophic forgetting,and the leaking of data privacy/proprietary.Most research efforts have been focusing on one of those concerned issues separately,paying less attention to the fact that most issues are entangled in practice.The prevailing big model paradigm,which has achieved impressive results in natural language processing and computer vision applications,has not yet addressed those issues,whereas becoming a serious source of carbon emissions.This article offers an overview of the learnware paradigm,which attempts to enable users not to need to build machine learning models from scratch,with the hope of reusing small models to do things even beyond their original purposes,where the key ingredient is the specification which enables a trained model to be adequately identified to reuse according to the requirement of future users who know nothing about the model in advance.

关键词： artificial intelligence machine learning learnware

Enhancing unsupervised domain adaptation by exploiting the conceptual consistency of multiple self-supervised tasks

在线全文

同方期刊数据库

学校读者我要写书评

暂无评论

Science China(Information Sciences) 2023年第4期66卷 126-139页

作者： Hui SUN Ming LI National Key Laboratory for Novel Software Technology Nanjing University

Unsupervised domain adaptation(UDA) aims to transfer the knowledge from a label-rich source domain to an unlabeled target domain. Current approaches mainly focus on aligning the target domain’s data distribution with the source domain, but lacking attention to allowing target data to guide which features need to be captured. Source data dominates feature extraction. As a result, target embedding would lose some vital discriminative features and limit UDA in complex tasks. In this paper, we argue that utilizing auxiliary tasks to capture target-intrinsic patterns, which do not depend on source supervision, could further enhance UDA. Multiple auxiliary tasks understand the instance concept from different perspectives.Our findings show that exploiting the conceptual consistency of multiple auxiliary tasks to characterize the common part of these various understandings should reveal the target’s hidden ground truth. Furthermore, we propose a novel method named multiple self-supervision conceptual consistency domain adaptation(MSCC). Experiments and analysis on benchmark datasets show the effectiveness of our idea and method.

关键词： deep learning transfer learning domain adaptation domain shift self-supervised

A Region-Based Analysis for the Feature Concatenation in Deep Forests

在线全文

同方期刊数据库

学校读者我要写书评

暂无评论

Chinese Journal of Electronics 2022年第6期31卷 1072-1080页

作者： LYU Shen-Huan CHEN Yi-He ZHOU Zhi-Hua National Key Laboratory for Novel Software Technology Nanjing University

Deep forest is a tree-based deep model made up of non-differentiable modules that are trained without backpropagation. Despite the fact that deep forests have achieved considerable success in a variety of tasks, feature concatenation, as the ingredient for forest representation learning, still lacks theoretical understanding. In this paper, we aim to understand the influence of feature concatenation on predictive performance. To enable such theoretical studies, we present the first mathematical formula of feature concatenation based on the two-stage structure, which regards the splits along new features and raw features as a region selector and a region classifier respectively. Furthermore, we prove a region-based generalization bound for feature concatenation,which reveals the trade-off between Rademacher complexities of the two-stage structure and the fraction of instances that are correctly classified in the selected region.As a consequence, we show that compared with the prediction-based feature concatenation(PFC), the advantage of interaction-based feature concatenation(IFC) is that it obtains more abundant regions through distributed representation and alleviates the overfitting risk in local regions. Experiments confirm the correctness of our theoretical results.

关键词： Deep forest Overfitting Generalization bound Representation learning

LAMDA-SSL:a comprehensive semi-supervised learning toolkit

在线全文

同方期刊数据库

学校读者我要写书评

暂无评论

Science China(Information Sciences) 2024年第1期67卷 306-307页

作者： Lin-Han JIA Lan-Zhe GUO Zhi ZHOU Yu-Feng LI National Key Laboratory for Novel Software Technology Nanjing University

Machine learning, particularly deep learning, has achieved remarkable success across a wide range of tasks. However,most of these tasks demand a substantial amount of labeled training data, which can be challenging to obtain in many real-world applications due to difficulties, costs, or time constraints associated with labeling. In this context,semi-supervised learning (SSL) has emerged as a promising paradigm to alleviate the dependency on large labeled datasets by harnessing the power of unlabeled data [1–3].

关键词：

A unified pruning framework for vision transformers

在线全文

同方期刊数据库

学校读者我要写书评

暂无评论

Science China(Information Sciences) 2023年第7期66卷 303-304页

作者： Hao YU Jianxin WU State Key Laboratory for Novel Software Technology Nanjing University

The transformer architecture [1] has been widely used for natural language processing(NLP) tasks. Under the inspiration of its excellent performance in NLP, transformer-based models [2, 3] have established many new records in various computer vision tasks. However, most vision transformers(Vi Ts) suffer from large model sizes, large run-time memory consumption, and high computational costs. Therefore, impending needs exist to develop and deploy lightweight and efficient vision transformers.

关键词：

PyCIL: a Python toolbox for class-incremental learning

在线全文

同方期刊数据库

学校读者我要写书评

暂无评论

Science China(Information Sciences) 2023年第9期66卷 291-292页

作者： Da-Wei ZHOU Fu-Yun WANG Han-Jia YE De-Chuan ZHAN State Key Laboratory for Novel Software Technology Nanjing University

With the rapid development of deep learning, current deep models can learn a fixed number of classes with high performance. However, in our ever-changing world, data often come from the open environment, which is with stream format or available temporarily due to privacy issues. As a result, the classification model should learn new classes incrementally instead of restarting the training process.

关键词：

在线全文

同方期刊数据库

学校读者我要写书评

暂无评论

Deep forest

National Science Review 2019年第1期6卷 74-86页

作者： Zhi-Hua Zhou Ji Feng National Key Laboratory for Novel Software Technology Nanjing University

Current deep-learning models are mostly built upon neural networks, i.e. multiple layers of parameterized differentiable non-linear modules that can be trained by backpropagation. In this paper, we explore the possibility of building deep models based on non-differentiable modules such as decision trees. After a discussion about the mystery behind deep neural networks, particularly by contrasting them with shallow neural networks and traditional machine-learning techniques such as decision trees and boosting machines,we conjecture that the success of deep neural networks owes much to three characteristics, i.***.layer-by-layer processing, in-model feature transformation and sufficient model complexity. On one hand,our conjecture may offer inspiration for theoretical understanding of deep learning; on the other hand, to verify the conjecture, we propose an approach that generates deep forest holding these characteristics. This is a decision-tree ensemble approach, with fewer hyper-parameters than deep neural networks, and its model complexity can be automatically determined in a data-dependent way. Experiments show that its performance is quite robust to hyper-parameter settings, such that in most cases, even across different data from different domains, it is able to achieve excellent performance by using the same default setting. This study opens the door to deep learning based on non-differentiable modules without gradient-based adjustment, and exhibits the possibility of constructing deep models without backpropagation.

关键词： deep forest deep learning machine learning ensemble methods decision trees

Deep multiple instance selection

在线全文

同方期刊数据库

学校读者我要写书评

暂无评论

Science China(Information Sciences) 2021年第3期64卷 18-32页

作者： Xin-Chun LI De-Chuan ZHAN Jia-Qi YANG Yi SHI State Key Laboratory for Novel Software Technology Nanjing University

Multiple instance learning(MIL) assigns a single class label to a bag of instances tailored for some real-world applications such as drug activity prediction. Classical MIL methods focus on figuring out interested instances, that is, region of interests(ROIs). However, owing to the non-differentiable selection process, these methods are not feasible in deep learning. Thus, we focus on fusing ROIs identification with deep MILs in this paper. We propose a novel deep MIL framework based on hard selection, that is, deep multiple instance selection(DMIS), which can automatically figure ROIs out in an end-to-end approach.To be specific, we propose DMIS-GS for instance selection via gumbel softmax or gumbel top-k, and then make predictions for this bag without the interference of redundant instances. For balancing exploration and exploitation of key instances, we apply a cooling down approach to the temperature in DMIS-GS, and propose a variance normalization method to make this hyper-parameter tuning process much easier. Generally, we give a theoretical analysis of our framework. The empirical investigations reveal the proposed frameworks’ superiorities against classical MIL methods on generalization ability, positioning ROIs, and comprehensibility on both synthetic and real-world datasets.

关键词： multiple instance learning instance selection gumbel softmax variance normalization hard attention

Why over-parameterization of deep neural networks does not overfit?

在线全文

同方期刊数据库

学校读者我要写书评

暂无评论

Science China(Information Sciences) 2021年第1期64卷 242-244页

作者： Zhi-Hua ZHOU National Key Laboratory for Novel Software Technology Nanjing University

Deep neural networks often come with a huge number of parameters,even larger than the number of training examples,but it seems that these over-parameterized models have not suffered from overfitting.This is quite strange and why over-parameterization does not overfit?poses a fundamental question concerning the mysteries behind the success of deep neural networks.

关键词： Deep Neural Networks perspective AI Over-parameterization machine learning