检索结果-南通市图书馆

Large-scale self-normalizing neural networks

Journal of Automation and Intelligence 2024年第2期3卷 101-110页

作者： Zhaodong Chen Weiqin Zhao Lei Deng Yufei Ding Qinghao Wen Guoqi Li Yuan Xie University of California Santa BarbaraSanta Barbara93106CAUnited States University of Hong Kong Hong Kong999077China Tsinghua University Beijing100084China University of California San DiegoSan Diego92093CAUnited States University of Sydney Sydney2006Australia Institute of Automation Chinese Academy of ScienceBeijing100190China University of Chinese Academy of Science Beijing100190China The Hong Kong University of Science and Technology Hong Kong999077China

Self-normalizing neural networks(SNN)regulate the activation and gradient flows through activation functions with the self-normalization *** SNNs do not rely on norms computed from minibatches,they are more friendly to data parallelism,kernel fusion,and emerging architectures such as ReRAM-based ***,existing SNNs have mainly demonstrated their effectiveness on toy datasets and fall short in accuracy when dealing with large-scale tasks like *** lack the strong normalization,regularization,and expression power required for wider,deeper models and larger-scale *** enhance the normalization strength,this paper introduces a comprehensive and practical definition of the self-normalization property in terms of the stability and attractiveness of the statistical fixed *** is comprehensive as it jointly considers all the fixed points used by existing studies:the first and second moment of forward activation and the expected Frobenius norm of backward *** practicality comes from the analytical equations provided by our paper to assess the stability and attractiveness of each fixed point,which are derived from theoretical analysis of the forward and backward *** proposed definition is applied to a meta activation function inspired by prior research,leading to a stronger self-normalizing activation function named‘‘bi-scaled exponential linear unit with backward standardized’’(bSELU-BSTD).We provide both theoretical and empirical evidence to show that it is superior to existing *** enhance the regularization and expression power,we further propose scaled-Mixup and channel-wise scale&*** these three techniques,our approach achieves 75.23%top-1 accuracy on the ImageNet with Conv MobileNet V1,surpassing the performance of existing self-normalizing activation *** the best of our knowledge,this is the first SNN that achieves comparable accuracy to batch normalization on ImageNet.

关键词： Self-normalizing neural network Mean-field theory block dynamical isometry Activation function

来源：

维普期刊数据库评论

在线全文

维普期刊数据库

学校读者我要写书评

暂无评论

欢迎您,

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

在线全文

请选择保存的检索档案：

请选择收藏分类：

通借通还

欢迎您,

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

在线全文

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：