A hybrid-model optimization algorithm based on the Gaussian process and particle swarm optimization for mixed-variable CNN hyperparameter automatic search

作     者：Han YAN Chongquan ZHONG Yuhu WU Liyong ZHANG Wei LU Han YAN;Chongquan ZHONG;Yuhu WU;Liyong ZHANG;Wei LU

作者机构：School of Control Science and EngineeringDalian University of TechnologyDalian 116024China 

出 版 物：《Frontiers of Information Technology & Electronic Engineering》 (信息与电子工程前沿（英文版）)

年 卷 期：2023年第24卷第11期

页      面：1557-1573页

核心收录：

学科分类：0831[工学-生物医学工程（可授工学、理学、医学学位）] 1002[医学-临床医学] 08[工学] 081104[工学-模式识别与智能系统] 0805[工学-材料科学与工程（可授工学、理学学位）] 0714[理学-统计学（可授理学、经济学学位）] 0811[工学-控制科学与工程] 0701[理学-数学] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：supported by the National Natural Science Foundation of China (Nos.62073056 and 61876029) the Applied Basic Research Project of Liaoning Province,China (No.2023JH2/101300207) the Key Field Innovation Team Project of Dalian,China (No.2021RT14)。 

主　　题：Convolutional neural network Gaussian process Hybrid model Hyperparameter optimization Mixed-variable Particle swarm optimization 

摘      要：Convolutional neural networks(CNNs)have been developed quickly in many real-world fields.However,CNN’s performance depends heavily on its hyperparameters,while finding suitable hyperparameters for CNNs working in application fields is challenging for three reasons:(1)the problem of mixed-variable encoding for different types of hyperparameters in CNNs,(2)expensive computational costs in evaluating candidate hyperparameter configuration,and(3)the problem of ensuring convergence rates and model performance during hyperparameter search.To overcome these problems and challenges,a hybrid-model optimization algorithm is proposed in this paper to search suitable hyperparameter configurations automatically based on the Gaussian process and particle swarm optimization(GPPSO)algorithm.First,a new encoding method is designed to efficiently deal with the CNN hyperparameter mixed-variable problem.Second,a hybrid-surrogate-assisted model is proposed to reduce the high cost of evaluating candidate hyperparameter configurations.Third,a novel activation function is suggested to improve the model performance and ensure the convergence rate.Intensive experiments are performed on image-classification benchmark datasets to demonstrate the superior performance of GPPSO over state-of-the-art methods.Moreover,a case study on metal fracture diagnosis is carried out to evaluate the GPPSO algorithm performance in practical applications.Experimental results demonstrate the effectiveness and efficiency of GPPSO,achieving accuracy of 95.26%and 76.36%only through 0.04 and 1.70 GPU days on the CIFAR-10 and CIFAR-100 datasets,respectively.