Design of Deep Reinforcement Learning Controller Through Data-assisted Model for Robotic Fish Speed Tracking

作     者：Palmani Duraisamy Manigandan Nagarajan Santhanakrishnan Amirtharajan Rengarajan Palmani Duraisamy;Manigandan Nagarajan Santhanakrishnan;Amirtharajan Rengarajan

作者机构：Robotics LabSchool of EEESASTRA Deemed UniversityThanjavurTamil Nadu613401India Department of ECESchool of EEESASTRA Deemed UniversityThanjavurTamil Nadu613401India 

出 版 物：《Journal of Bionic Engineering》 (仿生工程学报（英文版）)

年 卷 期：2023年第20卷第3期

页      面：953-966页

核心收录：

学科分类：0710[理学-生物学] 080202[工学-机械电子工程] 08[工学] 0804[工学-仪器科学与技术] 0802[工学-机械工程] 0811[工学-控制科学与工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：We would like to thank Rakesh Kumar S for his useful feedback that improved this paper 

主　　题：Bioinspired robot Data-assisted model Genetic algorithm Reinforcement learning control Robotic fish 

摘      要：It is common for robotic fish to generate thrust using reactive force generated by the tail’s physical motion, which interacts with the surrounding fluid. The coupling effect of the body strongly correlates with this thrust. However, hydrodynamics cannot be wholly modeled in analytical form. Therefore, data-assisted modeling is necessary for robotic fish. This work presents the first method of its kind using Genetic Algorithm (GA)-based optimization methods for data-assistive modeling for robotic fish applications. To begin, experimental data are collected in real time with the robotic fish that has been designed and fabricated using 3D printing. Then, the model’s influential parameters are estimated using an optimization problem. Further, a model-based deep reinforcement learning (DRL) controller is proposed to track the desired speed through extensive simulation work. In addition to a deep deterministic policy gradient (DDPG), a twin delayed DDPG (TD3) is employed in the training of the RL agent. Unfortunately, due to its local optimization problem, the RL-DDPG controller failed to perform well during training. In contrast, the RL-TD3 controller effectively learns the control policies and overcomes the local optima problem. As a final step, controller performance is evaluated under different disturbance conditions. In contrast to DDPG and GA-tuned proportional-integral controllers, the proposed model with RL-TD3 controller significantly improves the performance.