Deep reinforcement learning using least-squares truncated temporal-difference

作     者：Junkai Ren Yixing Lan Xin Xu Yichuan Zhang Qiang Fang Yujun Zeng 

作者机构：College of Intelligence Science and TechnologyNational University of Defense TechnologyChangshaChina State Key Laboratory of Astronautic DynamicsXi'an Satellite Control CenterXi'anChina 

出 版 物：《CAAI Transactions on Intelligence Technology》 (智能技术学报（英文）)

年 卷 期：2024年第9卷第2期

页      面：425-439页

核心收录：

学科分类：12[管理学] 1201[管理学-管理科学与工程(可授管理学、工学学位)] 081104[工学-模式识别与智能系统] 08[工学] 0835[工学-软件工程] 0811[工学-控制科学与工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：Joint Funds of the National Natural Science Foundation of China,Grant/Award Number:U21A20518 National Natural Science Foundation of China,Grant/Award Numbers:62106279,61903372 

主　　题：Deep reinforcement learning policy evaluation temporal difference value function approximation 

摘      要：Policy evaluation(PE)is a critical sub-problem in reinforcement learning,which estimates the value function for a given policy and can be used for policy ***,there still exist some limitations in current PE methods,such as low sample efficiency and local convergence,especially on complex *** this study,a novel PE algorithm called Least-Squares Truncated Temporal-Difference learning(LST2D)is *** LST2D,an adaptive truncation mechanism is designed,which effectively takes advantage of the fast convergence property of Least-Squares Temporal Difference learning and the asymptotic convergence property of Temporal Difference learning(TD).Then,two feature pre-training methods are utilised to improve the approximation ability of ***,an Actor-Critic algorithm based on LST2D and pre-trained feature representations(ACLPF)is proposed,where LST2D is integrated into the critic network to improve learning-prediction *** simulation studies were conducted on four robotic tasks,and the corresponding results illustrate the effectiveness of *** proposed ACLPF algorithm outperformed DQN,ACER and PPO in terms of sample efficiency and stability,which demonstrated that LST2D can be applied to online learning control problems by incorporating it into the actor-critic architecture.