Approximate Optimal Filter Design for Vehicle System through Actor‑Critic Reinforcement Learning

作     者：Yuming Yin Shengbo Eben Li Kaiming Tang Wenhan Cao Wei Wu Hongbo Li 

作者机构：School of Mechanical EngineeringZhejiang University of TechnologyHangzhou 310023China School of Vehicle and MobilityTsinghua UniversityBeijing 100084China Beijing Geekplus Tech.Co.LtdBeijingChina 

出 版 物：《Automotive Innovation》 (汽车创新工程（英文）)

年 卷 期：2022年第5卷第4期

页      面：415-426页

核心收录：

学科分类：0808[工学-电气工程] 0809[工学-电子科学与技术（可授工学、理学学位）] 080902[工学-电路与系统] 08[工学] 0802[工学-机械工程] 0823[工学-交通运输工程] 

基　　金：the financial support from National Natural Science Foundation of China (NSFC) International Science & Technology Cooperation Program of China Tsinghua-Toyota Joint Research Institute Inter-Disciplinary Program 

主　　题：Vehicle state estimation Kalman filter Estimation-control duality Reinforcement learning 

摘      要：Precise state and parameter estimations are essential for identification,analysis and control of vehicle engineering problems,especially under significant model and measurement uncertainties.The widely used filtering/estimation algorithms,such as Kalman series like Kalman filter,extended Kalman filter,unscented Kalman filter,and particle filter,generally aim to approach the true state/parameter distribution via iteratively updating the filter gain at each time step.However,the optimal-ity of these filters would be deteriorated by unrealistic initial condition or significant model error.Alternatively,this paper proposes to approximate the optimal filter gain by considering the effect factors within infinite time horizon,on the basis of estimation-control duality.The proposed approximate optimal filter(AOF)problem is designed and subsequently solved by actor-critic reinforcement learning(RL)method.The AOF design transforms the traditional optimal filtering problem with the minimum expected mean square error into an optimal control problem with the minimum accumulated estimation error,in which the estimation error is used as the surrogate system state and the infinite-horizon filter gain is the control input.The estimation-control duality is proved to hold when certain conditions about initial vehicle state distributions and policy structure are maintained.In order to evaluate of the effectiveness of AOF,a vehicle state estimation problem is then demonstrated and compared with the steady-state Kalman filter.The results showed that the obtained filter policy via RL with different discount factors can converge to theoretical optimal gain with an error within 5%,and the average estimation errors of vehicle slip angle and yaw rate are less than 1.5×10–4.