UAV swarm air combat maneuver decision-making method based on multi-agent reinforcement learning and transferring

作     者：Zhiqiang ZHENG Chen WEI Haibin DUAN 

作者机构：State Key Laboratory of Virtual Reality Technology and SystemsSchool of Automation Science and Electrical Engineering Beihang University 

出 版 物：《Science China(Information Sciences)》 (中国科学:信息科学(英文版))

年 卷 期：2024年第67卷第8期

页      面：49-66页

核心收录：

学科分类：12[管理学] 1201[管理学-管理科学与工程(可授管理学、工学学位)] 081104[工学-模式识别与智能系统] 08[工学] 082503[工学-航空宇航制造工程] 0835[工学-软件工程] 0825[工学-航空宇航科学与技术] 0811[工学-控制科学与工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：supported by National Key R&D Program of China (Grant No. 2023YFC3011001) National Natural Science Foundation of China (Grant Nos. U20B2071, 62350048, T2121003) 

主　　题：UAV swarm short-range air combat multi-agent reinforcement learning reward assignment transfer 

摘      要：During short-range air combat involving unmanned aircraft vehicle(UAV) swarms, UAVs must make accurate maneuver decisions based on information from both enemy and friendly UAVs. This dual requirement of competition and cooperation presents a significant challenge in the field of unmanned air combat. In this paper, a method based on multi-agent reinforcement learning(MARL) is proposed to address this issue. An actor network containing three subnetworks that can handle different types of situational information is designed. Hence, the results from simpler one-on-one scenarios are leveraged to enhance the complex swarm air combat training process. Separate state spaces for local and global information are designed for the actor and critic networks. A detailed reward function is proposed to encourage *** prevent lazy participants in air combat, a reward assignment operation is applied to distribute these dense rewards. Simulation testing and ablation experiments demonstrate that both the transfer operation and reward assignment operation can effectively deal with the swarm air combat scenario, and reflect the effectiveness of the proposed method.