无人水下航行器（Unmanned Underwater Vehicle，UUV）集群的任务分配问题是UUV集群形成水下功能的重要问题之一，但是，受限于通信以及探测能力，UUV在水下只能获取有限的信息，不能得到很好的应用。提出一种基于深度强化学习的任务分配算法，针对水下信息缺失、奖励稀少的问题，在近端策略优化算法的基础上加入Curiosity模块，给智能体一种减小环境中不确定性的期望，鼓励UUV探索环境中不可预测的部分，实现UUV集群的最优任务分配。最后的仿真实验表明，相较于传统智能算法，该方法收敛更快，可靠性更强。
The Assignment problem problem of the UUV cluster is one of the important problems for the formation of the underwater function of the UUV cluster. However, due to the communication and detection capabilities, UUV can only obtain limited information underwater and cannot be used well. A task allocation algorithm based on deep reinforcement learning is proposed. Aiming at the problem of lack of underwater information and scarce rewards, the Curiosity module is added on the basis of the near end strategy optimization algorithm, giving agents an expectation to reduce the uncertainty in the environment, encouraging UUV to explore the unpredictable part of the environment, and realizing the optimal task allocation of UUV clusters. The final simulation experiment shows that compared to traditional intelligent algorithms, it converges faster and has stronger reliability.
2024,46(12): 84-89 收稿日期：2023-08-16
DOI：10.3404/j.issn.1672-7649.2024.12.015
分类号：U661
基金项目：中国科协青年人才托举工程项目（2020-JCJQ-QT-013）
作者简介：董经纬(1998-),男,硕士,研究方向为水下无人作战使用、水下攻防体系设计
参考文献：
[1] ROGOWSKI P, TERRILL E, CHEN J, et al. Observations of the frontal region of a buoyant river plume using an autonomous underwater vehicle[J]. Journal of Geophysical Research, 2014, 119(11): 7547-7567.
[2] YAN Z , HAO B , LIU Y , et al. Movement control in recovering UUV based on two-stage discrete t-s fuzzy model[J]. Discrete Dynamics in Nature and Society, 2014, 2014: 362787.
[3] 陶伟, 张晓霜. 国外水下无人集群应用及关键技术研究[J]. 舰船电子工程, 2021, 41(2): 9-13+54.
TAO Wei, ZHANG Xiaoshuang. Research on the application and key technologies of underwater unmanned cluster abroad[J]. Ship Electronic Engineering, 2021, 41(2): 9-13+54.
[4] FENG J, YAO Y , WANG H, et al. Multi-AUV terminal guidance method based on underwater visual positioning[C]// 2020 IEEE International Conference on Mechatronics and Automation (ICMA), Beijing, China, 2020: 314–319.
[5] 冯景祥, 姚尧, 潘峰, 等. 国外水下无人装备研究现状及发展趋势[J]. 舰船科学技术, 2021, 43(23): 1-8.
FENG Jingxiang YAO Yao, PAN Feng, et al. Research status and development trends of underwater unmanned equipment abroad[J]. Ship Science and Technology, 2021, 43(23): 1-8.
[6] 冯景祥, 谢飞跃, 张平, 等. 美海上分布式作战研究现状及发展趋势[C]//中国指挥与控制学会. 第九届中国指挥控制大会论文集. 兵器工业出版社, 2021: 150–155.
FENG Jingxiang, XIE Feiyue, ZHANG Ping, et al. Research status and development trends of distributed operations at sea in the united states [C]//Chinese Command and Control Society. Proceedings of the 9th China Command and Control Conference. Ordnance Industry Press, 2021: 150–155.
[7] 李亚哲, 姚尧, 冯景祥, 等. 基于有限状态机的UUV集群围捕策略研究[J]. 舰船电子对抗, 2022, 45(1): 22-27.
LI Yazhe, YAO Yao, FENG Jingxiang, et al. Research on UUV cluster siege strategy based on finite state machine[J]. Ship Electronic Countermeasures, 2022, 45(1): 22-27.
[8] 吴俊成, 周锐, 冉华明, 等. 遗传算法和拍卖算法在任务分配中的性能比较 [J]. 电光与控制, 2016, 23(2): 11–15+82.
WU Juncheng, ZHOU Rui, RAN Huaming, et al. Comparison of performance between genetic algorithm and auction algorithm in task allocation [J] Electro Optics and Control, 2016, 23(2): 11–15+82.
[9] SUN S, SONG B, WANG P, et al. Real-time mission-motion planner for multi-UUVs cooperative work using tri-Level Programing[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23, 1260–1273.
[10] DING Yingying, HE Yan, JIANG Jingping. Multi-robot cooperation method based on the ant algorithm[J]. Robot, 2003, 25(5): 414–418.
[11] ZHAO Wenlai, HU Huosheng. An adaptive task assignment algorithm for a swarm of autonomous underwater vehicles[J]. IEEE Transactions on Robotics, 2016, 32(2), 466–477.
[12] 郝冠捷, 姚尧, 常鹏, 等. 基于深度强化学习的分布式UUV集群任务分配算法[J]. 指挥控制与仿真, 2023, 45(3): 25-33.
HAO Guanjie, YAO Yao, CHANG Peng, et al. Distributed UUV cluster task allocation algorithm based on deep reinforcement learning[J]. Command Control and Simulation, 2023, 45(3): 25-33.
[13] SCHULMAN J, WOLSKI F, DHARIWAL P, et al. Proximal Policy Optimization Alorithms[J]. ARXIV, 2017: 1707.06347.
