CENTRALIZED LEARNING FOR THE DEEP Q-LEARNING MODELS
DOI:
https://doi.org/10.32689/maup.it.2024.2.1Keywords:
deep Q-learning, reinforcement learning, knowledge distillation, exchange of knowledge, centralized trainingAbstract
The article is devoted to centralized learning and knowledge sharing between Deep Q-leaning agents. Multi- agent systems are fault-tolerant and capable of self-organization, but achieving this can require a lot of resources. The agent independently explores the environment, gradually adapting to different situations. For systems where the state space is continuous, and therefore has many options, and the outcome of the transition in the future is unknown, it is difficult for the agent to choose to explore the space of actions and states, select a more profitable strategy and not get stuck in pseudo-winning strategies (local minima). The goal is to increase the stability of the learning process. On the example of the MADDPG approach and the KnowSR framework, the following methodology was proposed: to use several agents that exchange experience and knowledge between models, forming a common buffer. The scientific novelty is the use of centralized learning to increase the stability of actions of Deep Q learning agents with a mechanism for sharing already learned knowledge.
References
Eysenbach B., Kumar A. Reinforcement learning is supervised learning on optimized data. The BAIR Blog. 2020. February 1, 2024, Retrieved from https://bair.berkeley.edu/blog/2020/10/13/supervised-rl/
GaoZ.,XuK.,DingB.,WangH.,LiY.,JiaH.KnowSR:KnowledgeSharingamongHomogeneousAgentsinMulti-agent Reinforcement Learning. 2021. (arXiv preprint arXiv:2105.11611).
Hinton,Geoffrey;Vinyals,Oriol;Dean,Jeff(2015).«Distillingtheknowledgeinaneuralnetwork».arXiv:1503.02531
Leitão, Paulo; Karnouskos, Stamatis (March 26, 2015). Industrial agents: emerging applications of software agents in industry. Leitão, Paulo, Karnouskos, Stamatis. Amsterdam, Netherlands. ISBN 978-0128003411. OCLC 905853947.
M. Brambilla, E. Ferrante, M. Birattari and M. Dorigo, «Swarm robotics: A review from the swarm engineering perspective», Swarm Intell., vol. 7, no. 1, pp. 1-41, 2013.
M.Dorigo,G.TheraulazandV.Trianni,«Reflectionsonthefutureofswarmrobotics»,Sci.Robot.,vol.5,no.49,2020. 7. Mnih V. et al. Playing atari with deep reinforcement learning //arXiv preprint arXiv:1312.5602. 2013.
Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (2nd edition). 2020.
Stefano V. Albrecht, Filippos Christianos, Lukas Schäfer. Multi-Agent Reinforcement Learning: Foundations and
Modern Approaches. MIT Press, 2024. https://www.marl-book.com/
Wooldridge, Michael. An Introduction to MultiAgent Systems. John Wiley & Sons. 2002. p. 366. ISBN 978-0-
-49691-5.
