Computer Science and Information Systems 2024 Volume 21, Issue 1, Pages: 335-362
https://doi.org/10.2298/CSIS221210071A
Full text (
3320 KB)
Knowledge transfer in multi-objective multi-agent reinforcement learning via generalized policy improvement
de Almeida Vicente N. (Instituto de Informática, Universidade Federal do Rio Grande do Sul (UFRGS) Porto Alegre, Brazil), [email protected]
Alegre Lucas N. (Instituto de Informática, Universidade Federal do Rio Grande do Sul (UFRGS) Porto Alegre, Brazil), [email protected]
Bazzan Ana L.C. (Instituto de Informática, Universidade Federal do Rio Grande do Sul (UFRGS) Porto Alegre, Brazil), [email protected]
Even though many real-world problems are inherently distributed and multi-objective, most of the reinforcement learning (RL) literature deals with single agents and single objectives. While some of these problems can be solved using a single-agent single-objective RL solution (e.g., by specifying preferences over objectives), there are robustness issues, as well the fact that preferences may change over time, or it might not even be possible to set such preferences. Therefore, a need arises for a way to train multiple agents for any given preference distribution over the objectives. This work thus proposes a multi-objective multi-agent reinforcement learning (MOMARL) method in which agents build a shared set of policies during training, in a decentralized way, and then combine these policies using a generalization of policy improvement and policy evaluation (fundamental operations of RL algorithms) to generate effective behaviors for any possible preference distribution, without requiring any additional training. This method is applied to two different application scenarios: a multi-agent extension of a domain commonly used in the related literature, and traffic signal control, which is more complex, inherently distributed and multi-objective (the flow of both vehicles and pedestrians are considered). Results show that the approach is able to effectively and efficiently generate behaviors for the agents, given any preference over the objectives.
Keywords: reinforcement learning, multi-agent systems, multi-objective decision making, generalized policy improvement, traffic signal control
Show references
Abels, A., Roijers, D.M., Lenaerts, T., Nowé, A., Steckelmacher, D.: Dynamic weights in multiobjective deep reinforcement learning. In: Proceedings of the 36th International Conference on Machine Learning. vol. 97, pp. 11-20. International Machine Learning Society (IMLS) (2019)
Alegre, L.N.: SUMO-RL. https://github.com/LucasAlegre/sumo-rl (2019)
Alegre, L.N., Bazzan, A.L.C., da Silva, B.C.: Optimistic linear support and successor features as a basis for optimal policy transfer. In: Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., Sabato, S. (eds.) Proceedings of the 39th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 162, pp. 394-413. PMLR (17-23 Jul 2022), https://proceedings.mlr.press/v162/alegre22a.html
Barreto, A., Dabney, W., Munos, R., Hunt, J.J., Schaul, T., van Hasselt, H.P., Silver, D.: Successor features for transfer in reinforcement learning. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017)
Barreto, A., Hou, S., Borsa, D., Silver, D., Precup, D.: Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences 117(48), 30079-30087 (2020)
Bazzan, A.L.C.: Opportunities for multiagent systems and multiagent reinforcement learning in traffic control. Autonomous Agents and Multiagent Systems 18(3), 342-375 (June 2009)
Bellman, R.E.: Dynamic Programming. Princeton University Press, Princeton (1957)
Borsa, D., Barreto, A., Quan, J., Mankowitz, D.J., Munos, R., Hasselt, H.V., Silver, D., Schaul, T.: Universal successor features approximators. In: Proceedings of the 7th International Conference on Learning Representations (ICLR) (2019)
Bus，oniu, L., Babuska, R., De Schutter, B.: A comprehensive survey of multiagent reinforcement learning. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on 38(2), 156-172 (2008)
Cheng, H.T.: Algorithms for partially observable Markov decision processes. Ph.D. thesis, University of British Columbia (1988), https://open.library.ubc.ca/ collections/ubctheses/831/items/1.0098252
Dayan, P.: Improving generalization for temporal difference learning: The successor representation. Neural Computation 5(4), 613-624 (1993)
Duan, H., Li, Z., Zhang, Y.: Multiobjective reinforcement learning for traffic signal control using vehicular ad hoc network. EURASIP Journal on Advances in Signal Processing 2010 (12 2010)
Egea, A.C., Connaughton, C.: Assessment of reward functions in reinforcement learning for multi-modal urban traffic control under real-world limitations (2020), arXiv preprint arXiv:2010.08819
Fedus, W., Ramachandran, P., Agarwal, R., Bengio, Y., Larochelle, H., Rowland, M., Dabney, W.: Revisiting fundamentals of experience replay. In: Proceedings of the 37th International Conference on Machine Learning. Vienna, Austria (2020)
Gimelfarb, M., Barreto, A., Sanner, S., Lee, C.G.: Risk-aware transfer in reinforcement learning using successor features. In: Proceedings of the 35th Annual Conference on Advances in Neural Information Processing Systems. Online (2021)
Gupta, T., Mahajan, A., Peng, B., Böhmer, W., Whiteson, S.: Uneven: Universal value exploration for multi-agent reinforcement learning (2021), arXiv preprint arXiv:2010.02974
Hayes, C.F., Rădulescu, R., Bargiacchi, E., Källström, J., Macfarlane, M., Reymond, M., Verstraeten, T., Zintgraf, L.M., Dazeley, R., Heintz, F., Howley, E., Irissappane, A.A., Mannion, P., Nowé, A., Ramos, G., Restelli, M., Vamplew, P., Roijers, D.M.: A practical guide to multiobjective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems 36(1), 26 (Apr 2022), https://doi.org/10.1007/s10458-022-09552-y
Khamis, M.A., Gomaa, W.: Enhanced multiagent multi-objective reinforcement learning for urban traffic light control. In: 2012 11th International Conference on Machine Learning and Applications. vol. 1, pp. 586-591 (2012)
Kim, S.H., Stralen, N.V., Chowdhary, G., Tran, H.T.: Disentangling successor features for coordination in multi-agent reinforcement learning. In: International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022. pp. 751-760 (2022)
Kober, J., Bagnell, J.A., Peters, J.: Reinforcement learning in robotics: A survey. The International Journal of Robotics Research 32(11), 1238-1274 (2013)
Liu,W., Niu, D., Dong, L., Sun, C.: Efficient exploration for multi-agent reinforcement learning via transferable successor features. IEEE/CAA Journal of Automatica Sinica 9 (2022)
Lopez, P.A., Behrisch, M., Bieker-Walz, L., Erdmann, J., Flötteröd, Y.P., Hilbrich, R., Lücken, L., Rummel, J., Wagner, P., Wiesner, E.: Microscopic traffic simulation using SUMO. In: The 21st IEEE International Conference on Intelligent Transportation Systems (2018)
Mazyavkina, N., Sviridov, S., Ivanov, S., Burnaev, E.: Reinforcement learning for combinatorial optimization: A survey. Computers and Operations Research 134, 105400 (2021)
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529-533 (Feb 2015)
Noaeen, M., Naik, A., Goodman, L., Crebo, J., Abrar, T., Far, B., Abad, Z.S.H., Bazzan, A.L.C.: Reinforcement learning in urban network traffic signal control: A systematic literature review (2021), engrxiv.org/ewxrj
Rˇadulescu, R., Mannion, P., Roijers, D., Nowé, A.: Multi-objective multi-agent decision making: a utility-based analysis and survey. Autonomous Agents and Multi-Agent Systems 34 (04 2020)
Roess, R.P., Prassas, E.S., McShane, W.R.: Traffic Engineering. Prentice Hall, 3rd edn. (2004)
Roijers, D.: Multi-Objective Decision-Theoretic Planning. Ph.D. thesis, University of Amsterdam (2016)
Roijers, D.M., Vamplew, P., Whiteson, S., Dazeley, R.: A survey of multi-objective sequential decision-making. J. Artificial Intelligence Research 48(1), 67-113 (Oct 2013)
Shapley, L.S.: Stochastic games. Proceedings of the National Academy of Sciences 39(10), 1095-1100 (1953)
Silva, F.L.d., Costa, A.H.R.: A survey on transfer learning for multiagent reinforcement learning systems. Journal of Artificial Intelligence Research 64, 645-703 (2019)
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T., Simonyan, K., Hassabis, D.: Mastering chess and shogi by self-play with a general reinforcement learning algorithm (2017)
Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction. The MIT Press, second edn. (2018)
Vamplew, P., Foale, C., Dazeley, R.: The impact of environmental stochasticity on value-based multiobjective reinforcement learning. Neural Computing and Applications (Mar 2021)
Vamplew, P., Yearwood, J., Dazeley, R., Berry, A.: On the limitations of scalarisation for multiobjective reinforcement learning of pareto fronts. In: Wobcke, W., Zhang, M. (eds.) AI 2008: Advances in Artificial Intelligence. pp. 372-378. Springer, Berlin, Heidelberg (2008)
Wei, H., Zheng, G., Gayah, V.V., Li, Z.: A survey on traffic signal control methods (2020), http://arxiv.org/abs/1904.08117, preprint arXiv:1904.08117
Yang, R., Sun, X., Narasimhan, K.: A generalized algorithm for multi-objective reinforcement learning and policy adaptation. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché- Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32. pp. 14610-14621 (2019)
Yau, K.L.A., Qadir, J., Khoo, H.L., Ling, M.H., Komisarczuk, P.: A survey on reinforcement learning models and algorithms for traffic signal control. ACM Comput. Surv. 50(3) (2017)
Yin, B., Menendez, M.: A reinforcement learning method for traffic signal control at an isolated intersection with pedestrian flows. pp. 3123-3135 (07 2019)
Yu, C., Liu, J., Nemati, S., Yin, G.: Reinforcement learning in healthcare: A survey. ACM Comput. Surv. 55(1) (nov 2021)