Collaboration is indispensable for solving complex tasks. Learning to collaborate effectively is one of the key problems in artificial intelligence. Cooperative multi-agent reinforcement learning (MARL) potentially provides a promising solution, but faces two fundamental challenges: scalability and credit assignment. In this talk, I will discuss a MARL paradigm with factored value functions to address these challenges. I will first present formal analysis on factored value learning, revealing its implicit credit assignment mechanism and properties of convergence and optimality. Inspired by these theoretical insights, two novel MARL methods will then be introduced with linear and non-linear value factorization, respectively, which achieves state-of-the-art performance. Building on factored MARL, I will also briefly discuss approaches for addressing other challenges of cooperative MARL, such as learning efficiency, partial observability, and exploration.
