Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces
Actor-accelerated PDA learns a policy network to approximate PDA optimization subproblems, speeding up continuous-action reinforcement learning while preserving convergence guarantees and outperforming PPO.
Published 2026Atlanta Poster Session 4 · Thu, Dec 10, 4:30 PM–7:30 PM local time · Hall C1arXiv ↗OpenReview ↗
Only vote on papers you've read. Sign in with GitHub to vote.
Actor-accelerated PDA rigorously extends mirror-descent theory to continuous actions with convergence guarantees and strong PPO benchmarks, yet its practical claims remain undermined by missing ablations of actor approximation error, unspecified dataset counts, and unavailable code.
Abstract
Policy Dual Averaging (PDA) offers a principled Policy Mirror Descent (PMD) framework that more naturally admits value function approximation than standard PMD, enabling the use of approximate advantage (or Q-) functions while retaining strong convergence guarantees. However, applying PDA in continuous state and action spaces remains computationally challenging, since action selection involves solving an optimization sub-problem at each decision step. In this paper, we propose \textit{actor-accelerated PDA}, which uses a learned policy network to approximate the solution of the optimization sub-problems, yielding faster runtimes while maintaining convergence guarantees. We provide a theoretical analysis that quantifies how actor approximation error impacts the convergence of PDA under suitable assumptions. We then evaluate its performance on several benchmarks in robotics, control, and operations research problems. Actor-accelerated PDA achieves superior performance compared to popular on-policy baselines such as Proximal Policy Optimization (PPO). Overall, our results bridge the gap between the theoretical advantages of PDA and its practical deployment in continuous-action problems with function approximation.