Iterative Policy Refinement through Semantic Rollout Analysis
A closed-loop framework iteratively refines structured imitation-learning policies via LLM analysis of rollout tables, improving performance by up to 15% and cutting compute 75%.
Published Oct 1, 2026arXiv ↗
Only vote on papers you've read. Sign in with GitHub to vote.
The framework delivers real gains through iterative diagnostic code and rollout feedback, but relies on toy benchmarks, vague baselines, and unverified compute claims that leave real-world scaling unclear.
Abstract
Structured policies improve efficiency, robustness, and interpretability in imitation learning by introducing task-specific inductive bias, but existing structure generation methods rely either on extensive human input or on static domain knowledge encoded in LLMs, which may be inconsistent with the expert demonstrations. We propose a closed-loop framework that iteratively refines structured policies using LLM-guided analysis of policy rollouts. By logging rollouts as semantically meaningful tabular data and prompting the LLM to generate diagnostic analysis code, our method identifies suboptimalities in the policy structure and iteratively corrects them without requiring human instruction. Experiments on car racing and door opening tasks show that our approach improves imitation learning performance by up to 15% over zero-shot LLM-generated structures and requires 75% less compute to achieve the same reinforcement learning performance. These results demonstrate that tabular rollout analysis provides an effective feedback signal to align LLM-generated policy structures with expert demonstrations, and we can utilize it to generate good policy structures automatically.