Decision Titan: Test-Time Training for Long-Term Memory in Offline Reinforcement Learning
Decision Titan applies test-time training to offline RL, enabling long-term dependencies 20x beyond context windows and 1.7x length generalization while revealing time embeddings and encoding as critical factors.
Published Oct 1, 2026arXiv ↗
Only vote on papers you've read. Sign in with GitHub to vote.
Decision Titan delivers striking long-range memory and explicit parameter-storage mechanics via gate visualizations, but its claims rest entirely on X-Maze, lack RL benchmarks, and leave critical computational costs unmeasured.
Abstract
Long-term dependencies remain a major challenge for sequential decision-making in the field of AI: RNNs suffer from vanishing gradients and the limited expressivity of vector-based hidden states, whilst Transformer-based models are limited by the quadratic scaling of attention. Recent work has proposed tackling this problem with the Test-Time Training (TTT) framework, which stores episodic memories in the parameters of a neural network through gradient descent at both train and test-time. This approach has seen success in the domain of Natural Language Processing, however, to the best of our knowledge it has not yet been applied to the domain of Reinforcement Learning (RL), nor has there been a study analysing how this memory practically functions. In this paper, we study the potential of the TTT framework for offline RL by augmenting a Decision Transformer with TTT layers, dubbed the Decision Titan. We analyse performance and properties of the model in the X-Maze environment, an extension of T-Maze designed to test sequential memory, and investigate how the memory mechanism learns by visualising gate values over time. Our key findings are that Decision Titan can learn long-term dependencies with ranges 20x longer than the context window, generalises to lengths 1.7x the training data, but crucially temporal generalisation depends on the time embeddings used, and the ability to learn long-term dependencies depends on how the relevant information is encoded.