Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories
Safety-aligned LLMs remain vulnerable to mid-generation token injections altering subsequent outputs, and trajectory-level alignment improves robustness beyond shallow token defenses.
Published 2026Sydney Poster Session 3 · Wed, Dec 9, 10:00 AM–1:00 PM local time · Hall 1-4arXiv ↗OpenReview ↗
Only vote on papers you've read. Sign in with GitHub to vote.
This must-read paper proves hidden-state refusal alignment fails to predict injection robustness and advances trajectory-level training as a transferable process fix, though critics caution its simulated benchmarks leave real-world deployment and toxicity costs untested.
Abstract
Safety-aligned Large Language Models (LLMs) remain vulnerable to interventions during inference that redirect generation toward harmful outputs. Recent work attributes this to shallow safety, where alignment concentrates in the first few output tokens. We show that shallow safety is a special case of a broader inference-time vulnerability, in which short token injections at any generation step can substantially alter subsequent safety behavior. We also find that a model's alignment with refusal directions in its hidden states does not predict its robustness to such injection, revealing that internal state alone does not determine generation behavior under perturbation. To address this, we align models directly on generation trajectories constructed by simulating mid-sequence perturbation, and show that this improves robustness to mid-sequence injection and generalizes to attacks that exploit early-token generation. Our work argues that robust safety alignment requires training on the generation process itself, not only its outputs.