HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing

Hybrid Linear Attention introduces query-dependent chunk-level routing for Gated DeltaNet, improving long-context benchmarks by up to 5.57 points via adaptive recurrent memory composition.

Zhuokun Chen, Xi Lin, Xiyu Wu, Jiahao He and 2 more

Published Oct 5, 2026 · ▲ 2 on Hugging Face

Periscope: Extending Frozen Language Models Beyond Their Context Window

Periscope arranges text chunks in a grid to build an evidence map via local and strided probes, letting frozen language models answer questions across multi-million-token contexts with sublinear cost and small GPU memory.

Mohamed Eltahir, Anas Obayd, Raed Rashid, Abdulrahman Alghamdi and 4 more

Published Oct 2, 2026 · ▲ 8 on Hugging Face · Code ★ 1

Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It

Pretrained transformers stop following references after 1.4, 3.6 lines, but a rank-8 LoRA at one early layer extends computation to 50, 160 lines without changing frozen weights.

Zehao Jin, Ruixuan Deng, 君然 王

Published Sep 29, 2026 · 0 citations · ▲ 75 on Hugging Face · Code ★ 3

Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling

Triadic linear attention uses 3D tensor states via triadic outer products to scale recurrent state size efficiently, substantially improving long-context modeling and recall.

Oliver Sieberling, Bharat Runwal, David Jin, Ryan Chin and 2 more

Published Sep 29, 2026 · 0 citations · ▲ 27 on Hugging Face · Code ★ 10

Context Language Models

Context language models treat context as self-modified files to learn context management, outperforming external strategies with lower compute and enabling in-context and parametric learning of management strategies.

Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, Yuetai Li and 9 more

Published Sep 29, 2026 · 0 citations · ▲ 40 on Hugging Face · Code ★ 571

Multilinguality in Hybrid Attention LLMs

Hybrid attention LLMs develop cross-lingual alignment tied to recurrent and full-attention layer ordering, with a spike at the first full-attention layer; distillation shows starting with full attention learns up to 2.5× faster.

Lucas Bandarkar, Junlin Hu, Chenyuan Yang, Mohsen Fayyaz and 1 more

Published Sep 28, 2026 · 0 citations · ▲ 2 on Hugging Face

LongReward: Improving Long-context Large Language Models with AI Feedback

LongReward improves long-context LLMs by applying AI feedback to long-text instruction data via a multi-granularity reward model that evaluates both global coherence and local accuracy.

Jiajie Zhang, Zhongni Hou, Xin Lv, Shulin Cao and 6 more

Published 2025 · 4 citations

Universal and Efficient Computation with 2D Attention

Christos Tzamos, Guoqing Zheng, Athul Jacob

Sydney Poster Session 6, Thu, Dec 10, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Scaling Limits of Long-Context Transformers

Giuseppe Bruno, Chen, Zhengjiang Lin, Yury Polyanskiy and 1 more

Sydney Poster Session 4, Wed, Dec 9, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Short-Context Dominance: How Much Local Context Natural Language Actually Needs?

Vala Vakilian, Zimeng Wang, Ankit Rawat, Christos Thrampoulidis

Sydney Poster Session 4, Wed, Dec 9, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Breaking the Exactness Barrier: Interleaved DeepSeek Sparse Attention for Efficient Long Context Reasoning

Yifan GUO, Wei Cui

Sydney Poster Session 6, Thu, Dec 10, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Diffusion Language Models Can Approximate Optimal Infilling Lengths Implicitly

Hengchang Liu, Zhao Yang, Bing Su

Atlanta Poster Session 2, Wed, Dec 9, 4:30 PM–7:30 PM, Hall C1 · Published 2026

AsdaKV: Attention-Overlap Driven Semantic KV Retrieval for Long-Context LLMs

Tianming Yan, Kun Yang, Tingwang You, Kui Ren

Sydney Poster Session 4, Wed, Dec 9, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Raven: High-Recall Sequence Modeling via Sparse Memory Routing

Arshia Afzal, Aviv Bick, Eric Xing, Volkan Cevher and 1 more

Sydney Poster Session 5, Thu, Dec 10, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

C3: Long-Horizon Character Consistency via Causal-Continuous State Dynamics and Memory Rewriting

Minghao Chen

Sydney Poster Session 4, Wed, Dec 9, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Necessary but Not Sufficient: Spectral Tests for Value-Linear Attention Surrogates

Blaž Škrlj, Ivan Can Arisoy, Solal Vernier

Paris Poster Session 6, Fri, Dec 11, 2:30 PM–4:30 PM, Paris Poster Hall · Published 2026

Context as Low-Rank Weights: Bounded Parametric Dynamic Memory for Unbounded Context

Xiangyu Zhang, Yu Zhou, Taolue Chen

Sydney Poster Session 2, Tue, Dec 8, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

State-Resolving Attention for Length-Extrapolating Transformers

Zicheng Liu, Di Wu, Jintao Chen, Di Huang

Sydney Poster Session 6, Thu, Dec 10, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

CutAttn: Discovering Cognitive Transition Layers for Efficient Long-Context Prefilling

Wentao Liu, Xiabao Wu, Yongchao Liu, Haitao Zhang and 2 more

Sydney Poster Session 1, Tue, Dec 8, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

The Missing Positional Story in LLMs: A Case Study of Shift-Invariant Attention

Benjamin Huh, Hak Hyun Kim, Yuting Tian, Jason Peng and 2 more

Atlanta Poster Session 2, Wed, Dec 9, 4:30 PM–7:30 PM, Hall C1 · Published 2026