HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing

Hybrid Linear Attention introduces query-dependent chunk-level routing for Gated DeltaNet, improving long-context benchmarks by up to 5.57 points via adaptive recurrent memory composition.

Zhuokun Chen, Xi Lin, Xiyu Wu, Jiahao He and 2 more

Published Oct 5, 2026 · ▲ 2 on Hugging Face

Periscope: Extending Frozen Language Models Beyond Their Context Window

Periscope arranges text chunks in a grid to build an evidence map via local and strided probes, letting frozen language models answer questions across multi-million-token contexts with sublinear cost and small GPU memory.

Mohamed Eltahir, Anas Obayd, Raed Rashid, Abdulrahman Alghamdi and 4 more

Published Oct 2, 2026 · ▲ 8 on Hugging Face · Code ★ 1

Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It

Pretrained transformers stop following references after 1.4, 3.6 lines, but a rank-8 LoRA at one early layer extends computation to 50, 160 lines without changing frozen weights.

Zehao Jin, Ruixuan Deng, 君然 王

Published Sep 29, 2026 · 0 citations · ▲ 75 on Hugging Face · Code ★ 3

Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling

Triadic linear attention uses 3D tensor states via triadic outer products to scale recurrent state size efficiently, substantially improving long-context modeling and recall.

Oliver Sieberling, Bharat Runwal, David Jin, Ryan Chin and 2 more

Published Sep 29, 2026 · 0 citations · ▲ 27 on Hugging Face · Code ★ 10

Context Language Models

Context language models treat context as self-modified files to learn context management, outperforming external strategies with lower compute and enabling in-context and parametric learning of management strategies.

Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, Yuetai Li and 9 more

Published Sep 29, 2026 · 0 citations · ▲ 43 on Hugging Face · Code ★ 595

Multilinguality in Hybrid Attention LLMs

Hybrid attention LLMs develop cross-lingual alignment tied to recurrent and full-attention layer ordering, with a spike at the first full-attention layer; distillation shows starting with full attention learns up to 2.5× faster.

Lucas Bandarkar, Junlin Hu, Chenyuan Yang, Mohsen Fayyaz and 1 more

Published Sep 28, 2026 · 0 citations · ▲ 2 on Hugging Face

LongReward: Improving Long-context Large Language Models with AI Feedback

LongReward improves long-context LLMs by applying AI feedback to long-text instruction data via a multi-granularity reward model that evaluates both global coherence and local accuracy.

Jiajie Zhang, Zhongni Hou, Xin Lv, Shulin Cao and 6 more

Published 2025 · 4 citations

Universal and Efficient Computation with 2D Attention

Christos Tzamos, Guoqing Zheng, Athul Jacob

Sydney Poster Session 6, Thu, Dec 10, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Scaling Limits of Long-Context Transformers

Giuseppe Bruno, Chen, Zhengjiang Lin, Yury Polyanskiy and 1 more

Sydney Poster Session 4, Wed, Dec 9, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Short-Context Dominance: How Much Local Context Natural Language Actually Needs?

Vala Vakilian, Zimeng Wang, Ankit Rawat, Christos Thrampoulidis

Sydney Poster Session 4, Wed, Dec 9, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Breaking the Exactness Barrier: Interleaved DeepSeek Sparse Attention for Efficient Long Context Reasoning

Yifan GUO, Wei Cui

Sydney Poster Session 6, Thu, Dec 10, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Diffusion Language Models Can Approximate Optimal Infilling Lengths Implicitly

Hengchang Liu, Zhao Yang, Bing Su

Atlanta Poster Session 2, Wed, Dec 9, 4:30 PM–7:30 PM, Hall C1 · Published 2026

AsdaKV: Attention-Overlap Driven Semantic KV Retrieval for Long-Context LLMs

Tianming Yan, Kun Yang, Tingwang You, Kui Ren

Sydney Poster Session 4, Wed, Dec 9, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Raven: High-Recall Sequence Modeling via Sparse Memory Routing

Arshia Afzal, Aviv Bick, Eric Xing, Volkan Cevher and 1 more

Sydney Poster Session 5, Thu, Dec 10, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

C3: Long-Horizon Character Consistency via Causal-Continuous State Dynamics and Memory Rewriting

Minghao Chen

Sydney Poster Session 4, Wed, Dec 9, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Necessary but Not Sufficient: Spectral Tests for Value-Linear Attention Surrogates

Blaž Škrlj, Ivan Can Arisoy, Solal Vernier

Paris Poster Session 6, Fri, Dec 11, 2:30 PM–4:30 PM, Paris Poster Hall · Published 2026

Context as Low-Rank Weights: Bounded Parametric Dynamic Memory for Unbounded Context

Xiangyu Zhang, Yu Zhou, Taolue Chen

Sydney Poster Session 2, Tue, Dec 8, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

State-Resolving Attention for Length-Extrapolating Transformers

Zicheng Liu, Di Wu, Jintao Chen, Di Huang

Sydney Poster Session 6, Thu, Dec 10, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

CutAttn: Discovering Cognitive Transition Layers for Efficient Long-Context Prefilling

Wentao Liu, Xiabao Wu, Yongchao Liu, Haitao Zhang and 2 more

Sydney Poster Session 1, Tue, Dec 8, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

The Missing Positional Story in LLMs: A Case Study of Shift-Invariant Attention

Benjamin Huh, Hak Hyun Kim, Yuting Tian, Jason Peng and 2 more

Atlanta Poster Session 2, Wed, Dec 9, 4:30 PM–7:30 PM, Hall C1 · Published 2026

GEAR: Generator-Adaptive State Space Models for Associative Recall

Jun Meng, Mohammadhossein Amouei, Zengyu Lin, Xinyu Hu and 1 more

Sydney Poster Session 2, Tue, Dec 8, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

TrajLift: Encoding Verbal Memory Dynamics via Heat Diffusion on Semantic Hierarchies

Jiawen Kang, Jinchao Li, Mingyu Cui, Junan Li and 2 more

Sydney Poster Session 4, Wed, Dec 9, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

DeltaFugue: Orchestrating Spatial and Associative Memory for Algorithmic Length Generalization

Anh T Nguyen, Quan Dao, Bing Liu

Atlanta Poster Session 5, Fri, Dec 11, 10:00 AM–1:00 PM, Hall C1 · Published 2026

Can 4D Foundation Models Remember?

Guangzhao He, Hadar Averbuch-Elor, Wei-Chiu Ma

Sydney Poster Session 3, Wed, Dec 9, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

Resilient Latent Readouts for Long-Context Question Answering

Jingyi Liao, Wenhao Sun, YITING LI, Zhao Jin and 5 more

Sydney Poster Session 1, Tue, Dec 8, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

Prefix Likelihood-Ratio Control: Tail-Stable Training for Long-Horizon Language Generation

Yangyang Liu

Sydney Poster Session 2, Tue, Dec 8, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

RoPE Is Not a Proper Relative Position Embedding

Yui Oka, Kyosuke Nishida, Sho Yokoi

Sydney Poster Session 4, Wed, Dec 9, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Tree Rotary Positional Encoding for Extreme Length Extrapolation from Scratch

qiu wu chen, Ziteng Huang, Shuhai Zhang, Zimo Liu and 6 more

Sydney Poster Session 5, Thu, Dec 10, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

CHARM+: Cross-Hardware Attention with Re-Merge Multistream Mechanism

Xinhang Zhang, boning zhang, Chengchun Liu, Chunpu Li and 3 more

Sydney Poster Session 6, Thu, Dec 10, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Low-Rank Hierarchical Merging for Efficient Long-to-Short Reasoning

Zeqiu Yu, Xuesheng Zhang, Wenxiao Zhao, Shixiao Wang

Atlanta Poster Session 4, Thu, Dec 10, 4:30 PM–7:30 PM, Hall C1 · Published 2026

Long-Context Language Models Require Extreme Sparsity in Context Dimension

Prithvi Dixit, Sahil Joshi, Agniva Chowdhury, Anshumali Shrivastava and 4 more

Atlanta Poster Session 1, Wed, Dec 9, 10:00 AM–1:00 PM, Hall C1 · Published 2026

AdaCal: Adaptive Calibration for Robust Sparse Attention in Long-Context LLMs

Leqin Xiang, Yongbin Liu, Chunping Ouyang, Ying Yu

Sydney Poster Session 6, Thu, Dec 10, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale

Siddharth Gollapudi, Prasann Singhal, Nilesh Gupta, Sewon Min

Sydney Poster Session 3, Wed, Dec 9, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

Language Model Memory and Memory Models for Language

Language model embeddings retain little input information, but autoencoders achieve near-perfect memory; combined causal and retention objectives enable rich, decodable memory formation for efficient encoder-decoder architectures.

Benjamin L Badger

Sydney Poster Session 4, Wed, Dec 9, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Dual Dimensionality for Local and Global Attention

Distance-Adaptive Representation uses high-dimensional local and low-dimensional distant keys and values to cut KV cache size while matching full-dimensional baseline performance.

Zhiyuan Wang, Xuan Luo, Sirui Zeng, Xifeng Yan

Atlanta Poster Session 6, Fri, Dec 11, 4:30 PM–7:30 PM, Hall C1 · Published 2026

When Less is More: The LLM Scaling Paradox in Context Compression

Under lossy context compression, larger compressors reduce reconstruction error but increase unfaithfulness via knowledge overwriting and semantic drift, violating scaling laws for faithful preservation.

Ruishan Guo, Yibing Liu, Guoxin Ma, Yan Wang and 5 more

Sydney Poster Session 5, Thu, Dec 10, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

MemDLM: Memory-Enhanced DLM Training

MemDLM augments diffusion language model training via bi-level optimization with parametric memory, improving convergence, long-context representations, and needle retrieval.

Zehua Pei, Hui-Ling Zhen, Weizhe Lin, Sinno Pan and 3 more

Sydney Poster Session 5, Thu, Dec 10, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026 · ▲ 3 on Hugging Face · Code ★ 11

Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

Gated DeltaNet-2 decouples erase and write with channel-wise gates to improve linear attention, achieving top performance among recurrent and hybrid models at 1.3B scale with strong long-context retrieval.

Ali Hatamizadeh, Yejin Choi, Jan Kautz

Atlanta Poster Session 1, Wed, Dec 9, 10:00 AM–1:00 PM, Hall C1 · Published 2026 · ▲ 30 on Hugging Face · Code ★ 326

StateTree: Enhancing Long-term Dialogue Reasoning via Reinforcement Learning

StateTree uses RL on tree-structured multi-session path-tracing tasks to train long-context dialogue reasoning, achieving up to +23.60% gains on LongMemEval and outperforming larger baselines while preserving short-context reasoning.

Naen Xu, Wanqing Cui, Yibo Hu, Shixin Hong and 4 more

Sydney Poster Session 6, Thu, Dec 10, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Learning What to Remember: Test-Time Training via Context Distillation

TTCD uses a long-window teacher to supervise a short-window student's fast weights via hidden-state discrepancy, allocating limited memory to future-relevant context and outperforming existing long-context methods with minimal architectural changes.

Zixuan Wang, Xingyu Dang, Rui-Jie Zhu, Zixin Wen and 3 more

Atlanta Poster Session 3, Thu, Dec 10, 10:00 AM–1:00 PM, Hall C1 · Published 2026