Towards Looped Models Done Right, Part II: Rethinking at Fixed Points

Looped language models use fixed-point convergence to enable truncated training, shared KV caches, faster prefill, and faster RL updates, while a learned depth prior and orthogonal input injection improve perplexity across scales.

Benhao Huang, Chufan Shi, Junlin Chen, Shicheng Wen and 3 more

Published Oct 5, 2026 · ▲ 17 on Hugging Face · Code ★ 30

The Numerical Linear Algebra of Large Language Models

This survey explains large language model core concepts to numerical analysts and highlights key numerical linear algebra contributions to LLM techniques.

Abdelkader Baggag, Yousef Saad

Published Oct 3, 2026 · ▲ 4 on Hugging Face

Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale

Fixed token codes can replace trainable input embeddings in 1.7B-scale language models, removing 100.7M parameters while preserving substantial capabilities without requiring token-specific vectors.

A. Bochkov

Published Oct 2, 2026

Cross-Lingual Alignment for Decoder-Only Models using MoE Routers

Cross-lingual MoE router alignment improves multilingual LLM performance by aligning router outputs across languages instead of hidden states.

Lucas Bandarkar, Clark Peng, Ahmed Haj Ahmed, Aditi Khandelwal and 1 more

Published Oct 1, 2026 · 0 citations · ▲ 1 on Hugging Face · Code

Scaling and Distilling Text Embeddings for Better Diffusibility

Scaling and distilling text embeddings improves latent diffusion by yielding more connected, diffusible spaces that boost generative performance beyond autoregressive baselines.

Zekai Zhang, Yunjie Tian, Yanjin He, Xiaoyan Zhang and 3 more

Published Oct 1, 2026 · 0 citations · ▲ 57 on Hugging Face · Code ★ 3

Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts

LOOM stabilizes looped MoE via bounded residual updates and per-loop routers to scale loops to 9, 12, cutting perplexity from 9.62 to 7.77.

Di He, Pengxiang Li, Da Chang, Qingyan Meng and 2 more

Published Oct 1, 2026 · 0 citations · ▲ 16 on Hugging Face · Code ★ 8

Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models

Clock Diffusion introduces semi-autoregressive continuous diffusion language models with position-dependent noise schedules, efficient training and sampling, and Cache Grab acceleration to achieve state-of-the-art diffusion likelihoods and competitive reasoning performance.

Yair Schiff, Omer Belhasin, Roy Uziel, Matan Rusanovsky and 6 more

Published Oct 1, 2026 · 0 citations

E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models

E-MoE improves few-step non-factorized diffusion language models via Mixture-of-Experts routing as a discrete shared latent, boosting sample quality without extra active parameters.

Arseny Ivanov, Alexander Kolesov, Alexander Korotin, Ivan Oseledets and 1 more

Published Sep 29, 2026 · 0 citations · ▲ 66 on Hugging Face

How to Loop MoE: Flatten the Experts, Untie the Attention

Foil improves looped MoE by flattening experts and untying attention, reducing pretraining loss by 0.012 nat and improving routing balance and confidence.

Shouren Wang, Chuang Ma, Mohsen Hariri, Debargha Ganguly and 5 more

Published Sep 28, 2026 · 0 citations · ▲ 9 on Hugging Face · Code ★ 2

Kimi K3: Open Frontier Intelligence

Kimi K3 is a 2.8 trillion-parameter Mixture-of-Experts model with native vision and 1-million-token context that achieves frontier performance across reasoning, coding, and agentic tasks and outperforms comparable open and proprietary models.

Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao and 36 more

Published Jul 27, 2026 · 0 citations · ▲ 525 on Hugging Face · Code ★ 8,900

Scaling Participation in Modular AI Systems

Modular participatory AI combines small stakeholder-trained models into compositional systems that outperform monolithic LLMs by up to 15.4% and exhibit emergent collaborative capabilities.

Shangbin Feng, Yike Wang, Weijia Shi, Luke Zettlemoyer and 2 more

Published Jun 5, 2026 · 0 citations

HRM-Text: Efficient Pretraining Beyond Scaling

HRM-Text replaces Transformers with a hierarchical recurrent model and trains on instruction pairs to achieve competitive 1B-parameter performance with 100, 900x fewer tokens and far less compute.

Guan Wang, Changling Liu, Chenyu Wang, Cai Zhou and 5 more

Published May 20, 2026 · 0 citations · ▲ 322 on Hugging Face · Code ★ 2,134

DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models

DataFlex unifies sample selection, mixture adjustment, and reweighting for LLMs via a modular LLaMA-Factory framework that improves MMLU and perplexity with faster runtimes.

Hao Liang, Zhengyang Zhao, Mingrui Chen, Meiyi Qiang and 21 more

Published Mar 27, 2026 · 0 citations · ▲ 279 on Hugging Face · Code ★ 2,958

Attention Residuals

Attention Residuals replace fixed residual accumulation with softmax attention over previous layer outputs for selective, input-dependent aggregation, improving scaling and downstream performance with minimal overhead.

Kimi Team, Guangyu Chen, Yu Zhang, Jianlin Su and 33 more

Published Mar 16, 2026 · 0 citations · ▲ 198 on Hugging Face · Code ★ 3,515

OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration

OPUS defines optimizer-induced update-space data utility for dynamic LLM pre-training selection, outperforming full-scale baselines with minimal overhead.

Shaobo Wang, Xuan Ouyang, Tianyi Xu, Yuzheng Hu and 8 more

Published Feb 5, 2026 · 0 citations · ▲ 354 on Hugging Face

A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM𝛥 Integration into Upcycled MoE

The method expands multilingual LLMs via post-training PARAMΔ integration into upcycled MoE for data-efficient language acquisition.

Hao Zhou, Tianhao Li, Zhijun Wang, Shuaijie She and 5 more

Published 2026 · 0 citations

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion

Rotation symmetry generalizes permutation symmetry continuously for transformers, improving parameter matching and model fusion across language and vision tasks.

Binchi Zhang, Zaiyi Zheng, Zhengzhang Chen, Jundong Li

Published Feb 1, 2025 · 0 citations

Uncovering Scaling Laws for Large Language Models via Inverse Problems

Inverse problem methods uncover scaling laws for large language models, revealing predictive relationships between model size, data, and performance from abstract evidence.

Arun Verma, Zhaoxuan Wu, Zijian Zhou, Xiaoqiang Lin and 14 more

Published 2025 · 0 citations

Position Paper: Data-Centric AI in the Age of Large Language Models

This position paper argues for data-centric AI in the age of large language models and proposes research directions.

Xinyi Xu, Zhaoxuan Wu, Rui Qiao, Arun Verma and 15 more

Published 2024 · 2 citations

LMBot: Distilling Graph Knowledge into Language Model for Graph-less Deployment in Twitter Bot Detection

LMBot distills graph neural network knowledge into language models for efficient graph-less Twitter bot detection, achieving state-of-the-art results across four benchmarks.

Cai, Zijian, Zhaoxuan Tan, Zhenyu Lei, Zhu, Zifeng and 3 more

Published Jun 30, 2023 · 0 citations

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

BERT pre-trains deep bidirectional Transformers using masked language modeling and next sentence prediction, achieving state-of-the-art results across language understanding tasks via fine-tuning.

Jacob Devlin, Ming‐Wei Chang, Kenton Lee, Kristina Toutanova

Published 2019 · 34,030 citations

Laplacian Heads Improve Transformers by Smoothing Token Representations

Yuchong Zhang, Vardan Papyan

Sydney Poster Session 5, Thu, Dec 10, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

Rethinking Language Model Scaling under Transferable Hypersphere Optimization

Liliang Ren, Yang Liu, yelong shen, Weizhu Chen

Atlanta Poster Session 6, Fri, Dec 11, 4:30 PM–7:30 PM, Hall C1 · Published 2026

Language Models Need Sleep

Sangyun Lee, Sean McLeish, Tom Goldstein, Giulia Fanti

Atlanta Poster Session 4, Thu, Dec 10, 4:30 PM–7:30 PM, Hall C1 · Published 2026

Meta-memorization and memorization scaling laws in transformers

Alex Nguyen, Kenneth Norman, Gautam Reddy

Paris Poster Session 1, Wed, Dec 9, 12:30 PM–2:30 PM, Paris Poster Hall · Published 2026

Express Language Modeling

Albert Gong, Annabelle M Carrell, Raaz Dwivedi, Lester Mackey

Atlanta Poster Session 1, Wed, Dec 9, 10:00 AM–1:00 PM, Hall C1 · Published 2026

Attention Is More Than You Need: Spectral Redundancy in Multi-Head Routing

Charles Cao

Atlanta Poster Session 2, Wed, Dec 9, 4:30 PM–7:30 PM, Hall C1 · Published 2026

Learning Rate Transfer in Normalized Transformers

Boris Shigida, Boris Hanin, Andrey Gromov

Atlanta Poster Session 2, Wed, Dec 9, 4:30 PM–7:30 PM, Hall C1 · Published 2026

Position Without Positional Embeddings: A Directional Mechanism in NoPE Transformers

Matan Avitan, Ido Nachum, Yanai Elazar, Yoav Goldberg

Sydney Poster Session 4, Wed, Dec 9, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion

Georgios Batzolis, Mark Girolami, Luca Ambrogioni

Sydney Poster Session 3, Wed, Dec 9, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

Closing the Loop with Fixed-Point Self-Attention

Mrinal Mathur, Barak Pearlmutter, Sergey Plis

Sydney Poster Session 4, Wed, Dec 9, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

On the Origin of Algorithmic Progress in AI: Evidence from Language Model Pre-Training

Hans Gundlach, Alex Fogelson, Jayson Lynch, Ana Trisovic and 3 more

Sydney Poster Session 1, Tue, Dec 8, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

Token-Conditional Expert Dropout: Implicit Regularization for Stable MoE Pretraining

Haili Wang

Sydney Poster Session 5, Thu, Dec 10, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

Knocking-Heads Attention: Drop-in Shared Projections for Cross-Head Coordination

Zhanchao Zhou, Xiaodong Chen, Haoxing Chen, Zhenzhong Lan and 1 more

Sydney Poster Session 4, Wed, Dec 9, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Model Collapse is a Singular Complexity Trajectory

Sarwesh Rauniyar

Sydney Poster Session 3, Wed, Dec 9, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

Pretraining Data Statistics Shape the Phases of Learning Entity Comparison in Language Models

Yik S Chan, Jing Huang, Yanai Elazar, Atticus Geiger

Sydney Poster Session 2, Tue, Dec 8, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Self-Distillation of Hidden Layers for Masked Self-Supervised Representation Learning

Scott C. Lowe, Anthony Fuller, Sageev Oore, Evan Shelhamer and 1 more

Paris Poster Session 1, Wed, Dec 9, 12:30 PM–2:30 PM, Paris Poster Hall · Published 2026

Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes

LIU Hanqing, Jianjun Cao, Yuanze Li, Zijian Zhou

Paris Poster Session 1, Wed, Dec 9, 12:30 PM–2:30 PM, Paris Poster Hall · Published 2026

On the Information Loss of Multi-Token Prediction: Origin and Solution

Tangyu Jiang, Zhanke Zhou, Haodi Wang, Yiu-ming Cheung and 2 more

Sydney Poster Session 6, Thu, Dec 10, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Geometrically Disentangling Concept Learning from the Language Modeling Loss

Yupei Wang, Neil R Mallinar, Misha Belkin, Alex Warstadt

Atlanta Poster Session 2, Wed, Dec 9, 4:30 PM–7:30 PM, Hall C1 · Published 2026

PipeFSDP: Efficient Pipeline Parallel under Fully Sharded Data Parallel for Large Language Model Training

Xinglin Pan, Mingji Han, Penghao Zhao, Lin Zheng and 4 more

Sydney Poster Session 3, Wed, Dec 9, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

Towards Precise Knowledge Distillation for Large Language Models via Knowledge Probing

Jiajun Liu, Yao He, Wenjun Ke, Peng Wang and 3 more

Sydney Poster Session 5, Thu, Dec 10, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

PGSB: Pretrained-Guided Shared Basis for LoRA Model Merging

Muqing Liu, Chongjie Si, Zhuoya Liu, Yuheng Jia

Sydney Poster Session 5, Thu, Dec 10, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

Rethinking Softmax Attention: Polynomial Activations for Transformers

Hemanth Saratchandran, Jianqiao Zheng, Yiping Ji, Wenbo Zhang and 1 more

Sydney Poster Session 4, Wed, Dec 9, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

One Temperature to Rule Them All?

Aviv Orly, Ori Shem Ur, Yaron Oz

Paris Poster Session 3, Thu, Dec 10, 12:30 PM–2:30 PM, Paris Poster Hall · Published 2026

Scaling Laws for SAE Training Data

Nikita Koriagin, Nikita Balagansky, Daniil Gavrilov

Sydney Poster Session 5, Thu, Dec 10, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

Harder the Task, Sparser the Representation: Sparsity as a Learning Signature of Capability in LLMs

Mingyu Jin, Yutong Yin, Jingcheng Niu, Qingcheng Zeng and 7 more

Atlanta Poster Session 4, Thu, Dec 10, 4:30 PM–7:30 PM, Hall C1 · Published 2026

First-Token Attraction in Mamba Dynamics

Trinh Nguyen, Duy-Tung Pham, Minh-Khoi Nguyen-Nhat, Hoang-Son Do and 2 more

Sydney Poster Session 6, Thu, Dec 10, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

The Cost of Absolute Position: A Spread-Expressivity Tradeoff for Additive Positional Encodings

Noah Mitchell, Isaac Gabriel, Alexander Wyatt

Atlanta Poster Session 4, Thu, Dec 10, 4:30 PM–7:30 PM, Hall C1 · Published 2026

Circuit-Level Knowledge Distillation for Large Language Models

Yashuo Luo, Tongxu Wang, Siyuan He, Chunyu Wei

Sydney Poster Session 6, Thu, Dec 10, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

Virtual Head Attention

Xibo Ding, Guoxia Wang, Jinle Zeng, Jiabin Yang and 3 more

Sydney Poster Session 1, Tue, Dec 8, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

Inside Emergence: Structure-Behaviour Gaps in Language Model Training

Hak Hyun Kim, Yash Raj, Soroush Vosoughi

Atlanta Poster Session 2, Wed, Dec 9, 4:30 PM–7:30 PM, Hall C1 · Published 2026

Decomposing and Reshaping Scaling Laws through Token Learning Times

Pingjie Wang, zechenhu, Peiru Yang, Jingtao Han and 1 more

Sydney Poster Session 3, Wed, Dec 9, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

Contribution-Aware Structured Sparsity for Model Merging

Yan Li, GUIPING CAO, Meng XU, Tao Jiang and 4 more

Sydney Poster Session 3, Wed, Dec 9, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

Speculative Self-Distillation enables Efficient Knowledge Internalization

Shayan Talaei, Agam Bhatia, Arshia Soltani Moakhar, Jonas Hübotter and 2 more

Sydney Poster Session 6, Thu, Dec 10, 5:00 PM–8:00 PM, Hall 1-4 · Published 2026

SAGE: Semantically Disentangled Representation Learning through Latent Geometry Constraint and Large Language Model

Qiuyu Chen, Liang Xu, Yunnan Wang, Mingqi Yuan and 8 more

Sydney Poster Session 3, Wed, Dec 9, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

A Simple Unigram Cross-Entropy Lens on the Lexical Imprint of Pre-training Data

Jeonghoon Kim, Woojin Chung, Woomin Song, Cheonbok Park and 2 more

Sydney Poster Session 3, Wed, Dec 9, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026

DepthGraft: Structural Regularization through Hierarchical Cross-Layer KV Reconstruction

Xiaohan Qin, Xiangdong Zhang, Yu Wang, Huaijin Wu and 3 more

Paris Poster Session 5, Fri, Dec 11, 11:30 AM–1:30 PM, Paris Poster Hall · Published 2026

ExpertNavigator: Functionally Coherent Expert Grouping and Pairwise-Ranked Routing for High-Fidelity Dense-to-MoE Conversion

Xiaohan Qin, Cancheng Zhang, Xiaoxing Wang, Xiangdong Zhang and 3 more

Paris Poster Session 3, Thu, Dec 10, 12:30 PM–2:30 PM, Paris Poster Hall · Published 2026

Why Routers Freeze: Infinite Width Learning Dynamics for Mixture of Experts

Anish Dhir, Volkan Cevher, Leena Chennuru Vankadara

Sydney Poster Session 3, Wed, Dec 9, 10:00 AM–1:00 PM, Hall 1-4 · Published 2026