Good Papers

Showing LLM pretraining & scaling laws Show all papers

83%Must read
?Must readVote to see the score
arXivLLM pretraining & scaling laws

Towards Looped Models Done Right, Part II: Rethinking at Fixed Points

Looped language models use fixed-point convergence to enable truncated training, shared KV caches, faster prefill, and faster RL updates, while a learned depth prior and orthogonal input injection improve perplexity across scales.

Benhao Huang, Chufan Shi, Junlin Chen, Shicheng Wen and 3 more

Published Oct 5, 2026 · ▲ 17 on Hugging Face · Code ★ 30

– ReadersNo votes yet
13/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 13 of 20 reviewers recommend it
lenient 3/5
medium 8/10
strict 2/5
67%Highly rated
?Highly ratedVote to see the score
arXivLLM pretraining & scaling laws

The Numerical Linear Algebra of Large Language Models

This survey explains large language model core concepts to numerical analysts and highlights key numerical linear algebra contributions to LLM techniques.

Abdelkader Baggag, Yousef Saad

Published Oct 3, 2026 · ▲ 4 on Hugging Face

– ReadersNo votes yet
2/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 2 of 20 reviewers recommend it
lenient 2/5
medium 0/10
strict 0/5
74%Highly rated
?Highly ratedVote to see the score
arXivLLM pretraining & scaling laws

Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale

Fixed token codes can replace trainable input embeddings in 1.7B-scale language models, removing 100.7M parameters while preserving substantial capabilities without requiring token-specific vectors.

A. Bochkov

Published Oct 2, 2026

– ReadersNo votes yet
9/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 9 of 20 reviewers recommend it
lenient 3/5
medium 4/10
strict 2/5
76%Highly rated
?Highly ratedVote to see the score
arXivU California, Los AngelesMila - Quebec Artificial IntelliMcGillLLM pretraining & scaling laws

Cross-Lingual Alignment for Decoder-Only Models using MoE Routers

Cross-lingual MoE router alignment improves multilingual LLM performance by aligning router outputs across languages instead of hidden states.

Lucas Bandarkar, Clark Peng, Ahmed Haj Ahmed, Aditi Khandelwal and 1 more

Published Oct 1, 2026 · 0 citations · ▲ 1 on Hugging Face · Code

– ReadersNo votes yet
10/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 10 of 20 reviewers recommend it
lenient 4/5
medium 6/10
strict 0/5
83%Must read
?Must readVote to see the score
arXivLLM pretraining & scaling laws

Scaling and Distilling Text Embeddings for Better Diffusibility

Scaling and distilling text embeddings improves latent diffusion by yielding more connected, diffusible spaces that boost generative performance beyond autoregressive baselines.

Zekai Zhang, Yunjie Tian, Yanjin He, Xiaoyan Zhang and 3 more

Published Oct 1, 2026 · 0 citations · ▲ 57 on Hugging Face · Code ★ 3

– ReadersNo votes yet
13/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 13 of 20 reviewers recommend it
lenient 4/5
medium 8/10
strict 1/5
83%Must read
?Must readVote to see the score
arXivLLM pretraining & scaling laws

Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts

LOOM stabilizes looped MoE via bounded residual updates and per-loop routers to scale loops to 9, 12, cutting perplexity from 9.62 to 7.77.

Di He, Pengxiang Li, Da Chang, Qingyan Meng and 2 more

Published Oct 1, 2026 · 0 citations · ▲ 16 on Hugging Face · Code ★ 8

– ReadersNo votes yet
13/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 13 of 20 reviewers recommend it
lenient 4/5
medium 8/10
strict 1/5
80%Must read
?Must readVote to see the score
arXivLLM pretraining & scaling laws

Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models

Clock Diffusion introduces semi-autoregressive continuous diffusion language models with position-dependent noise schedules, efficient training and sampling, and Cache Grab acceleration to achieve state-of-the-art diffusion likelihoods and competitive reasoning performance.

Yair Schiff, Omer Belhasin, Roy Uziel, Matan Rusanovsky and 6 more

Published Oct 1, 2026 · 0 citations

– ReadersNo votes yet
12/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 12 of 20 reviewers recommend it
lenient 4/5
medium 7/10
strict 1/5
71%Highly rated
?Highly ratedVote to see the score
arXivLLM pretraining & scaling laws

E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models

E-MoE improves few-step non-factorized diffusion language models via Mixture-of-Experts routing as a discrete shared latent, boosting sample quality without extra active parameters.

Arseny Ivanov, Alexander Kolesov, Alexander Korotin, Ivan Oseledets and 1 more

Published Sep 29, 2026 · 0 citations · ▲ 66 on Hugging Face

– ReadersNo votes yet
7/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 7 of 20 reviewers recommend it
lenient 3/5
medium 3/10
strict 1/5
80%Must read
?Must readVote to see the score
arXivCase Western ReserveKyotoLLM pretraining & scaling laws

How to Loop MoE: Flatten the Experts, Untie the Attention

Foil improves looped MoE by flattening experts and untying attention, reducing pretraining loss by 0.012 nat and improving routing balance and confidence.

Shouren Wang, Chuang Ma, Mohsen Hariri, Debargha Ganguly and 5 more

Published Sep 28, 2026 · 0 citations · ▲ 9 on Hugging Face · Code ★ 2

– ReadersNo votes yet
12/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 12 of 20 reviewers recommend it
lenient 4/5
medium 7/10
strict 1/5
74%Highly rated
?Highly ratedVote to see the score
arXivLLM pretraining & scaling laws

Kimi K3: Open Frontier Intelligence

Kimi K3 is a 2.8 trillion-parameter Mixture-of-Experts model with native vision and 1-million-token context that achieves frontier performance across reasoning, coding, and agentic tasks and outperforms comparable open and proprietary models.

Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao and 36 more

Published Jul 27, 2026 · 0 citations · ▲ 525 on Hugging Face · Code ★ 8,900

– ReadersNo votes yet
9/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 9 of 20 reviewers recommend it
lenient 3/5
medium 5/10
strict 1/5
83%Must read
?Must readVote to see the score
arXivU WashingtonStanfordLLM pretraining & scaling laws

Scaling Participation in Modular AI Systems

Modular participatory AI combines small stakeholder-trained models into compositional systems that outperform monolithic LLMs by up to 15.4% and exhibit emergent collaborative capabilities.

Shangbin Feng, Yike Wang, Weijia Shi, Luke Zettlemoyer and 2 more

Published Jun 5, 2026 · 0 citations

– ReadersNo votes yet
13/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 13 of 20 reviewers recommend it
lenient 5/5
medium 7/10
strict 1/5
86%Must read
?Must readVote to see the score
arXivMassachusetts Institute of TechnLLM pretraining & scaling laws

HRM-Text: Efficient Pretraining Beyond Scaling

HRM-Text replaces Transformers with a hierarchical recurrent model and trains on instruction pairs to achieve competitive 1B-parameter performance with 100, 900x fewer tokens and far less compute.

Guan Wang, Changling Liu, Chenyu Wang, Cai Zhou and 5 more

Published May 20, 2026 · 0 citations · ▲ 322 on Hugging Face · Code ★ 2,134

– ReadersNo votes yet
14/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 14 of 20 reviewers recommend it
lenient 5/5
medium 8/10
strict 1/5
78%Highly rated
?Highly ratedVote to see the score
arXivPekingScience FactoryShanghai Jiao TongLLM pretraining & scaling laws

DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models

DataFlex unifies sample selection, mixture adjustment, and reweighting for LLMs via a modular LLaMA-Factory framework that improves MMLU and perplexity with faster runtimes.

Hao Liang, Zhengyang Zhao, Mingrui Chen, Meiyi Qiang and 21 more

Published Mar 27, 2026 · 0 citations · ▲ 279 on Hugging Face · Code ★ 2,958

– ReadersNo votes yet
11/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 11 of 20 reviewers recommend it
lenient 5/5
medium 6/10
strict 0/5
80%Must read
?Must readVote to see the score
arXivLLM pretraining & scaling laws

Attention Residuals

Attention Residuals replace fixed residual accumulation with softmax attention over previous layer outputs for selective, input-dependent aggregation, improving scaling and downstream performance with minimal overhead.

Kimi Team, Guangyu Chen, Yu Zhang, Jianlin Su and 33 more

Published Mar 16, 2026 · 0 citations · ▲ 198 on Hugging Face · Code ★ 3,515

– ReadersNo votes yet
12/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 12 of 20 reviewers recommend it
lenient 4/5
medium 8/10
strict 0/5
89%Must read
?Must readVote to see the score
arXivShanghai Jiao TongAlibaba GroupU Wisconsin–MadisonU Illinois Urbana-ChampaignMila - Quebec Artificial IntelliLLM pretraining & scaling laws

OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration

OPUS defines optimizer-induced update-space data utility for dynamic LLM pre-training selection, outperforming full-scale baselines with minimal overhead.

Shaobo Wang, Xuan Ouyang, Tianyi Xu, Yuzheng Hu and 8 more

Published Feb 5, 2026 · 0 citations · ▲ 354 on Hugging Face

– ReadersNo votes yet
16/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 16 of 20 reviewers recommend it
lenient 5/5
medium 9/10
strict 2/5
67%Highly rated
?Highly ratedVote to see the score
ACL 2026NanjingAlibaba GroupAlibaba GroupZhejiangLLM pretraining & scaling laws

A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM𝛥 Integration into Upcycled MoE

The method expands multilingual LLMs via post-training PARAMΔ integration into upcycled MoE for data-efficient language acquisition.

Hao Zhou, Tianhao Li, Zhijun Wang, Shuaijie She and 5 more

Published 2026 · 0 citations

– ReadersNo votes yet
2/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 2 of 20 reviewers recommend it
lenient 1/5
medium 1/10
strict 0/5
76%Highly rated
?Highly ratedVote to see the score
ICML 2025U VirginiaLLM pretraining & scaling laws

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion

Rotation symmetry generalizes permutation symmetry continuously for transformers, improving parameter matching and model fusion across language and vision tasks.

Binchi Zhang, Zaiyi Zheng, Zhengzhang Chen, Jundong Li

Published Feb 1, 2025 · 0 citations

– ReadersNo votes yet. 2 from authors or colleagues not counted
10/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 10 of 20 reviewers recommend it
lenient 4/5
medium 6/10
strict 0/5
57%Worth a look
?Worth a lookVote to see the score
EMNLP 2025FindingsSingapore-MIT Alliance for ReseaNational University of SingaporeAgency for Science, Technology aSystems, Applications & ProductsAI SingaporeLLM pretraining & scaling laws

Uncovering Scaling Laws for Large Language Models via Inverse Problems

Inverse problem methods uncover scaling laws for large language models, revealing predictive relationships between model size, data, and performance from abstract evidence.

Arun Verma, Zhaoxuan Wu, Zijian Zhou, Xiaoqiang Lin and 14 more

Published 2025 · 0 citations

– ReadersNo votes yet
1/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 1 of 20 reviewers recommend it
lenient 1/5
medium 0/10
strict 0/5
67%Highly rated
?Highly ratedVote to see the score
Agency for Science, Technology aNational University of SingaporeSingapore-MIT Alliance for ReseaChinese University of Hong Kong,LLM pretraining & scaling laws

Position Paper: Data-Centric AI in the Age of Large Language Models

This position paper argues for data-centric AI in the age of large language models and proposes research directions.

Xinyi Xu, Zhaoxuan Wu, Rui Qiao, Arun Verma and 15 more

Published 2024 · 2 citations

– ReadersNo votes yet
2/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 2 of 20 reviewers recommend it
lenient 2/5
medium 0/10
strict 0/5
80%Must read
?Must readVote to see the score
arXivXi'an JiaotongU Notre DameU VirginiaLLM pretraining & scaling laws

LMBot: Distilling Graph Knowledge into Language Model for Graph-less Deployment in Twitter Bot Detection

LMBot distills graph neural network knowledge into language models for efficient graph-less Twitter bot detection, achieving state-of-the-art results across four benchmarks.

Cai, Zijian, Zhaoxuan Tan, Zhenyu Lei, Zhu, Zifeng and 3 more

Published Jun 30, 2023 · 0 citations

– ReadersNo votes yet
12/20 AI panelreviewers recommend it

Readers and the AI panel: vote on this paper to see what they said.

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel: 12 of 20 reviewers recommend it
lenient 5/5
medium 7/10
strict 0/5
Show 20 more papers