How to Loop MoE: Flatten the Experts, Untie the Attention
Foil improves looped MoE by flattening experts and untying attention, reducing pretraining loss by 0.012 nat and improving routing balance and confidence.
Published Sep 28, 2026 · 0 citations · ▲ 9 on Hugging Face · Code ★ 2
Readers and the AI panel: vote on this paper to see what they said.
Only vote on papers you've read. Sign in with GitHub to vote.