Sparse Attention as Compact Kernel Regression

Sparse attention corresponds to compact kernel regression, with normalized ReLU and sparsemax arising from Epanechnikov kernels and α-entmax mapping to biweight and triweight kernels. This unifies sparsity with kernel design and yields competitive kernel-based transformers on language modeling and i

Saul Santos, Nuno Gonçalves, Daniel McNamee, Marcos Treviso and 1 more

Paris Poster Session 4, Thu, Dec 10, 5:30 PM–7:30 PM, Paris Poster Hall · Published 2026