NeurIPS 2026Deep learning theorySpotlightTechnical University of Denmark Max Planck Institute for Mathem UCLA

The Symmetries of Three-Layer ReLU Networks

This paper characterizes generic parameter fibers of three-layer ReLU bottleneck networks via semi-algebraic descriptions, yielding polynomial-time functional equivalence tests and analyzing symmetry-induced gradient flow conservation laws.

Johanna Marie Gegenfurtner, Moritz Grillo, Guido Montufar

Published 2026Sydney Poster Session 4 · Wed, Dec 9, 5:00 PM–8:00 PM local time · Hall 1-4arXiv ↗OpenReview ↗

70%

OverallHighly rated

OverallHighly ratedVote to see the scoreThe exact score shows once you've voted, so every vote is your own call. The first half of each home page shelf shows its scores.

Readers

–

Only vote on papers you've read. Sign in with GitHub to vote.

AI panel5/20reviewers recommend it

lenient 1/5

medium 3/10

strict 1/5

AI panel?Vote to see what the 20 AI reviewers said

Abstract

We develop a framework for analyzing parameter symmetries in deep ReLU networks and obtain a complete characterization of the generic parameter fibers for three-layer bottleneck architectures. Our approach provides explicit semi-algebraic descriptions of these fibers and yields a polynomial time algorithm for deciding functional equivalence of two parameters. The symmetries include discrete and continuous transformations arising from layer composition, and depend on whether deeper layers hide or preserve geometric structure from preceding layers. Finally, we show that some of these symmetries induce local conservation laws along gradient flow, while others do not.