NAISS
SUPR
NAISS Projects
SUPR
Equivariant Neural Scaling Laws
Dnr:

NAISS 2026/3-943

Type:

NAISS Medium

Principal Investigator:

Daniel Persson

Affiliation:

Chalmers tekniska högskola

Start Date:

2026-11-01

End Date:

2027-11-01

Primary Classification:

10199: Other Mathematics

Allocation

Abstract

Equivariant neural networks incorporate symmetry directly into their architecture. This project investigates how this choice affects neural scaling laws for image classification: the relationship between predictive performance and the computational resources used for training. We ask whether explicit equivariance improves performance at a fixed computational budget and, if so, whether it changes the scaling exponent or shifts the scaling curve over the range studied. We use the Platonic Transformer architecture [1] on ImageNet-22k [2] and compare four symmetry groups: the trivial group, with no imposed symmetry; horizontal reflection (D1); rotations by multiples of 180 degrees (C2); and rotations by multiples of 90 degrees (C4). By training families of models at different sizes and computational budgets, we estimate the empirical compute-loss frontier for each group. This identifies the lowest observed loss at a given training cost and allows us to compare the computational value of the different symmetry constraints. The project is already underway. Initial training sweeps and benchmarking on collaborators' HPC resources have produced preliminary results. We have optimized the training pipeline and data loading, and tested the container-based training environment on Grace Hopper GPUs. The requested Arrhenius allocation will support two further studies. First, we will repeat selected configurations with a different random seed to assess the sensitivity of the preliminary results to initialization. Second, we will compare the current dense implementation based on regular representations with a sparse implementation based on Fourier transforms. We will repeat selected parts of the scaling study to measure whether, and by how much, the sparse implementation improves predictive performance per GPU-hour. The proposed follow-up programme would extend these studies through additional seed repetitions, targeted refinement of the model-size and training-budget grids, and matched comparisons of the dense and sparse implementations. The choice of follow-up runs would be guided by the initial results, concentrating resources on the configurations needed to resolve the main scientific questions. The research is led by Daniel Persson at Chalmers University of Technology, with Elias Nyholm, in collaboration with the groups of Tommi Jaakkola at MIT, Robin Walters at Northeastern University, and Erik Bekkers at the University of Amsterdam. It builds on our previous work on designing [3, 4, 5, 6], analyzing [7], and training [8, 9, 10, 11] symmetry-aware models. (References given in the next text box).