As generative models grow to billions of parameters, the cost of adapting, storing and running them has become a central computational challenge. This project studies how to replace large models by reduced-order ones at minimal loss in performance, through two related lines of work.
The first line, now completed, addressed low-rank adaptation (LoRA), which fine-tunes a pretrained weight matrix through a low-rank update ΔW = BA while the pretrained weights stay frozen. We derived a data-driven rule for initialising the adaptation matrices A and B.
The second line, the focus of this allocation, concerns deep state-space models (SSMs). SSMs originate in control and system identification, and stacked inside deep architectures (S4, S5, Mamba-2) they now rival Transformers on long sequences. Their state dimension sets both their memory and their computational and storage cost, so choosing it is non-trivial. Reducing it requires choosing both the order of each layer and the coordinate basis in which to truncate.
Our aim is a theoretically grounded, data-driven balanced truncation for trained SSMs. It uses Gramians weighted by the actual input data, allocates states across layers under a global budget, and comes with error guarantees that can be computed. We validate it on image, speech, long-range and language benchmarks, against existing pruning and truncation methods. Each experiment evaluates many compressed models on full test sets, across state budgets, random seeds and baselines, for models up to billions of parameters.