The geometry underlying neural network representations allows to formally characterize and control several important aspects that underpin the success of deep learning. Indeed, the ability of a model to generalize to unseen data, its robustness to adversarial and random perturbations, as well as its transferability to downstream tasks is jointly controlled by the geometry of (1) the data, (2) the neural network architecture as well as (3) the associated optimization landscape. This project studies how the geometry of learning is affected by inductive biases implicitly imposed by deep network architectures, the choice of loss and optimizer, as well as the data distribution, with focus on supervised and self-supervised learning. The research work studies regularization implicitly emerging from learning and connects curvature of the optimization landscape and the model's input space to generalization, robustness and transferability with special focus on self-supervised learning. The project outcomes include novel regularization strategies, activation functions as well as model adaptation regimes.