Foundation models pretrained on population-scale event streams could provide early pandemic predictions before supervised methods become viable and potentially be used to model various aspects of the pandemic. We will train an autoregressive transformer on Swedish register data up to fixed cutoffs and evaluate out-of-time performance on COVID-19 outcomes against supervised baselines trained under the same constraints.