In this project, we explore ways to overcome data scarcity for medical image machine learning. Among other things, we attempt to leverage radiology reports as privileged information when training image-only classifiers. To this end, we have compared the performance of models pre-trained with and without text-alignment, as well as employed distillation to explore the potential of utilizing the texts during fine-tuning. We have also explored lymphoma segmentation in PET-CT images, and doing so more effectively by extracting key anatomical regions from the CT scan using a pre-trained model. Moving forward we plan to additionally explore the causal effect of treatments on images. This could for example be predicting the effect of a neoadjuvant therapy on a follow-up image in an oncological study.