This project aims to develop a Swedish medical benchmark for evaluating Large Language Models (LLMs) in healthcare. Most existing medical benchmarks are designed in English, which makes it difficult to measure how well LLMs perform in Swedish medical settings. The benchmark will support the evaluation of medical knowledge, clinical reasoning, safety, and communication in Swedish. The project will use established English medical datasets and translate them into Swedish. Models will then be evaluated on the same questions in both languages. This allows a direct study of the performance gap between English and Swedish. The benchmark will include both multiple-choice and open-ended tasks. Multiple-choice questions will measure factual accuracy, while open-ended answers will be evaluated using structured rubrics. The core innovation is a semi-automated pipeline for creating and validating Swedish medical evaluation data and rubrics. The pipeline will combine LLM support, medical sources, and expert review. This will reduce the amount of manual work while helping to preserve medical correctness and clear Swedish language. The methodology involves: (1) Translating and validating established medical datasets from English into Swedish; (2) Evaluating open-weight and proprietary LLMs on matched English and Swedish tasks; (3) Developing structured rubrics for open-ended answers, with criteria for medical correctness, reasoning, safety, and communication; (4) Studying the reliability of LLM-based evaluation by comparing automatic scores with expert judgments; (5) Using the translated data to train or distill smaller open-weight models for Swedish medical applications; and (6) Exploring parameter-efficient training methods to reduce the computational cost of model adaptation. As an adjacent activity, the project will also support ongoing work on longitudinal ICD prediction. This work studies how a patient’s previous diagnosis codes can be used to predict future diagnosis codes. Initial baseline models are being trained on structured diagnosis-code histories. Later work will study how these models can be combined with pretrained LLMs to use both longitudinal patient patterns and broader medical knowledge. Expected outcomes include a public Swedish medical benchmark, baseline results for current LLMs, validated rubrics for open-ended evaluation, and smaller models adapted to Swedish medical tasks. The project will also support initial methods and baselines for longitudinal ICD prediction. Overall, the project addresses the lack of reliable evaluation and adaptation resources for Swedish medical AI and contributes to safer and more accessible LLMs for healthcare.