Spatio-temporal prediction is a central task in many domains, e.g., epidemiology, climatology, and transportation systems, usually addressed by either data-driven methods or sophisticated simulation models, both suffering from respective failure modes: Simulation is limited by the underlying (inevitably simplifying) assumptions, and machine learning degrades in boundary cases with scarce data or non-stationary data distributions. We therefore aim at optimally integrating both approaches to achieve zero failure. In principle, the system will take both the real data and the simulator as inputs and learn a predictor end-to-end. This involves learning if and to what extent the different components can be trusted and if the integrated information is in itself insufficient for enabling accurate and robust predictions. We work with large datasets - both tabular and image datasets - and use Deep Neural Networks to serve our Hybrid Inference objective