Geospatial foundation models (GFMs) have emerged as a powerful tool for extracting insights from Earth Observation (EO) data. These models, trained on large-scale EO datasets, aim to capture fundamental geospatial features that can be leveraged for various downstream tasks. However, the rapid development of GFMs has outpaced the establishment of a standardised and robust evaluation methodology. This lack of a standardized evaluation protocol poses a significant challenge for the GFM community. It hinders the ability to effectively compare models, assess their true capabilities, and identify areas for improvement. Consequently, researchers and practitioners struggle to make informed decisions about which GFMs to utilize for their specific needs. To address this critical gap, this project aims to establish a robust and widely applicable benchmark by investigating the current geospatial foundation models' capabilities on a comprehensive set of diverse downstream datasets, spanning from marine debris segmentation to crop type mapping, addressing several domains (e.g., urban, agricultural, marine, forest), temporality, geographical areas, resolutions (e.g., high-resolution, medium-resolution), and sensor types (e.g., optical, synthetic aperture radar (SAR)).