Geospatial foundation models (GFMs) have demonstrated remarkable potential for unlocking insights from Earth Observation (EO) data at scale. However, our recent benchmarking efforts reveal that existing GFMs still face major limitations when confronted with the inherent complexities of EO data—especially in handling multi-modal, multi-temporal, and specular (domain-specific and context-sensitive) characteristics such as varying resolutions, dynamic land cover, and cross-sensor heterogeneity. This follow-up project aims to advance the state of GFMs by improving their geospatial embeddings, with a focus on more effectively leveraging EO-specific structures and relationships. Specifically, we plan to explore novel training strategies, modality fusion techniques, and architectural improvements that enhance the robustness and adaptability of GFM representations across diverse geographies and tasks. The ultimate goal is to develop more context-aware and inclusive geospatial embeddings that improve model generalization, fairness, and applicability for downstream EO tasks such as land cover mapping, environmental monitoring, and disaster response, especially in underrepresented regions.