Recent advances in natural language processing have produced large language models (LLMs) with remarkable capabilities, yet these models often exhibit miscalibrated confidence: they make assertive claims when they should hedge, and rarely recognize when they lack the information needed to answer reliably. The aim of this research project is to develop methods for Bayesian calibration and uncertainty quantification in foundation models, with a primary focus on large language models. The main focus is on instruction-tuning LLMs to produce outputs whose stated confidence follows Bayesian probabilities, and to recognize when they need additional information before committing to an answer. We will study how training objectives, data curation, and evaluation protocols can be designed so that calibration and information-seeking emerge as learned behaviors rather than post-hoc corrections. In closely related directions, we will explore Joint Embedding Predictive Architectures (JEPA) as a framework for learning predictive representations that can be analyzed probabilistically, and Tabular Prior-data Fitted Networks (TabPFN) as an example of approximate Bayesian inference at scale. We will also investigate text-to-SQL as a downstream testbed, since query generation provides a natural setting in which a model must decide whether the user's request is sufficiently specified or whether clarification is needed before producing an answer. The potential impact of this research is better-calibrated language models that know when to ask for more information could substantially reduce hallucination in deployed systems and improve their reliability in high-stakes settings such as data analysis, information retrieval, and decision support.