Large-scale optimization plays a crucial role in various domains and industries, enabling organizations to make informed decisions, improve operational efficiency, and maximize resource utilization. This project proposal aims to investigate and develop advanced optimization techniques specifically tailored for tackling large-scale problems. The primary objective is to address the challenges posed by complex optimization scenarios, such as high-dimensional search spaces, large amounts of data, and communication constraints. Here are some of the projects/proposals: 1. LLM training: it investigates training efficiency and performs intrinsic analyses of training dynamics in LLM training. 2. Decentralized optimization: it investigates decentralized algorithms for training neural networks based on GPUs, which may improve in terms of lower communication cost and more system robustness compared with centralized schemes.