The emergence of big data has caused a dramatic shift in the operating regime for optimization algorithms. Since over a decade, focus has turned from interior-point methods to (stochastic) first-order algorithms to achieve better scalability. However, these methods still fall short in many modern applications. Increasingly often, data is spread across geographically dispersed locations and problem dimensions are huge, both in terms of decision vector sizes and the number of data points used. Communication, not computations, is becoming the bottleneck! In our research, we develop distributed optimization algorithms for machine learning. The research encompasses both relatively minor enhancement of optimization algorithms (e.g. better step-size policies) to the development of brand new distributed optimization algorithms and/or approaches. Recently, we have also begun to develop comprehensive system-level designs encompassing communication, computation, and storage interactions to accelerate the training process and reduce the wall-clock time required for the training. This project continues and extend our previous project; see our activity report.