Theoretical and empirical advances in modern machine learning (ML) over the last decade have been informed by state-of-the-art developments in multiple disciplines, including mathematics, physics and computer science. Central to these developments is the use of the modelling tools and perspectives from dynamical systems, leading to principled approaches and cutting-edge advances in understanding and designing learning models. In particular, the use of dynamical models has been behind recent successes in deep learning. However, the roles and effects (particularly those constructive ones) of randomness on the learning behaviour of modern deep networks are largely unexplored. Since randomness appears in various forms during a typical learning process for these networks, solid understanding of its effects is pivotal to advance the field. The main objective of this project is to investigate the roles of randomness in learning with modern deep networks through the lens of dynamical systems via a series of interconnected studies. Random elements, particularly in the form of stochastic noise, are capable of inducing various non-trivial effects on the system dynamics. Therefore, we expect to discover various interesting noise-induced phenomena and study their implications for ML systems. Our approach is to employ dynamical system frameworks and methods to (a) explore the impacts of noise on the learning behaviour of several deep network models in various directions and settings; and (b) exploit the findings in (a) to construct efficient and reliable learning models for physical applications. The lens of dynamical systems not only provides a principled way to study the problems in (a)-(b) but also allows us to build on the available techniques and tools to tackle these problems.