Deep Learning model complexity and training data volume continue to grow rapidly. Training with a single-GPU, or even a single-node of GPUs, is often too slow for the iterative nature of model development and optimization. In this talk, we will discuss several aspects of building scalable GPU-based clusters for improving training time including scaling training to multiple-nodes, optimizing inter-node collective operations, and optimizing the data-cache hierarchy.
Craig Tierney is a Senior Solution Architect at NVIDIA supporting high performance computing (HPC) and deep learning (DL). His focus includes the architecture of GPU based systems to maximize HPC and DL performance and scalability. Prior to joining NVIDIA, Craig spent over 15 years providing high performance computing architecture and computational science support to NOAA and several other government and educational organizations including DOE, DOD, NASA and Stanford University. Craig holds a Ph.D. in Aerospace Engineering Sciences from the University of Colorado at Boulder.
