Enrico Meloni (University of Siena)

When: May 25th, 2022 – 11:00 – 11:45 AM

Where: Google meet link

Description

Pipeline Parallelism for Real-time Online Learning
Deep Learning research has seen an ongoing increase in model complexity. It has been shown that scaling up network capacity is often an effective approach to enhance the performances of neural models. On the other hand, hardware capabilities are unable to scale as fast as required by such highly complex architectures, raising the need for the development of alternative parallel computations to take advantage of multiple GPUs. Usually, these parallel algorithms are tailored to the task at hand, making their design a difficult trade-off between flexibility, scaling capacity and achievable performances. During this seminary, we will talk about Pipeline Parallelism, which was introduced in “GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism” by Huang, Y. et al. to address easy parallelization over a feed-forward convolutional network that cannot fit in the memory of a single accelerator. Finally, we will talk about how Pipeline Parallelism can be used in real-time online learning, where a live input stream is fed directly to the learning agent and thus batch processing is not a viable option.