Using Malleable Task Scheduling to Accelerate Package Manager Installations
HPC Center Planning and Operations
Parallel Application Frameworks
TimeMonday, 18 November 20199:32am - 10am
DescriptionPackage managers, containers, automated testing, and Continuous Integration (CI), are becoming an essential part of HPC development workflows. These automated tools often require software recompilation. However, large stacks such as those deployed on HPC clusters can have combinatorial dependencies, and may take a system several days to compile. Despite the use of simple parallelization (such as 'make -j'), build execution time often do not scale with system resources. For such cases, it is possible to improve overall installation time by compiling parts of software stack independently, each scheduled on a subset of available cores. We apply malleable-task scheduling algorithms to better exploit available parallelism in build system workflows and improve stack build time overall. Using a prototype implementation in the Spack package manager, malleable-task scheduling can improve build times by more than 2x.