Similar articles

Abstract: Large-scale deep learning (DL) models are trained distributedly due to memory and computing resource limitations. Few existing strategy generation approaches take optimal memory minimization as the objective. To fill this gap, we propose a novel algorithm that generates optimal parallelism strategies with the constraint of minimal memory redundancy. We propose a novel Redundant Memory Cost Model (RMCM) to calculate the memory overhead of each operator in a given parallel strategy. To generate the optimal parallelism strategy, we formulate the parallelism strategy searching problem into an integer linear programming problem and use an efficient solver to find minimal-memory intra-operator parallelism strategies. Furthermore, the proposed algorithm has been extended and implemented in a multi-dimensional parallel training framework and is characterized by the ability of high throughput and minimal memory redundancy. Experimental results demonstrate that our approach achieves significant memory savings of up to 67% compared to the latest Megatron-LM strategies, and has a similar throughput. The principal contribution of the present research lies in its provision of a novel algorithm that optimizes parallelism strategies, reducing memory redundancy in large-scale DL models. In conclusion, our paper introduces a memory-efficient algorithm for generating parallelism strategies, surpassing existing strategies in reducing memory requirements.