A research team led by Professor Seo Ji-won of the School of Computer Science at Hanyang University recently developed optimized technique to accelerate learning of large-scale deep learning systems such as BERT and GPT-3. This technique speeds up learning by increasing the utilization rate of hardware accelerators such as graphic card (GPU) which is used to train deep learning systems. Through this technology, Professor Seo's team made large-scale model learning possible with a relatively small number of hardware accelerators.

The study was published in the world's top conference on computer systems, "The European Conference on Computer Systems," and is drawing attention from academia and industry in that it can not only accelerate learning of large-scale neural network trainings but also be used in various ways in the future.

Existing training methods of deep learning systems were unable to schedule for efficient use of hardware accelerators since the computation is only scheduled in reverse order of deep learning models layers when performing BackProp algorithms. In order to resolve the shortcoming, Professor Seo's team devised a "scheduling algorithm" that computates BackProp algorithm in the order of analyzing the computation dependencies of the BackProp algorithm and optimizing the availability of hardware.

During the analysis process, Professor Seo's team found that in the case of Weight Gradient Computation, it is possible to schedule in different order without calculating the layers of the deep learning systems in reversed order, which led the research team develop an "Out-Of-Order BackProp" technique.

Out-Of-Order BackProp can be generally and widely applicable in training deep learning systems, which could be applied to distributed learning of large-scale deep learning systems to create efficient scheduling algorithms.

With this technique, professor Seo's team succeeded in applying scheduling technique that increases the priority of parameter slope computation on the Critical Path in Data Parallel Training and Pipeline Parallel Training, representative methods of distributed learning.

As a result, the research team's scheduling algorithm improved the learning speed of computer vision models such as DenseNet and MobileNet up to 1.5 times, as well as enhanced the learning speed of large natural language processing models such as BERT and GPT-3 up to 2 times.

Professor Seo's research results were presented by invitation at Imperial College London in the U.K. and Stanford University in the U.S. in recognition of their excellence after the announcement of the EuroSys academic conference. In addition, with the attention of the industry, seminars were held after being invited by Google Headquarters in US, Naver Clova, LG AI Research Institute, KT, SKT, Moloco, and MakinaRocks.

The study, supported by the Institute for Information and Communication Technology Planning and Evaluation (IITP) and KT Co., was conducted with Oh Hyung-joon, Lee Joon-yeol, and Kim Hyung-joo, students of Hanyang University's master's and doctorate programs. Meanwhile, Professor Seo Ji-won is conducting research for the application of this technique which has been developed in this study, participating in the KT AI One team.