Securing high accuracy only through one-time registration with AI domain application technology

Hanyang University Department of Biomedical Engineering Professor Im Chang-hwan’s team developed a new technology that recognizes a user’s facial expressions and reflects them on avatars in real-time through virtual reality (VR)-based metaverse service.

The VR-based metaverse service has recently been highlighted as a future social network platform, and diverse VR-based metaverse services such as Facebook Horizon, vTime, AltSpaceVR, and VRChat have been released until now.

Thanks to the advance in technology, lots of functional advances in screen definition and speed have been made since the beginning, but there have been limits in the lack of user engagement or a sense of reality due to the users’ exact faces not being projected to their avatars, as VR devices cover around the eyes.

To solve such problems, research has been done on analyzing real-time the electromyogram (EMG) signal that occurred in the muscle when making a facial expression by attaching electrodes that measure the biosignals where the VR headgear touches the facial skin.

However, the facial expression recognition technology using the EMG of the facial area requires the process of collecting a database related to the EMG signals created when facial expressions are made in advance. Therefore, users had to endure some inconveniences in registering facial expressions repetitively for 4 to 10 times.

Professor Im’s team improved such inconveniences by applying a domain adaptation technology, which is widely researched in the recent AI field. Domain adaptation is the most recent AI technology that allows quality function with fewer data through applying a machine learning model made with existing data to new information.

Professor Im’s team effectively decreased the amount of data required for model learning by adopting a previous machine learning model created through users’ registered data for new user data. As a result, registering facial expressions only one time allowed the recognition of 11 different facial expressions with 90% accuracy and successfully projecting to an avatar’s face.

Professor Im said that “The technology for restoring expressions, even not in the training database, and the system for enabling eye movement and facial expression at the same time are on the verge of completion” and that “These will be applied to the future VR-based metaverse services.”

Professor Im’s research results were published in the September 3rd edition of the renowned academic journal in the virtual reality field Virtual Reality. The 1st author of this paper, Dr. Cha Ho-seung is currently a researcher at Georgia Institute of Technology after receiving the doctor’s degree at Hanyang University in September 2020. This research was supported by Samsung Electronics Future Technology Development Center.