ByteDance Develops OmniHuman, an AI Framework That Can Generate Realistic Videos of Humans

ByteDance, the company behind TikTok, recently shared its research on a new artificial intelligence (AI) framework. Dubbed OmniHuman, it is a video-generation framework that can create realistic human videos with full-body movement and lip-syncing. The researchers stated that it requires a human image along with motion signals such as video or audio to generate output. Several demonstration videos generated using the AI model have also been shared, showcasing the realism of the final output. Notably, the company stated that the AI model is available in the public domain.

OmniHuman Can Generate Realistic Human Videos

The researchers shared several demonstrations and detailed the framework on its website. It is an end-to-end system that was built using a novel multimodality motion conditioning mixed training strategy, the post claimed. While the researchers did not share any benchmark metrics, they claimed that the AI model “significantly outperforms existing methods.”

OmniHuman can generate videos using an image of the person and a motion signal. Motion signals can be audio only, video only or a combination of audio and video. The AI model can generate realistic videos based on text prompts. These videos can be full-body where the limbs, facial expressions, and lip movement can be synced with the audio or music playing in the background. OmniHuman can generate videos in different aspect ratios, allowing flexibility to users.

OmniHuman output example
Photo Credit: OmniHuman

The use of motion signals is a novel technique, which the company is calling omni-conditions training. With this, the AI model is trained on different modalities, including text, image, audio, and video. Researchers said this allowed the model to learn mixed conditioning which overcame the scarcity of high-quality data.

Notably, the model was trained on 18,700 hours of human video data. The details about the training process have been documented in a paper published in the online pre-print journal arXiv.

The company also shared several demonstrations of videos generated using the model, and the results appear to be highly realistic with natural body movements, hand gestures, and lip movements. Such realism has also raised concerns about deepfakes. However, the company has specified that the AI model is currently not available to be downloaded, and there is no service people can use to access its capabilities.