February 18th.Kunlun WanweiAnnounced todayOpen SourceThe first nationally oriented AI skitscreativeVideo Generation Model SkyReels-V1The first SOTA-rated video base model-based algorithm for controlling facial expressions in China. SkyReels-A1.
The official introduction of Kunlun Wanwei said that SkyReels-V1 made marking for the details of the performance, and also dealt with the emotions, scenes, and performance appeals.Training fine-tuning with "10 million, high quality" Hollywood-grade data.
In addition, SkyReels-V1 realizes "micro-expression generation for movie and TV characters".Supports 33 character expressions and 400+ natural movement combinationsIt supports the generation of micro-expressions such as laughing, roaring, surprised, crying, etc. It also supports the generation of micro-expressions of real people's emotions.
Based on Hollywood-grade movie and television data training, every frame generated by current SkyReels is claimed to have a "Cinematic textures”.
SkyReels-V1 not only supports Vincennes videos.Also supports graphic videoIt is claimed to be "the model with the largest parameter in the open source video generation model that supports graph-generated video", and achieves the open source SOTA under the same resolution in various indexes.
To summarize, SkyReels-V1 can deliver:
- Movie Expressions Recognition System: 11 character expression comprehension for expressions of disdain, impatience, helplessness, and disgust in film and theater;
- Character spatial location awarenessBased on the human body 3D reconstruction technology, it realizes the understanding of the spatial relative relationship of multiple people in the video, and helps the model to generate film and television level character station;
- Understanding of Behavioral Intent: Constructs more than 400 kinds of behavioral semantic units to achieve accurate understanding of character behavior;
- Performance Scene Understanding: Realizing character-costume-scene-plot correlation analysis.
In terms of performance, with the support of the self-developed inference optimization framework "SkyReels-Infer", theThe model achieves 544p resolution, with inference based on a single 4090 in 80s, and also supports distributed multi-card parallelism.It supports Context Parallel, CFG Parallel, and VAE Parallel.
In addition, the model adopts fp8 quantization and parameter-level offload to meet the running requirements of user-level graphics cards with low graphics memory; supports flash attention, SageAttention, and model compilation optimization to further optimize the latency; and is based on the open-source diffuser library to enhance the ease of use.
In order to realize more accurate and controllable character video generation, KunlunWanwei has also open-sourced the SOTA-levelA video pedestal model-based algorithm for controllable expression movements SkyReels-A1The SkyReels-A1 supports video-driven, movie-quality expression capture against Runway's Act-One.
SkyReels-A1 can be based on any human scale (including portraits, half and full body compositions).Generate character motion video.
As shown in the figure below, using both the reference image (top) and the driver video (bottom left) as inputs, a new video was generated with the power of SkyReels-A1 - "grafting" the facial expressions and performance details from the driver video onto the character given the reference image. the facial expressions and performance details from the drive video were "grafted" onto the character of the given reference image.
Attach the open source address below:
- SkyReels-V1: https://github.com/SkyworkAI/SkyReels-V1
- SkyReels-A1: https://github.com/SkyworkAI/SkyReels-A1
Technical report:
- https://skyworkai.github.io/skyreels-a1.github.io/report.pdf
