Meta The GenAI team launched a Fairy The team demonstrated Fairy’s performance in several applications, including character/object replacement, stylization, and long-form video generation.
For example, a simple text prompt, such as "in the style of Van Gogh," is enough to edit the source video. For example, the text command "turn into a snowman" turns the astronaut in the video into a snowman.
Visual coherence in Fairy is a particularly challenging problem, as there are countless ways to modify a given image based on the same cue. Fairy uses cross-frame attention, a mechanism that implicitly propagates diffuse features, ensuring superior temporal coherence and high-fidelity synthesis.
The model can generate a 512x384 pixel, 120-frame (4 seconds at 30fps) video in just 14 seconds, at least 44 times faster than previous models. Like Meta's Emu video model, Fairy is based on a diffusion model for image processing, enhanced for video editing.
Fairy processes all frames of the source video without temporal downsampling or frame interpolation, and maintains the aspect ratio of the horizontal output video at 512. When tested with six A100 GPUs, Fairy was able to render a 27-second video in 71.89 seconds with high visual consistency.
Fairy’s performance was tested in an extensive user study with 1,000 generated samples. Both human judgment and quantitative metrics confirmed that Fairy outperformed Rerender, TokenFlow, and Gen-1.
However, the model currently has problems handling dynamic environmental effects like rain, fire, or lightning, which either don't fit well into the overall scene or produce visual errors.
Despite these issues, the research team believes their work represents a significant advance in the field of AI video editing, with a transformative approach to temporally consistent and high-quality video synthesis.