Meta, in collaboration with King Abdullah University of Science and Technology (KAUST), has launched MarDini, a new family of video diffusion models aimed at elevating video generation in generative AI.

This breakthrough technology offers capabilities, including filling in missing frames, creating animated sequences from single images, and extending short clips with continuous, natural frames. MarDini’s introduction follows Meta’s recent releases like Emu Video, Emu Edit, and Movie Gen, all of which showcase Meta's commitment to establishing a strong presence in the generative AI video domain.

Enhanced Video Generation Capabilities

MarDini is designed to handle various video generation tasks seamlessly, from interpolating frames for smoother scene transitions to generating fluid animations rivalling high-end, costly models. Leveraging masked auto-regression (MAR) within a single diffusion model (DM), MarDini offers creators a flexible tool for producing high-quality videos, enabling efficient video expansion and seamless frame creation.

Innovative Model Architecture and Training Approach

MarDini’s architecture comprises two main components: a planning model and a generation model. The planning model uses MAR to interpret input frames, creating guiding signals for frames that need to be synthesized. The generation model then uses a diffusion process to produce high-resolution frames, producing a polished and cohesive video. MarDini’s progressive training approach allows it to train directly on unlabelled video data, bypassing the need for pre-trained image models and enhancing its adaptability to different frame configurations.

Why MarDini Stands Out

MarDini’s unique flexibility and robust performance make it a standout in generative video AI. It excels in tasks like video interpolation, filling in frames to smooth out transitions; image-to-video generation, transforming a single frame into a dynamic video sequence; and video expansion, extending clips with continuous frames. Its efficient design allows it to set new benchmarks in video generation, offering high-quality output in significantly fewer steps than traditional models, making it a cost-effective and time-efficient solution.

According to Meta’s recent research publication, MarDini achieved competitive results on various animation and interpolation benchmarks with a lower computational demand than other models with similar parameters. MarDini represents an impressive stride in generative AI, underscoring Meta’s dedication to expanding the capabilities and accessibility of video content creation.

Source: MarDini

Image source: Unsplash