AI Music Video Lip Sync After H3 Went Open Source: What Changed in the Workflow

It is 2am and you are staring at a ten-second close-up of your singer. Her mouth opens half a beat after the kick drum. On the long chorus vowel she looks like she is singing a different word. Then you find a thread: H3 is open source, and someone got lip sync running on an 8GB graphics card. You start doing the math. Buy a GPU and run it yourself?

That is not really a question of how much tinkering you can stand. It is a reshuffle of who does what. Since MiniMax open-sourced H3 in August 2026, lip sync has split from “the video model does it on the side” into several layers: speed-ups, post-trained models and hosted endpoints. Each solves a different problem at a different price.

Most “H3 is open source” coverage is about parameter counts and benchmarks. This post answers three questions music video makers actually have: how to lip-sync sung shots, whether to run locally or use a hosted API, and when running the open weights yourself is really worth it.

Table of Contents

• What the H3 open-source release actually includes
• Three layers of derivatives speed post-training hosting
• How lip sync on sung shots has changed
• Local deployment vs hosted API
• When self-hosting the open weights pays off
• The music video workflow after open source
• Frequently asked questions

What the H3 open-source release actually includes

MiniMax’s open-source announcement is dated August 3, 2026. It ships two BF16 checkpoints:

• Base FL2VA: text to audio-video, plus first frame, last frame, or first-and-last frame to audio-video.
• Base Ref2VA: reference to audio-video, taking reference images, videos and audio.

The backbone is a 33B-parameter dense Transformer. About 13B of that sits in AdaLN branches that inference-only deployments can skip. The text encoder is the full Qwen3-VL-32B. Output runs 4 to 15 seconds at 24 FPS with 32 kHz stereo audio.

What was not released matters just as much. The hosted Context-IR preprocessing and the Regenerate-2K refinement module are not open. The 2K clips you see from MiniMax or fal’s hosted endpoints cannot come straight out of the local weights. The license is the MiniMax H3 Community License, not Apache or MIT.

Illustration: SunoMV Team · Open weights make a sung music video on your own machine possible, but not every capability comes with them

Practical rule: Separate what the open weights can do from what the hosted system does. 2K refinement and preprocessing live on the hosted side.

Three layers of derivatives speed post-training hosting

Three months in, the ecosystem is busy. MiniMax’s own awesome-minimax-h3-integration list collects a large number of community projects. For music video makers, three layers by purpose are enough.

Speed. The flagship is FastH3 from Hao AI Lab at UCSD (preview, August 27, 2026). DMD2 distillation cuts diffusion steps from 49 to 4, and video sparse attention runs at 90% sparsity. Their numbers: a 15-second clip in 47.2 seconds on one B200, against 678.7 seconds for base H3, about 14.4x. The preview supports text to audio-video only; reference-to-video and first/last frame are still in development. The community also has 8GB VRAM workflows, GGUF quantizations and a native Apple Silicon runtime.

Post-training. HeyGen Video is HeyGen’s general video model, built on H3 and post-trained by HeyGen: 5 to 15 second clips with sound, in text, image and reference modes. There are also hosted “H3 Max Lip Sync” style products that take a portrait plus an audio track and return a talking or singing video. Alibaba PAI released a Fun-ControlNet for H3 that drives motion from pose or depth.

Hosting. MiniMax’s own API, plus H3, H3 Max and H3 Max Turbo on fal. You never touch a GPU, you pay per second, and you get the full system, including the unreleased 2K refinement.

Illustration: SunoMV Team · Speed-ups earn their keep by producing more drafts to choose from, not by finishing in one take

In SunoMV’s last 90 days of clips, Seedance Fast made about 69% and H3 Max Turbo about 5.8%, roughly 6 times H3 Max. Music video makers lean toward fast tiers by nature. That is why the speed layer matters most for music videos: more draft rounds, not a better single shot.

How lip sync on sung shots has changed

H3’s Ref2VA already accepts reference audio. Feed it the song and the character moves its mouth to the voice. The community has a set of audio-driven lip sync ComfyUI workflows built on Ref2VA, and Japanese creators have published nodes that keep the external audio and only move the mouth.

That shifts lip sync from a post step into the generation step. In a music video we still treat it as two jobs:

1. Audio at generation time handles motion and emotion following the song. Body rhythm, breathing and expression across a chorus are a good fit for reference audio.
2. Lip sync after generation handles whether this lyric line looks right. On close-ups, long vowels and plosives, viewers catch the mismatch immediately.

Illustration: SunoMV Team · Only close-up sung shots need word-level lip sync; wide shots are fine with reference audio

In our experience only a handful of close-up sung shots in a music video need word-level sync. That is why SunoMV’s precise lip-sync only opens on shots of 15 seconds or less: it fixes the mouth, it does not reshoot. Our guide How to lip sync a Suno music video covers how to pick those shots.

Practical rule: Let reference audio carry rhythm in wide and medium shots, then fix mouths only on close-up sung lines. Do not run word-level lip sync on all forty shots.

Local deployment vs hosted API

The big draw of open source is “free.” The weights are free. The GPUs, the time and the license terms are not.

Factor	Run the open weights locally	Hosted API (fal / MiniMax)	Post-trained hosted model (e.g. HeyGen Video)
Best for	Teams with GPUs who batch or fine-tune	Independent musicians, small teams	Sung shots that need stronger faces and mouths
Quality ceiling	No 2K refinement; depends on quantization and steps	Full system, up to 2K	Depends on each tier
Speed	Hardware-bound; speed-ups can be an order of magnitude faster	Steady, queued	Steady
Cost shape	Upfront hardware, power, maintenance time	Per second	Per second
Compliance	Bound by the community license; check it yourself	Platform handles endpoint-side terms	Platform handles endpoint-side terms

The hardware bar is higher than the threads suggest. The official SGLang example uses 4 GPUs with sequence parallelism. According to nxcode’s breakdown, each task’s weight folder is about 144 GB, and Simon Willison’s MLX run on an M5 Max laptop took just under 45 minutes for one clip. 8GB cards work by trading precision and resolution.

When self-hosting the open weights pays off

I would consider running the weights myself in three cases:

• A fixed singer identity. If one virtual artist will appear in dozens of music videos, a character LoRA trained on the open weights is steadier than feeding nine reference images every time. MiniMax says it released the complete weights to support fine-tuning.
• Batch drafts. An album of a dozen songs, each with dozens of draft shots, run through a speed-up on your own GPU costs close to the power bill at the margin.
• Controllable motion. Pose- and depth-driven ControlNet exists only on the open side, and dance videos use it well.

Do not self-host if you make one or two videos, need 2K output, or live where the license restricts you. Per nxcode’s reading of the license, its territory excludes the United States, the United Kingdom, the European Union and South Korea, which need separate authorization from MiniMax. Organizations with more than $20 million in annual revenue need written commercial authorization, and using H3 outputs to train non-H3 models is restricted. That is a third-party reading; check MiniMax’s license text before you start.

Illustration: SunoMV Team · Self-hosting or hosted: count hardware, time and license first

This video walks through a free music video pipeline built on open H3 in ComfyUI, so you can see the local steps and timing:

Video: The AI Brief · A music video pipeline in ComfyUI with open MiniMax H3

Decision filter: Do you need one fixed singer across a large number of shots? Then local weights and a LoRA are worth a look. If not, hosted endpoints cost less.

The music video workflow after open source

Open source did not make hosting redundant. It lets each step use the layer that fits. A practical flow:

1. Storyboard and drafts: storyboard stills go through image-to-video on a fast tier (hosted H3 Max Turbo, or a speed-up on your own GPU) to rough out the whole song.
2. Face lock: shots where the singer recurs use reference-to-video. H3 Max Turbo and H3 Max cost the same to lock a face; Turbo is faster, H3 Max has higher quality and can pin frames. Details in H3 Max Turbo face lock for music videos compared.
3. Rhythm: medium and wide shots with the singer get the song as reference audio.
4. Lip sync: only close-up sung shots get lip-sync.
5. Final: key shots are re-rendered on a higher-quality tier and exported.

In SunoMV, steps 1, 2 and 4 already sit in one flow: Suno link to music video builds the storyboard, you render with H3 Max or Turbo, character images trigger face lock automatically, and close-ups get precise lip-sync. Turbo prompts are covered in the H3 Max Turbo prompt guide. For the whole H3 open-source map, read the pillar MiniMax H3 open-source ecosystem explained. For short dramas, the production-line view is on DramaSo.

Frequently asked questions

Is lip sync free now that H3 is open source?

The weights are free, but running them takes GPUs and time, and 2K refinement was not released. For a few music videos, a hosted endpoint plus post-generation lip sync is usually less work.

Can FastH3 make face-locked music videos today?

Not yet. The FastH3 preview supports text to audio-video only. Reference-to-video and first/last frame are in development, so it is a draft tool for now.

How is HeyGen Video related to H3?

HeyGen describes HeyGen Video as a general video model built on MiniMax H3 and post-trained by HeyGen, with text, image and reference modes and 5 to 15 second clips with sound.

If I feed the song at generation time, do I still need lip sync?

It depends on the shot. Reference audio handles rhythm and emotion, which is enough for wide and medium shots. Close-up sung lines that must match word for word still benefit from post-generation lip sync.

I am in the US or EU. Can I run the H3 weights myself?

Per third-party readings of the license, those regions need separate authorization from MiniMax. Check the official license text. Hosted endpoints are not affected by that clause.

Open source did not change who gets to make a music video. It changed which layer each step can use. Spend your money and time on the few sung close-ups viewers will actually stare at.

Make a music video where the mouths line up:

• 🌐 SunoMV: https://suno.bi/en
• 🎬 Audio to video: https://suno.bi/en/audio-to-video-ai-generator
• 🎤 Precise lip-sync: https://suno.bi/en/features/how-to-lip-sync-suno-music-video