A technique that uses AI to reshape the mouth of a face in video so it matches an audio track — used to dub, re-voice, or animate a talking face without reshooting.
Last verified · 2026-07-18 · by Moe Ameen
AI lip sync is the technique of driving the mouth movements of a face in video from an audio track, so the lips appear to speak whatever the audio says. You feed the model two inputs — a face (a video clip or a single still photo) and an audio clip (recorded, cloned, or synthesized) — and it regenerates the mouth region frame by frame to match the phonemes in the audio. The rest of the face and the background stay as they were. It is a video *editing* operation on the mouth, distinct from full talking-avatar generation, which creates an entire face from scratch.
There are two common shapes. The first is talking-photo: a single still image plus audio becomes a short clip of that face speaking, with the model inventing plausible mouth and slight head motion. The second, and the one that matters most commercially, is visual dubbing: taking existing footage of a real person and re-syncing their mouth to a new audio track — a different language, a re-recorded line, or a corrected take — so the dub no longer looks like an out-of-sync overdub.
Under the hood the model reads the audio as a sequence of features (typically a mel-spectrogram), maps those to mouth shapes, and generates the lip region conditioned on the surrounding face so identity, lighting, and skin texture are preserved. Early production models were GAN-based; by 2026 diffusion-based approaches dominate because they produce finer mouth detail and fewer artifacts in close-ups, where GAN blur was most visible. The trade-off is speed, which latent-diffusion and distillation techniques have largely closed.
AI lip sync is the enabling primitive under a lot of products people think of as separate: AI video translation and dubbing, talking-photo apps, and the native lip animation inside [avatar video](/glossary/avatar-video) platforms. It underperforms when the head turns sharply, when teeth and tongue detail get scrutinized in extreme close-up, and — the honest caveat — it is the same technology that powers non-consensual deepfakes, which is why disclosure and likeness rules increasingly apply to it. For the buyer's-guide version of translating footage this way, see [how to translate a video with AI](/how-to/translate-a-video-with-ai).
The category's technical anchor is Wav2Lip, published as "A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild" at ACM Multimedia 2020 by researchers at IIIT Hyderabad (Prajwal, Mukhopadhyay, Namboodiri, and Jawahar). Its key idea was to train the generator against a strong pre-trained lip-sync discriminator — a "lip-sync expert" — rather than relying on reconstruction loss alone, which pushed accuracy on arbitrary faces close to real synced video and made lip sync work "in the wild" on people the model had never seen. Wav2Lip was open-sourced and became the reference implementation the whole space built on.
The lineage runs forward from there. The Wav2Lip team went on to found Sync Labs (sync.), which turned the research into a commercial visual-dubbing API and, in 2026, shipped sync-3 as a sharper diffusion-era model. In parallel, HeyGen made lip-synced translation mainstream — its video-translate product dubs a clip into 175+ languages with the speaker's mouth re-synced to the new audio — and D-ID kept the talking-photo path cheap and accessible. The architectural through-line was the 2023–2025 shift from GAN-based mouth-shape generation to diffusion-based facial animation, which is what took lip sync from "obviously edited" to "hard to spot in a 30-second clip."
| Platform | Behavior |
|---|---|
| Wav2Lip (open-source) | The 2020 GAN reference model. Free, self-hostable, works on arbitrary faces, and still the baseline many pipelines start from. Output is lower-resolution and softer around the mouth than modern commercial models, so it is best for prototypes, low-stakes clips, or as a stage inside a larger pipeline. |
| sync. (sync-3) | The commercial descendant of Wav2Lip, built for developers as a visual-dubbing API. Diffusion-era quality with a focus on a clean, controllable mouth to build products around, rather than a consumer app. |
| HeyGen | Lip sync wrapped inside translation and avatar products rather than exposed as a raw tool. Its video-translate feature re-syncs a real speaker’s mouth to a cloned voice in 175+ languages, and its avatars lip-sync generated scripts natively at render time. |
| D-ID | Talking-photo focused and cheapest entry point. Turns a single still image plus audio into a talking-head clip. Strong for simple explainers and product demos; mouth realism trails HeyGen on filmed-video re-sync. |
The useful way to think about lip sync is as a *repair-and-relocalize* tool, not a *create* tool. It shines when you already have footage and want to change what the mouth says — dub it, fix a line, re-voice it — and it is the wrong instrument when you need net-new talking video from nothing. That distinction is why, inside [Kompozy](/), lip sync is not a standalone button but the native engine underneath the persona video formats. When Kompozy generates a [Persona Short](/glossary/persona-shorts) or a [Persona Frames](/glossary/persona-frames) clip, HeyGen renders the avatar speaking the generated script with lip-sync baked in at generation time — the mouth is synced to synthesized speech from the first frame, so there is no separate re-sync pass and no out-of-sync artifact to repair. That is the practical division of labor: dedicated lip-sync tools like [sync.](/ai-tools/sync) exist to fix or dub footage you shot, while an engine like Kompozy generates the talking video from a script and then fans it across nine platforms with captions and scheduling attached. If your job is localizing an existing library, reach for a dubbing tool; if your job is producing a steady stream of on-brand persona video you never filmed, the lip sync should be invisible and upstream.
AI lip sync is a technique that reshapes the mouth of a face in video to match an audio track, so the lips appear to speak whatever the audio says. You give a model a face (a clip or a still photo) plus audio, and it regenerates the mouth region frame by frame to match the phonemes while keeping the rest of the face and background unchanged.
The model reads the audio as features such as a mel-spectrogram, maps those to mouth shapes, and generates the lip region conditioned on the surrounding face so identity and lighting are preserved. Early production models like Wav2Lip were GAN-based; by 2026 diffusion-based approaches dominate because they produce finer mouth detail and fewer close-up artifacts.
Lip sync edits the mouth of a face you already have to match new audio. Avatar video generates an entire talking figure from scratch. They overlap — avatar platforms use lip-sync engines internally — but a dedicated lip-sync tool is for dubbing or fixing existing footage, while avatar generation creates net-new video.
Wav2Lip is the influential open-source lip-sync model published at ACM Multimedia 2020 by researchers at IIIT Hyderabad, titled "A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild." It trained the generator against a strong pre-trained lip-sync discriminator, which made accurate lip sync work on arbitrary faces the model had never seen. Its team later founded the commercial platform sync.
The biggest use is video dubbing and translation — re-syncing a real speaker’s mouth to a new-language or re-recorded audio track so the dub does not look out of sync. It is also used for talking-photo clips from a single image, and for fixing a flubbed line without a reshoot. AI dubbing this way runs far cheaper and faster than traditional studio dubbing.
It is the underlying technology deepfakes use, but the term itself is neutral — it covers legitimate dubbing, localization, and animation as much as misuse. The dividing line is consent and disclosure: use a likeness you own or have a release for, and label AI-generated content per each platform’s rules.