TBDub: Production-Oriented Visual Dubbing

TBDub is a production-oriented visual dubbing system developed by the TaoLiveAIGC Team at Taobao & Tmall Group, Alibaba Group.

Given a source video and driving audio, TBDub generates synchronized lip movements while preserving the speaker's identity, facial details, and temporal consistency. The model is designed to remain robust under large head poses, rapid motion, facial occlusion, cross-speaker audio, and AIGC-generated source videos.

Highlights

• High-fidelity identity preservation: preserves lip color, teeth structure, facial texture, and fine-grained identity details.
• Temporal consistency: reduces flickering, unstable illumination, and long-sequence identity degradation.
• Robust visual dubbing: supports challenging poses, occlusion, rapid head motion, and cross-domain inputs.
• Efficient inference: includes a distilled two-step Student model for faster generation.

Model Variants

File	Description
tbdub_base.safetensors	Base TBDub checkpoint
tbdub_finetune.safetensors	Fine-tuned checkpoint used for standard inference
tbdub_student.safetensors	Distilled Student checkpoint for two-step inference
null_prompt_emb.pt	Null text embedding used during inference

For standard inference, load tbdub_base.safetensors first and then overlay tbdub_finetune.safetensors.

For distilled inference, load tbdub_base.safetensors first and then overlay tbdub_student.safetensors.

Installation and Inference

The inference code and complete installation instructions are available in the GitHub repository:

• GitHub: https://github.com/TaoLiveAIGC/TBDub

Clone the repository:

git clone https://github.com/TaoLiveAIGC/TBDub.git
cd TBDub

Download the TBDub checkpoints:

hf download TaoLiveAIGC/TBDub --local-dir checkpoints

Run standard inference:

bash infer.sh path/to/source.mp4 path/to/driving.wav results

Please refer to the GitHub README for auxiliary Wan, UMT5, HuBERT, and DWPose checkpoints and detailed command-line options.

Intended Use

TBDub is intended for research and authorized applications involving visual dubbing, video localization, digital humans, and audio-driven facial animation.

Users are responsible for obtaining appropriate consent and ensuring that generated content complies with applicable laws, licenses, and platform policies.

Limitations

The visual quality of the generated video remains dependent on the quality of the source video. Although TBDub can improve the apparent clarity of the lips and teeth relative to the input, it cannot recover high-fidelity facial details from severely low-resolution videos. Additional face restoration or face super-resolution may therefore be required for such inputs.

The current quantitative evaluation is primarily based on self-driven reconstruction, where each source video is reconstructed using its original audio. These metrics may not fully reflect performance in cross-driven settings with mismatched or cross-identity audio. Moreover, automatic metrics do not always align with human perception; quantitative results should therefore be considered together with qualitative video comparisons.

Authors

Bihan Li, Xinyang Li, Zeran Xu, Meiguang Jin, and Junfeng Ma
TaoLiveAIGC Team, Taobao & Tmall Group, Alibaba Group

License

The released model weights and code are provided under the Apache License 2.0 unless otherwise specified. Third-party dependencies and checkpoints remain subject to their respective licenses.

Citation

@misc{tbdub2026,
  title  = {TBDub: Production-Oriented Visual Dubbing},
  author = {Li, Bihan and Li, Xinyang and Xu, Zeran and
            Jin, Meiguang and Ma, Junfeng},
  year   = {2026},
  url    = {https://github.com/TaoLiveAIGC/TBDub}
}