comfyui-voice-pipeline
Generate character voices using TTS, voice cloning, and lip-sync tools. Supports Chatterbox, F5-TTS, TTS Audio Suite, RVC, and ElevenLabs. Use when creating speech audio for characters or syncing audio to video.
- Installs
- 401
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/mckruz/comfyui-expert --skill comfyui-voice-pipeline- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads comfyui-voice-pipeline automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
ComfyUI Voice Pipeline
Creates character voices through TTS/voice cloning and synchronizes them with generated video.
Voice Generation Decision Tree
VOICE REQUEST
|
|-- Have reference audio of target voice?
| |-- Yes (5+ seconds) → Chatterbox (MIT, paralinguistic tags)
| |-- Yes (10-15 seconds) → F5-TTS (fastest zero-shot)
| |-- Yes (10+ minutes) → RVC training (highest fidelity)
| |-- Yes (any length, budget) → ElevenLabs (production quality)
|
|-- No reference audio?
| |-- Need emotion control → IndexTTS-2 (8-emotion vectors)
| |-- Need multi-language → TTS Audio Suite (23 languages)
| |-- Need voice design → ElevenLabs Voice Design (describe voice)
| |-- Quick prototype → Any TTS with default voice
|
|-- Need multi-speaker dialog?
| |-- Chatterbox (4 voices) or TTS Audio Suite (character switching)
|
|-- Need lip-sync?
| |-- Best accuracy → Wav2Lip + CodeFormer
| |-- Need head movement → SadTalker
| |-- Full expression control → LivePortrait
| |-- Unlimited length → InfiniteTalk
Tool Reference
Chatterbox (Recommended Open-Source)
Strengths: MIT license, beats ElevenLabs 63.8% in blind tests, 5-second sample, emotion control, sub-200ms latency.
Paralinguistic tags:
[laugh] [chuckle] [sigh] [gasp] [cough] [clear throat]
[whisper] [excited] [sad] [angry] [surprised]
Key parameter: exaggeration (0.25-2.0) controls expressiveness.
Limit: 40-second generation cap. Split longer content.
F5-TTS
Strengths: Fastest zero-shot cloning, <15 second samples, MIT license, multi-language.
Requirements: Reference audio must be paired with .wav + .txt (matching transcription).
Languages: English, German, Spanish, French, Japanese, Hindi, Thai, Portuguese.
TTS Audio Suite
Strengths: Unified multi-engine platform, 23 languages, character switching.
Special features:
- Character switching:
[CharacterName]tags - Language switching:
[de:Alice],[fr:Bob] - Pause control:
[pause:1s] - SRT timing sync
Integrates: F5-TTS, Chatterbox, Higgs Audio 2, VibeVoice, IndexTTS-2, RVC.
IndexTTS-2
Strengths: 8-emotion vector control with per-segment parameters.
Emotions: happy, angry, sad, surprised, afraid, disgusted, calm, melancholic.
RVC (Voice Conversion)
Use case: Train a model on target voice (10+ min audio), then convert any TTS output.
Pipeline: Text → Any TTS → Base Audio → RVC Model → Character Voice
Training: 300-500 epochs, RMVPE feature extraction.
ElevenLabs (Commercial)
Tiers:
- Instant Clone: 1-minute sample, good quality
- Professional Clone: 30+ minutes (3h ideal), near-indistinguishable
- Voice Design: Describe voice in text (no sample needed)
Voice Profile Setup
For each character, establish a voice profile in projects/{project}/characters/{name}/profile.yaml:
voice:
cloned: true
model: "chatterbox"
sample_file: "references/voice_sample.wav"
settings:
exaggeration: 1.2
default_emotion: "neutral"
notes: "Warm, confident tone. Slight Italian-American undertones."
Script Preparation
Text Formatting for TTS
- Punctuation matters: Commas create pauses, periods create stops
- Phonetic hints: Spell unusual words phonetically if mispronounced
- Emotion cues: Use Chatterbox tags or split by emotion for IndexTTS-2
- Length: Split into 30-40 second segments for Chatterbox limit
Multi-Speaker Script
[Sage] Hello! *laughs* I've been looking forward to this.
[pause:0.5s]
[Alex] [excited] Same here! Let's dive right in.
[Sage] [whisper] But first, I need to tell you something...
Audio Post-Processing
Requirements for Lip-Sync Input
- Sample rate: 16-24kHz (model dependent)
- Format: WAV (uncompressed)
- Mono channel
- Trim leading silence
- Add 0.2s trailing silence
- Normalize to -3dB peak
FFmpeg Processing
# Convert to mono 24kHz WAV, normalized
ffmpeg -i input.wav -ac 1 -ar 24000 -af "loudnorm=I=-16:TP=-3" output.wav
# Trim silence from start/end
ffmpeg -i input.wav -af "silenceremove=start_periods=1:start_threshold=-50dB,areverse,silenceremove=start_periods=1:start_threshold=-50dB,areverse" trimmed.wav
# Concatenate segments
ffmpeg -f concat -safe 0 -i filelist.txt -c copy combined.wav
Lip-Sync Methods
Wav2Lip (Best Accuracy)
Settings:
wav2lip_model: "wav2lip_gan.pth" # Better than wav2lip.pth
face_detect_batch: 16
nosmooth: false
pad_bottom: 10
MUST post-process: CodeFormer (fidelity 0.7) after Wav2Lip output.
SadTalker (Head Movement)
Settings:
preprocess: "full" # Better for novel faces
enhancer: "gfpgan"
pose_style: 10-20 # Natural conversation range
LivePortrait (Expression Control)
Settings:
lip_zero: 0.03 # Reduces unnatural lip movement
stitching: true # Seamless face blending
Best for: Premium avatar creation, expression transfer from driving video.
LatentSync 1.6 (Newest, Highest Quality)
ByteDance model trained at 512x512 with TREPA modules for temporal consistency.
InfiniteTalk (Unlimited Length)
For videos longer than standard lip-sync limits. Integrates with Wan for joint generation.
Complete Talking Head Workflow
Pipeline A: Quick (Image → Talk)
1. [Text] → Chatterbox/F5-TTS → audio.wav
2. [Character Image] + audio.wav → SadTalker → video.mp4
3. video.mp4 → GFPGAN/CodeFormer → final.mp4
Time: ~2 minutes. Quality: Good.
Pipeline B: Quality (Image → Video → Lip-Sync)
1. [Text] → Chatterbox → audio.wav
2. [Character Image] → Wan I2V → base_video.mp4
Prompt: "person talking, slight head movement, indoor"
3. base_video.mp4 + audio.wav → Wav2Lip → lipsync.mp4
4. lipsync.mp4 → FaceDetailer batch → enhanced.mp4
5. enhanced.mp4 → Color correct + Deflicker → final.mp4
Time: ~10 minutes. Quality: Production.
Pipeline C: Premium (Expression Transfer)
1. Record driving video (actor performing lines)
2. [Text] → Voice Clone TTS → audio.wav
3. [Character Image] + driving.mp4 → LivePortrait → expression_video.mp4
4. expression_video.mp4 + audio.wav → Wav2Lip → lipsync.mp4
5. lipsync.mp4 → CodeFormer → final.mp4
Time: ~15 minutes. Quality: Premium.
Troubleshooting
| Issue | Solution |
|---|---|
| Audio out of sync | Offset with ffmpeg: ffmpeg -itsoffset 0.1 -i audio.wav ... |
| Subtle mouth movements | Use wav2lip_gan.pth, increase audio volume |
| Face artifacts | Post-process with CodeFormer (fidelity 0.6-0.8) |
| Robotic voice clone | Use longer/cleaner reference, increase exaggeration |
| Unnatural head movement | Lower SadTalker pose_style to 0-10 |
Reference
references/voice-synthesis.md- Full voice tool documentationreferences/models.md- Voice model download links- Character voice profiles in
projects/{project}/characters/
FAQ
What does the comfyui-voice-pipeline skill do?
Generate character voices using TTS, voice cloning, and lip-sync tools. Supports Chatterbox, F5-TTS, TTS Audio Suite, RVC, and ElevenLabs. Use when creating speech audio for characters or syncing audio to video.
How do I install comfyui-voice-pipeline?
Run npx skills add https://github.com/mckruz/comfyui-expert --skill comfyui-voice-pipeline in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/comfyui-voice-pipeline/ (personal) or .claude/skills/comfyui-voice-pipeline/ (project).
Which AI agents can use comfyui-voice-pipeline?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is comfyui-voice-pipeline safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains comfyui-voice-pipeline?
It is published by mckruz in mckruz/comfyui-expert. It has 401 recorded installs.
More from mckruz/comfyui-expert
comfyui-workflow-builder
mckruz/comfyui-expert
Generate, build, create, or design ComfyUI workflow JSON from natural language descriptions. Produces valid node graphs with correct classtypes, connections, output indices, and model-appropriate settings. Handles txt2img, img2img, inpainting, ControlNet, LoRA stacking, upscaling, and face detailing pipelines. Does NOT cover ComfyUI installation, custom node development, Python scripting, model training, hardware advice, or architectural explanations.
comfyui-api
mckruz/comfyui-expert
Connect to a running ComfyUI instance, queue workflows, monitor execution, and retrieve results. Supports both online (REST API) and offline (JSON export) modes. Use when executing ComfyUI workflows or checking server status.
comfyui-prompt-engineer
mckruz/comfyui-expert
Craft model-specific prompts optimized for the target checkpoint and identity method. Handles FLUX, SDXL, SD1.5, and Wan video models with proper syntax, quality tags, and negative prompts. Use when generating or refining prompts for ComfyUI workflows.
comfyui-video-pipeline
mckruz/comfyui-expert
Generate videos using ComfyUI with Wan 2.2, FramePack, or AnimateDiff. Handles image-to-video, text-to-video, talking heads, and motion-controlled animation. Use when creating any video content from character images or text descriptions.
comfyui-troubleshooter
mckruz/comfyui-expert
Diagnose ComfyUI errors, workflow failures, and quality issues. Suggests fixes based on error patterns, missing dependencies, and community-known workarounds. Use when ComfyUI workflows fail or produce unexpected results.
comfyui-character-gen
mckruz/comfyui-expert
Build identity-preserving character generation workflows and pipelines in ComfyUI. Selects the optimal identity method (InfiniteYou, FLUX Kontext, PuLID, InstantID, IP-Adapter) based on use case requirements. Handles face preservation, likeness transfer, cross-domain conversion (3D to photo), multi-reference consistency, iterative character editing, and character variation generation. Triggers on requests to generate consistent characters, preserve identity across images, create face-swapping workflows, or convert 3D renders to photorealistic portraits. Does NOT cover general image generation without identity preservation, model training/LoRA fine-tuning, animation, technical explanations, or workflow debugging.
Popular media & creative skills
hyperframes
heygen-com/hyperframes
Read first to make, edit, animate or render any video with HyperFrames (HTML rendered to video), including launch films, explainers, music videos, charts, maps, vertical clips, title cards, captions and overlays. Holds the render contract, the work loop and what each kind of video needs.
media-use
heygen-com/hyperframes
Get or make any media for a HyperFrames video - music, sound effects, images, icons, logos, voiceover, avatar and generated video, transcripts, cutouts, colour grades and looks - with HeyGen or free local tools, and mix placed audio. Also for footage that looks dark, flat or dull, or should be stylized or hidden.
hyperframes-registry
heygen-com/hyperframes
About 450 ready-made HyperFrames blocks and components installed as editable source - transitions, overlays, titles, kinetic type, captions, charts, map overlays, 3D, camera moves. Use it before hand-building any named visual (CRT scanlines, glitch, film grain, a chart); also for hyperframes add and catalog.
hyperframes-creative
heygen-com/hyperframes
Non-animation creative direction for HyperFrames videos. Use for design spec (frame.md / design.md) handling, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns, and brand / style decisions. For atomic motion patterns and scene blueprints, use hyperframes-animation.
ai-video-generation
101-skills/superpowers
Generate AI videos with Google Veo, Seedance 2.0, HappyHorse, Wan, Grok and 40+ models via inference.sh CLI. Models: Veo 3.1, Veo 3, Seedance 2.0, HappyHorse 1.0, Wan 2.5, Grok Imagine Video, OmniHuman, Fabric, HunyuanVideo. Capabilities: text-to-video, image-to-video, reference-to-video, video editing, lipsync, avatar animation, video upscaling, foley sound. Use for: social media videos, marketing content, explainer videos, product demos, AI avatars. Triggers: video generation, ai video, text to video, image to video, veo, animate image, video from image, ai animation, video generator, generate video, t2v, i2v, ai video maker, create video with ai, runway alternative, pika alternative, sora alternative, kling alternative, seedance, happyhorse
ai-image-generation
101-skills/superpowers
Generate AI images with GPT-Image-2, FLUX, Gemini, Grok, Seedream, Reve and 50+ models via inference.sh CLI. Models: GPT-Image-2, FLUX Dev LoRA, FLUX.2 Klein LoRA, Gemini 3 Pro Image, Grok Imagine, Seedream 4.5, Reve, ImagineArt. Capabilities: text-to-image, image-to-image, inpainting, LoRA, image editing, upscaling, text rendering. Use for: AI art, product mockups, concept art, social media graphics, marketing visuals, illustrations. Triggers: flux, image generation, ai image, text to image, stable diffusion, generate image, ai art, midjourney alternative, dall-e alternative, text2img, t2i, image generator, ai picture, create image with ai, generative ai, ai illustration, grok image, gemini image, gpt image, openai image, chatgpt image