Abiray/FastH3-Omni-GGUF

🎬 FastH3-Omni-8Step-Pruned — GGUF (ComfyUI Ready)

High-precision GGUF quantizations of FastVideo/FastH3-Omni-Comfy (fasth3_omniref_8step_pruned_bf16.safetensors), the 8-step consistency-distilled Omni-Reference (Ref2VA) model built on MiniMax-H3 by FastVideo (Hao AI Lab) in collaboration with Nuva Lab and Reactor.

Unlike standard naive quantizations that crush timestep modulation tables and output heads into low-bit blocks, this GGUF suite uses hybrid precision shielding to keep critical audio/video decoding heads, 32-head PDD (Progressive Distillation) offsets, and AdaLN interpolation tables in lossless FP32, while compressing the 50-block DiT + Video Sparse Attention (VSA) backbone for consumer RTX GPUs.

---

✨ What Can FastH3-Omni Do in 8 Steps?

• Synchronized Video + Stereo Audio in One Pass: Generates 24 fps video alongside synchronized dialogue, voice, sound effects, and music in just 8 sampling steps (down from 49+ steps).
• Multi-Image Omni-Reference (Ref2VA): Lock in character identity, wardrobe, props, and environment across shots using up to 3 reference images at real-time speed (or up to 9 reference images in full reference mode).
• Voice & Audio Reference Cloning: Pass up to 3 reference audio clips (2–15s each) to guide character voice timbre, ambient soundscapes, or musical style.
• First & Last Frame Keyframing + Clip Continuation: Supports exact start/end frame control plus latent-mode and frame-mode continuation.

---

📦 Quantized Model Lineup & VRAM Guide

All files are converted from the 44.24 GB Pruned BF16 checkpoint (FastVideo/FastH3-Omni-Comfy) compatibility with city96/ComfyUI-GGUF.

Filename	Quant Type	File Size	Recommended GPU VRAM	Quality & Use-Case
FastH3-Omni-8Step-Pruned-Q3_K_M.gguf	Q3_K_M	9.82 GB	12 GB – 16 GB	Ultra-lightweight fit for 12GB/16GB GPUs; protected FP32 heads preserve audio/video coherence.
FastH3-Omni-8Step-Pruned-Q4_K_S.gguf	Q4_K_S	12.7 GB	16 GB	Compact 4-bit K-quant for 16GB cards needing extra headroom for longer clip lengths.
FastH3-Omni-8Step-Pruned-Q4_K_M.gguf	Q4_K_M	12.7 GB	16 GB – 24 GB	⭐ Golden Sweet Spot: Best balance of VRAM footprint, speed, and multi-reference fidelity.
FastH3-Omni-8Step-Pruned-Q5_K_S.gguf	Q5_K_S	15.5 GB	24 GB	High-precision 5-bit quant; great for RTX 3090 / 4090 users running 768p–1344×768 clips.
FastH3-Omni-8Step-Pruned-Q5_K_M.gguf	Q5_K_M	15.5 GB	24 GB	⭐ 24GB Sweet Spot: Near-BF16 motion stability and crisp facial/voice reference lock on 24GB GPUs.
FastH3-Omni-8Step-Pruned-Q6_K.gguf	Q6_K	18.4 GB	24 GB – 32 GB+	Archival production quality; virtually indistinguishable from the 44.2 GB BF16 master.

💡 System RAM Note: When running alongside the Qwen3-VL 32B text encoder in ComfyUI, 32 GB – 64 GB of system RAM (or choosing the int8_convrot / nvfp4_awq quantized text encoder with --lowvram offloading) is strongly recommended.

---

🛡️ Hybrid Precision Shielding Architecture

8-step consistency-distilled omni-reference models are highly sensitive to quantization noise in their conditioning projectors and progressive distillation heads. During conversion, the following layers are explicitly shielded in lossless FP32 (via 1D flattening + comfy.gguf.orig_shape reconstruction metadata so llama-quantize never touches them):

• adaln_t_table & adaln_basis: Prevents the timestep interpolation (lerp) bug and protects the 8-basis modulation matrices across all 8 denoising steps.
• final_layer.video_out & final_layer.audio_out: Preserves the 32-head PDD (Progressive Distillation) first-head-plus-offset decoding heads in pure FP32 so neither video latents nor stereo audio waveforms suffer quantization hiss or color banding.
• audio_patch_proj, video_patch_proj & condition_proj: Keeps native input patch embeddings and cross-attention conditioning projections at full precision.
• All 1D RMSNorms, LayerNorms & Biases: Maintained in FP32 to prevent NaN / Inf overflow on consumer GPUs.
• 50× Video Sparse Attention (attn.to_gate_compress) + 50× DiT Blocks: Cleanly K-quant folded along 256-element boundaries ([7168, 5376]), shrinking the 3.85 GB VSA gate overhead by ~70% without structural degradation.

---

🛠️ How to Use in ComfyUI

1. Required Custom Nodes & Companion Files

1. Update ComfyUI to the latest commit (must include native MiniMaxH3ReferenceToVideo and H3 node support).
2. Install ComfyUI-GGUF by city96.
3. Download your preferred .gguf file from this repository and the companion Text Encoder and Video + Audio VAEs from Comfy-Org/MiniMax-H3:
   • Text Encoders (Comfy-Org/MiniMax-H3/text_encoders): Download Here (Pick one: bf16, int8_convrot, or nvfp4_awq)
   • Video & Audio VAEs (Comfy-Org/MiniMax-H3/vae): Download Here (Download the audio_vae_fp32 plus either video_vae_fp16 or video_vae_int8_convrot)

2. Expected ComfyUI Folder Structure

Place your downloaded files into the following directories inside ComfyUI/models/:

ComfyUI/models/
├── unet/ (or diffusion_models/)
│   └── FastH3-Omni-8Step-Pruned-Q4_K_M.gguf            # (Or your preferred quant from this repo)
├── text_encoders/
│   ├── qwen3vl_32b_minimax_h3_bf16.safetensors         # (Full BF16)
│   ├── qwen3vl_32b_minimax_h3_int8_convrot.safetensors # (Recommended for 16GB–24GB VRAM)
│   └── qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors    # (Ultra-low VRAM RTX 40/50-series)
└── vae/
    ├── minimax_h3_audio_vae_fp32.safetensors           # (Required for stereo audio decoding)
    ├── minimax_h3_video_vae_fp16.safetensors           # (Standard FP16 Video VAE)
    └── minimax_h3_video_vae_int8_convrot.safetensors   # (Optional low-VRAM Video VAE)

3. Recommended 8-Step Sampler Settings (FastH3-Omni Contract)

Load the model using the UnetLoaderGGUF node and wire it to MiniMaxH3ReferenceToVideo with these exact distillation settings:

• Steps: 8
• CFG / Guidance Scale: 1.0 (Distilled — do not use high CFG)
• Video Sigma Shift: 12.0 (Note: FastH3-Omni uses 12.0, whereas FastH3 V2 used 10.0)
• Audio Sigma Shift: 3.0
• Frame Rate: 24 fps
• Supported Aspect Ratios: 1344×768 (16:9), 768×768 (1:1), 768×1344 (9:16), 1024×768 (4:3)
• Supported Clip Durations: 5.875s, 8.0s, 10.125s, 12.25s, or 14.375s

4. Prompting Tip for Image & Audio References

When connecting reference images and audio clips, explicitly assign roles in your prompt:

• Reference images: Label them in order and specify whether each is a character, setting, object, or style.
• Reference audio: State which character the voice belongs to and put the exact spoken dialogue inside quotes in your prompt.

---

📜 License & Credits

• Base Architecture: MiniMax-H3 by MiniMax — licensed under the MiniMax H3 Community License Agreement.
• Companion Text Encoders & VAEs: Packaged by Comfy-Org (Comfy-Org/MiniMax-H3).
• 8-Step Omni Distillation & Comfy Packaging: FastVideo / Hao AI Lab in collaboration with Nuva Lab and Reactor.
• GGUF Quantization & Hybrid FP32 Layer Shielding: Quantized and packaged by @Abiray using a patched diffusion build of llama.cpp and city96/ComfyUI-GGUF.