VTM-1.5.1

Turn one anime picture into a live VTuber. A keypoint-driven DiT draws your character in whatever pose and expression your face gives it, frame by frame.

These are the weights behind VTM Spark, the free Windows app that tracks you with a webcam or iPhone and sends the character to OBS, Discord or Zoom as a webcam.

Status: beta · Platform: Windows + NVIDIA CUDA

Every frame on the right was drawn by VTM-1.5.1.pt from the one picture on the left, recorded straight from the VTM Spark desk with Max FPS at 100 (no cap in practice) and the Ultra decoder on an RTX 5060 Ti: about 60 frames a second shown. The head motion is a recorded iPhone (iFacialMocap) session replayed through the app instead of a live face.

How it works

Use it

The easy way is VTM Spark. It downloads everything in this repo for you:

1. Download VTM Spark from vtmstudio.dev/download.
2. Double-click install.bat.
3. Double-click run.exe, add your character picture, start tracking, and pick the VTM Spark camera in OBS / Discord / Zoom.

You'll need Windows 10/11 and an NVIDIA RTX 30, 40 or 50 series GPU.

Your character picture

The model expects a still laid out like the one above:

• square, 768 × 768, solid green #00FF00 background
• chest-up, centred, facing the camera, chin level
• both eyes open, small closed-mouth smile
• no hands, props, text or extra people

VTM Spark ships this blueprint and a ready-to-paste image-AI prompt in character-blueprint/.

What's in this repo

VTM Spark places them under models/dit/, models/decoder/, models/trackers/ and vendor/tools/openseeface/models/. On first setup it also fetches a few models straight from their publishers (SD VAE, a tiny VAE and two anime-face detectors), about 1.24 GB in all.

Model

VTM-1.5.1.pt

• Architecture: DiT, 30.3M parameters (hidden 320, depth 10, 5 heads, patch 4 on SD-VAE latents), with RoPE on keypoints, QK-norm, SwiGLU and RMSNorm
• Output: 768 × 768 through the SD VAE, or through the Ultra fast decoder while streaming
• Pose input: 37 keypoints covering the face outline, brows, eyes, irises, nose, mouth and upper body
• Identity input: one reference image of the character, as image tokens plus face tokens
• Sampling: rectified flow, distilled to run in 1 step from a 30-step teacher (VTM-1.5.1-SFT, a fine-tune of the 1.5.1 base). Guidance is baked into the weights, so run it at 1 step, pose CFG 1.0, identity CFG 1.0 (VTM Spark's defaults)
• Speed: built for live use in VTM Spark. On an RTX 5060 Ti the model plus Ultra decoder draws about 117 frames a second on the GPU; the desk shows about 60–65, limited by the app's CPU-side work. Frame rate depends on the GPU and the app's Batch, Inbetweens and Max FPS settings.

The previous VTM-1.5.1.pt (distilled from a 10-step teacher, 1 or 2 steps, CFG not baked in) is still in this repo's commit history.

Ultra fast decoder

decoder/vtm-fast-decoder.pt is a small PixelShuffle decoder distilled from the Hybrid TinyVAE. It decodes the same SD latents in one pass (a whole live frame takes about 11 ms instead of 17 on an RTX 5060 Ti) and is what VTM Spark's "Ultra fast" stream mode runs. Without it the app falls back to the TinyVAE.

Data

Training used a small private character set:

• ~1,500 characters
• ~16–30 images per character

Generation quality in this release is limited mainly by model capacity (DiT-30M) and dataset scale.

Download

hf download sinBoo1/VTM-Spark VTM-1.5.1.pt --local-dir ./VTM-Spark

from huggingface_hub import hf_hub_download

ckpt = hf_hub_download(
    repo_id="sinBoo1/VTM-Spark",
    filename="VTM-1.5.1.pt",
)

Everything at once: hf download sinBoo1/VTM-Spark --local-dir ./VTM-Spark

The runtime (live camera → keypoints → this model → virtual camera) is VTM Spark.

Limitations

• Beta: soft detail, identity drift and pose errors happen
• Small data and DiT-30M capacity limit image quality
• Windows + NVIDIA CUDA only (no AMD, macOS, Linux or CPU)
• Framing: torso-up only (roughly head to mid-torso). Legs and most of the waist are not supported
• Hands: not supported
• Character types not supported: realistic humans; non-humanoid / furries
• Accessories: glasses and hats generally work; most other accessories are not supported

Intended use

Live VTubing and research on pose → image pipelines (live drive, pose retargeting). Not a finished production renderer.

Licences

No single licence covers every file here, so this repo is tagged license: other. Apache License 2.0 covers our training work: VTM-1.5.1.pt and our tracker fine-tunes. It does not re-license anyone else's weights, and some files here start from third-party weights with their own terms:

Full inventory and sources: THIRD_PARTY_NOTICES.md in the VTM Spark repo.