Drama-3 Config I GGUF — built for TheTom’s fork (experimental)

[!IMPORTANT] COMPATIBILITY NOTE: DEVELOPED AND TESTED WITH THETOM'S TURBOQUANT LLAMA.CPP FORK.

Anyone is welcome to run or adapt this work under the applicable licenses. The tested speech route requires this fork's custom llama-drama3-backbone bridge plus the pinned external audio tower and codec. Other engines have not been validated and may not work; a .gguf file does not imply plug-and-play compatibility. See PR #399 and the setup instructions. An initial RTX 5090 short-speech test now passes; see its measured limits below.

Built with Fish Audio.

A layer-aware TurboQuant derivative of fishaudio/drama-3, revision dd64a85cd82e9521f7202610ede97b367fc83946.

This release uses the TurboQuant llama.cpp fork's experimental Drama-3 bridge: llama.cpp executes the language backbone, while the official SGLang-Omni audio tower and local Drama-3 codec synthesize speech. This is not a standalone llama-cli TTS model and is not an official Fish Audio release.

Available variants

C24B keeps the first and last two routed-expert layers at the same BF16 precision as C0, while using smaller interior expert weights. It shares the existing audio runtime; its backbone plus runtime assets total approximately 26.65 GiB (28.61 GB) on disk, before dependencies and caches. Use C24B's own runtime manifest and instructions, rather than the C0 manifest.

C24B requires CPU offload in the tested configuration: 31 backbone blocks plus the output head run on GPU; nine backbone blocks run on CPU. A constrained RTX 5090 test sampled a total-device peak of 30,147 MiB including an 8,003.4375 MiB live reservation. Subtracting only that known reservation gives 22,143.5625 MiB (21.62 GiB), leaving 2,432.4375 MiB (2.38 GiB) within a 24 GiB budget. Sampling was every 100 ms and may miss shorter spikes. A physical 24 GB card has not been tested. CPU offload also needs host RAM and adds latency.

Four of five requests completed with EOS: cold/warm hello, neutral speech and fixed-reference English. Independent CPU ASR recovered all requested words in those four clips. The emphasis request failed at 1,024 frames without EOS; its failure and codes are retained. These are limited content and memory diagnostics, not proof of cue adherence, naturalness, speaker identity, long-form reliability or source parity. The reference is synthetic speech generated by the original model, not a real-person cloning sample.

The warm hello request took 8.71 seconds for 1.49 seconds of audio; the cold request took 173.17 seconds including verification/loading. The 16-word neutral request took 24.95 seconds. This is not a real-time performance claim. C24B samples, failed attempt, settings and memory evidence accompany the variant. C0 weights and shared audio assets remain unchanged.

Why keep higher precision?

Smaller quantizations are possible by lowering precision further or relaxing the protected tensor set. My goal here is to refine Config I's quality-versus-memory tradeoff, rather than chase the smallest file.

Config I allocates precision by tensor role and layer position: compress the large interior routed-expert banks, choose separate formats for gate/up projections and down projections, and protect the first and last two layers. This Drama-3 adaptation also preserves attention, routers, shared experts and the audio runtime. C0 uses TQ4_1S gate/up and Q4_K down; C24B moves the interior experts to TQ3_1S and Q3_K while keeping the protected routed-expert boundaries at BF16.

These protections are a quality-preservation strategy, not a demonstrated quality guarantee. I’m continuing to refine the policy through matched speech comparisons and checks for content, delivery, voice consistency, repetition and completion. Broader speech quality, long-form reliability and parity with the original remain unestablished.

C0 quantization policy

The policy is adapted from Tom Turney’s TurboQuant Plus paper, Tensor-Role-Aware Weight Compression for llama.cpp: TQ4_1S and the Config I Policy. Drama-3 C0 uses its tensor-role and boundary-protection approach while retaining attention and boundary layers at source precision; protected block-FP8 weights expand to BF16 for the GGUF runtime.

• Interior text layers 2 through 37: routed expert gate/up uses TQ4_1S; routed expert down uses Q4_K.
• First and last two text layers retain source precision, expanded from original block-FP8 to BF16 for this GGUF runtime. They are not requantized to Q8.
• Linear attention, full attention, routers, shared experts and audio weights are not additionally quantized in C0.
• TQ4_1S stores 20 bytes per 32 values (5 effective bits/weight, including scales). Q4_K stores 144 bytes per 256 values (4.5 effective bits/weight).

The backbone GGUF is 28,942,612,608 bytes (26.96 GiB). This excludes the separate audio tower, codec, caches, activations and runtime scratch. File size is not a complete VRAM measurement.

Current evidence and limitations

Long-form reliability is unresolved. In the newer six-case cue/long batch, C0 completed four attempts with EOS; neutral speech hit the 1,024-frame cap and the 185-word narration hit the 4,096-frame cap without EOS. The point-tag output completed but is unusually long and needs repetition/content review. These failures remain visible alongside the successful diagnostics.

A complete C0 diagnostic generated speech on an NVIDIA GB10 DGX Spark with explicit EOS. The first clip was listened to and positively assessed by its creator. Eleven of twelve seed-42 diagnostic prompts produced EOS-complete clips. The tongue-twister failed at seed 42 and subsequently completed at seed 43 (271 frames, 12.585 seconds of audio). This retry does not replace the seed-42 failure or establish a 100% same-seed completion rate. This is an early subjective result, not a systematic quality benchmark.

The first 27-row comparison against the original SGLang reference showed substantial final-hidden drift (relative L2 0.61054, cosine 0.79418), semantic-logit relative L2 0.16747, top semantic prediction agreement, and 4/10 top-ten overlap. Conversion/runtime differences and quantization effects are still being separated using an original-precision native control. Reference parity and broad speech-quality validation are not established.

RTX 5090 short-speech smoke passed under WSL2, using the target-built SM120 binary, C0 backbone and compact runtime. Two identical seed-42 requests both completed with EOS (47 frames, 2.1827 seconds of audio). This establishes a short-request fit, not long-context capacity or a quality pass. Sampled total-device peak was 31,787 / 32,607 MiB at 100 ms intervals, leaving only 820 MiB at that sampled peak; shorter spikes may be missed.

Cold wall includes asset verification, model loading, first-use setup and codec work. First-code time is not audible streaming latency. The test used context 16,384, max_frames 128 and native microbatch 8; sequence state was reset for the warm request. WAVs, settings, timings, transfer hashes and sampled memory evidence are preserved. Broader prompts, longer speech, stability and quality on this GPU remain pending.

Early prefill observations (DGX Spark)

The first request in a fresh process is much slower than requests that reuse loaded models. Across four fresh-process diagnostics, the combined prefill timer measured 24.60–26.09 seconds. Eight warm diagnostic requests measured 0.48–1.10 seconds, with a 0.54-second median. These prompts differ in length, so the range is not a matched-prompt speedup or a tokens-per-second benchmark.

A matched 55-row neutral-prompt profile now separates the stages (same text, seed 42, sequence state reset, warm weights reused):

The roughly 25-second cold delay is predominantly inside external audio-decoder prefill. Its underlying first-use mechanism has not been isolated; this is not evidence of a specific compilation or kernel-loading cause. Native synchronized timing includes host work and is not kernel-only time. Both profiling attempts subsequently hit the 1,024-frame cap without EOS; their prefill measurements remain useful, but neither is a completed speech sample or a quality pass. See profiling receipts and preserved failures. No code speedup is claimed from these diagnostic measurements.

Cold-start diagnostic (DGX Spark)

The following is first-request diagnostic evidence, not steady-state prefill performance.

Prompt: "Hey! Glad you made it. Grab a seat, and tell me how your day went."

Generation-only RTF was 1.046 (lower is better; 1.0 is real time). The very slow cold startup included NAS-backed weight loading. Filesystem cache was uncontrolled, so these are diagnostic observations, not a controlled speed benchmark. Seed 42, temperature 1.0, top-k 30, top-p 0.9; context 16,384; native microbatch 8. Time to an audio-code frame is not audible streaming latency.

A later warm request (fresh causal audio cache and reset native memory, reused weights) produced 8.824 seconds of audio in 8.979 seconds of generation and 10.832 seconds of total request time. Warm prefill was 0.529 seconds and the first audio-code frame arrived 1.322 seconds after request entry. This remains a diagnostic observation rather than a controlled benchmark.

One tongue-twister prompt reached the 1,024-frame limit without EOS in the initial seed-42 diagnostic batch; the separate seed-43 retry completed. That attempt is retained as failed evidence and is not presented as a completed clip. Long-form stability and difficult-prompt behavior still need work.

Listening examples and ongoing validation

EOS-complete diagnostic WAVs, generated frame codes and execution receipts are in samples/quantized and samples/original. The original set currently contains eleven canonical seed-42 completions plus a tongue-twister seed-43 comparison; the numbers prompt hit the length cap at seeds 42 and 43, then completed at seed 44 (13.978 seconds). The seed-44 clip is a separate attempt, not a replacement for failed evidence. Preserve alternate seeds as separate attempts. Original and C0 use different backends, so these are paired listening evidence, not an isolated quantization-only speed comparison.

All six original-model long/cue diagnostics now completed with EOS: neutral, whispering, excited, calm, sigh/laugh tags and a 185-word narration. The narration produced 67.384 seconds of audio in 70.052 seconds of request wall time (RTF 1.040), excluding the 589.285-second pipeline startup. These are source-backend observations with uncontrolled caches. See original long/cue samples. Cue adherence and complete transcript coverage still need assessment; EOS alone does not establish either. Four corresponding C0 cue attempts completed with EOS: whispering, excited, calm and point tags; see C0 long/cue clips and failures. The seed-42 neutral attempt hit the frame cap; a separately labeled seed-43 retry completed with EOS (10.217 seconds), preserving the seed-42 failure. The matched original seed-43 retry also completed with EOS and is included separately. The point-tag clip is unusually long (46.997 seconds versus 7.941 seconds on the source backend) and needs review for repetition and event behavior. EOS alone is not a cue-quality pass. The unconditioned C0 185-word narration hit 4,096 frames without EOS, while the original completed in 67.384 seconds. Its failed frame codes and profiling receipts are retained; no completed WAV exists for that attempt. Long-form reliability is not established. Fixed-reference, scoped-control, multilingual and duration diagnostics have now run; their EOS results are below, while adherence and transcript coverage remain unassessed. The RTX 5090 short-speech smoke passed with the memory and timing limits reported above. The initial matched cold/warm prefill profile is reported above; the underlying audio-decoder first-use mechanism is still being investigated. Negative results and configuration changes will remain visible alongside successful samples.

Nineteen of twenty C0 diversity cases have now produced EOS-complete diagnostic WAVs; see C0 diversity clips. They include scoped whisper/slow/emphasis, style and director notes, events, six language cases and code-switching. The language cases use one fixed synthetic reference generated by the original model; pronunciation, speaker continuity, content and cue adherence remain unassessed. Reference-conditioned long narration also reached the 4,096-frame cap without EOS; its failure and all frame codes are preserved. All 20 original-source diversity attempts completed with EOS, including the reference-conditioned long prompt (28.9785 seconds of audio, 31.1896 seconds request wall). EOS does not establish complete spoken coverage; the much shorter output needs transcript/listening review. Nineteen C0 cases have completed outputs; C0 reference-long failed without EOS. These results do not erase the separate unconditioned long-form failure.

For one duration-hint prompt, a 65-frame hint produced 63 generated frames (2.926 seconds), and a 130-frame hint produced 128 (5.944 seconds). These are two observed responses, not a guarantee that duration hints always match their targets.

Running

The fork implementation and copy-paste clone, environment, CUDA build, download and run instructions are in PR #399 and examples/drama3/README.md. The compact runtime contains all 192 unchanged audio tensors, codec and metadata (4.34 GB); the original backbone is not required. A fresh public-wheel ARM environment completed compact-manifest GPU synthesis with EOS (receipt). Its 85 frames and WAV were byte-identical to the prior canonical C0 full-source-assets execution (regression receipt). This validates the compact-loader regression, not parity with original precision or broad quality. The target-built RTX 5090 short-speech measurements are above. The C0 GGUF and all compact runtime assets are uploaded and remotely hash verified. This public release is experimental; the measured RTX 5090 short-speech fit does not establish compatibility with every 32 GB card or long-form reliability.

The first clean-environment GPU attempt exposed missing ninja during FlashInfer normalization JIT. The corrected public environment passed both full-source-assets and compact-manifest speech routes; PR #400 updates the recipe and remains unmerged. A fresh instrumented Spark compact run directly measured 16.033 seconds in the first FlashInfer normalization module build, 0.00049 seconds loading it, and 16.509 seconds in audio-decoder prefill. This identifies a first-use build cost for that run; it does not retroactively assign the exact older 24.703-second prefill to compilation. No warmup concealed the cost: total wrapper wall 124.942 seconds includes 50.776 seconds asset/GGUF verification and the JIT build. See structured first-use profile.

After the fork setup, activate its environment and install the tested build dependency before synthesis:

# After following the environment/build steps in the fork README:
source ../drama3-venv/bin/activate
python -m pip install ninja==1.13.0
ninja --version
export MAX_JOBS=2
hf download thetom-ai/Drama-3-ConfigI-GGUF --local-dir models/Drama-3-ConfigI
GGML_TQ_NATIVE=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
python examples/drama3/external_audio.py \
  --source models/Drama-3-ConfigI/runtime \
  --runtime-manifest models/Drama-3-ConfigI/runtime/runtime-manifest.json \
  --gguf models/Drama-3-ConfigI/Drama-3-ConfigI-C0.gguf \
  --binary build-drama3/bin/llama-drama3-backbone \
  --omni-source /absolute/path/to/sglang-omni --seed 42 \
  --text "Hello, how are you?" --out hello.wav

Use the Python environment created by the setup instructions. The CUDA 13 compiler target is 121a on the tested Spark and 120 for the tested RTX 5090 build. GGML_TQ_NATIVE=1 is required to keep the packed native TQ path. Successful output includes EOS and .complete.json receipts; do not treat a frame-limit failure as completed speech.

Prompting guide

Put delivery directions directly in --text. The pinned reference does not support a separate instructions API field. These formats are documented by Fish Audio; their reliability on this quantized checkpoint is still being tested. Directions are ordinary text, not a fixed enumeration. Keep them short, avoid nested brackets, and begin with one clear direction.

A duration hint is approximate: 65 codec frames are about three seconds, not a guarantee. The --max-frames CLI option is a hard generation cap, not a desired duration. The initial 1,024-frame cap is roughly 47.6 seconds; longer passages need an appropriate larger cap and enough context. A run that reaches its cap without EOS is failed evidence, not completed speech. Longer-run VRAM on the 5090 is pending.

Parentheses are spoken content; use the documented square/angle bracket formats for delivery control. A language hint goes inside the text; the separate language API field is unsupported in our pinned reference. Start without a reference, compare a neutral version against the same line with a cue, then reuse one encoded reference with its accurate transcript when testing voice continuity. Reference support uses --references and the fork's encode_reference.py; use exactly the same reference VQ codes for original/C0 comparisons. To reproduce this suite's synthetic reference, use --references references/original-01-hello.vq.json from this release; see reference provenance. This is a model-generated voice, not a real-person cloning benchmark.

Copyable examples are in prompt-examples.json. They are prompt recipes, not claims that every style or language has passed listening tests. See the official technical report, Appendix C/Table 10 and SGLang-Omni cookbook.

Bridge feature coverage

Plain single-speaker speech, inline/scoped directions and the single-speaker reference path are available for experiments. The current CLI keeps audio generation until EOS, then vocodes; first-code latency is not audible streaming latency.

Multi-character scenes, relative revisions with prior generated audio, local speech infilling, line-separated output and streaming text lookahead are model capabilities described in the report, but are not implemented and verified in this bridge. The pinned HTTP adapter rejects nonzero speaker IDs. Do not flatten a dialogue, supply a relative note without actual prior audio history, or regenerate an entire clip and call those scene support, revision or infilling.

License and attribution

See LICENSE and NOTICE. The original Fish Audio Community License applies to the weights and codec, including its use and redistribution conditions. Quantization does not change the upstream license. The llama.cpp fork and SGLang-Omni code have their own licenses in their respective repositories. No endorsement by Fish Audio is implied.