MOSS-TTS Local Transformer v1.5

OpenMOSS's multilingual TTS model, exported for loom.cpp. Family 10: text in, neural-codec tokens out -- pair it with moss-audio-tokenizer-v2-loom for 48 kHz stereo audio.

This is a loom.cpp export: a single self-describing GGUF that carries its own graph topologies, tokenizer (if any) and driver script, produced by loom-exporter.

Original model

Exported from OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5. Weights are unmodified; this repo packages the same parameters into loom.cpp's GGUF format.

License

apache-2.0, inherited from the base model above.

Language(s)

zh, yue, en, ar, cs, da, de, nl, es, fr, fi, el, he, hi, hu, ja, it, ko, mk, ms, ru, fa, pl, pt, sv, ro, sw, tl, th, tr, vi

Usage

Run it with loom-py -- loom-py-rt on PyPI:

pip install -U "loom-py-rt[hub]"

import loom

model = loom.Model.from_pretrained("loom-ai-org/moss-tts-local-transformer-v1.5-loom")

# What comes back is codec TOKENS, not audio -- frame-major, one row per frame, 12 codebooks wide, at
# 12.5 frames per second. `language=` is one of the codes in model.contract["languages"].
codes = model.text2codes.infer("The quick brown fox jumps over the lazy dog.", language="en", seed=1)
print(len(codes), "frames x", len(codes[0]), "codebooks")

# The second half of the pair, in a repo of its own. The codec has 32 codebooks and decodes the first
# 12 as a prefix: it declares which id means "absent", and codes2speech fills each row out with it.
codec = loom.Model.from_pretrained("loom-ai-org/moss-audio-tokenizer-v2-loom")
audio = codec.codes2speech.infer(codes)
print(audio.channels, "channels at", audio.sample_rate, "Hz =", round(audio.duration, 2), "s")
audio.save("out.wav")                         # 48 kHz stereo

# This model SAMPLES by default, at upstream's recommended settings; `seed=` pins a take. Greedy
# (both draws) reproduces upstream's generate(do_sample=False) exactly:
greedy = model.text2codes.infer("Hello there.", language="en", temperature=0, text_temperature=0,
                                max_new_tokens=20)

Cloning a voice

Upstream clones a voice from one or more reference clips by putting their codec tokens in the prompt. loom takes those tokens from a voice file: the clips already encoded, made once per voice with loom-exporter. Making one runs upstream's own processor and the MOSS-Audio-Tokenizer-v2 encoder in PyTorch, so it needs torch, torchaudio and transformers and both upstream checkpoints on disk -- this repo and the codec's carry decoders only. Using one needs nothing but loom.

git clone https://github.com/loom-ai-org/loom-exporter && cd loom-exporter
# Looks for the codec beside the TTS checkpoint (a directory named MOSS-Audio-Tokenizer-v2, any case),
# or pass --codec <dir>. --license is the RECORDING's, which only you know.
python -m loom_exporter.moss_tts_voices ~/models/MOSS-TTS-Local-Transformer-v1.5 -o voices \
    --wav me.wav --name me --license "CC0-1.0"

# A voice file is a path (or a name, for one under `voices/` beside the model file).
codes = model.text2codes.infer("The quick brown fox jumps over the lazy dog.", language="en",
                               voice="voices/me.gguf", seed=1)
codec.codes2speech.infer(codes).save("cloned.wav")

Several speakers are one voice file: repeat --wav, one clip per speaker in order, and write the text as [S1] ... [S2] .... A few clear seconds per clip is enough; a clip's frames count against the 2048 positions a prompt and its audio share, at 12.5 per second.

A voice file holds codec tokens, so it is stamped with a fingerprint of the codec's quantizer and fits any model that reads this codec's codes; loom refuses one made with a different codec. The voice's licence is its recording's, and so is the consent: clone only voices you have the right to use.

The layer underneath

The call above is the high-level door: one per task, named for the modality pair it maps between, with the windowing, sampling and assembly this model needs already applied. Under it, model.infer(...) passes your arguments straight to the driver this GGUF embeds -- which is where you go for a knob the door does not name.

model.driver_source prints that driver, including a header comment documenting every argument it accepts for this model, and is the authority on it. See loom-py for the API and loom.cpp for what the engine does between the two.

Known limitations

This model does not produce audio. It emits 12 streams of MOSS-Audio-Tokenizer codec tokens, and the codec turns those into a waveform -- moss-audio-tokenizer-v2-loom. They stay separate because the codec is shared across the MOSS family and because the codes are the useful intermediate.

12 codebooks into a 32-codebook codec, and nothing to do about it. The codec is a residual quantizer, and decoding the first 12 of its 32 codebooks is what upstream's own processor asks for. The codec file declares an id meaning "this codebook is absent", and codes2speech.infer fills each 12-wide row out with it, so the usage above passes the codes straight across.

language= is a code from the contract: the 31 languages this model supports, model.contract["languages"]. It selects the prompt's language line, which upstream recommends setting; omitted, the line says None, which upstream's README says can regress some languages slightly.

It samples by default, at upstream's recommended settings -- audio temperature 1.7, top-k 25, top-p 0.8 -- so two runs of a sentence give two takes, and seed= pins one. temperature=0, text_temperature=0 decodes greedily and reproduces upstream's generate(do_sample=False) exactly: every code of 38 and 40 frames identical, and the same frame chosen to stop on. The sampled decode is verified the same way, with its random draws pinned on both sides.

Cloning takes a voice file, not a clip. Upstream clones by putting a reference clip's codec tokens in the prompt, which needs the codec's ENCODER; neither this repo nor the codec's carries it, so the clip is encoded once, in Python, into a voice file (see above), and the prompt it builds is upstream's exactly -- verified code for code with one reference and with two. Without a voice the model picks one, and the seed decides which. Upstream's continuation mode (a prompt clip continued as the model's own speech) and the template's other optional lines (a duration in tokens, a free-form instruction, quality, sound events) are not exposed.

max_new_tokens counts audio frames at 12.5 per second. One frame is one pass of the 36-layer backbone plus twelve of a one-layer local transformer that draws the codebooks in turn; the model decides for itself when to stop.

It is a big download, and slow on a CPU: 16.8 GB, F32, like the rest of this collection (4.55B parameters, less the 1.56 GB text head this release never reads). A 24-core workstation generates about 3 frames per second, against the 12.5 real time needs.

Files

• moss-tts-local-transformer-v1.5.gguf -- the model, exported with loom-exporter.