BranchShine-CR

BranchShine-CR transcribes speech into International Phonetic Alphabet (IPA) characters. It is a teenny-tiny 25M-parameter E-Branchformer CTC model with intermediate self-conditioned CTC and two-view consistency regularization, trained from scratch on 17,038 hours of IPAPack++ speech (training parts 1-4, 142 language labels). You can find more details regarding the model, the dataset used here as well as its training regimen in the paper below.

This repository is to help you to fine-tune it with your own data :)

Note to keep in mind: This repository contains the model trained on the full-corpus and not the checkpoint evaluated in the paper’s main comparison, which was only trained on the ipapack_train_3, a subset of the full IPApack set.

Quick start

Python 3.10 or newer.

pip install "branchshine-cr @ git+https://huggingface.co/nikhilnavas/branchshine-cr"
branchshine-transcribe recording.wav --model nikhilnavas/branchshine-cr
branchshine-transcribe clips/ --model nikhilnavas/branchshine-cr --output ipa.jsonl --timestamps

I've tried installing with pip. Sometimes, it just doesn't work for the machines I've tried on. If the above commands don't work, try these:

# Usual creation and activation of isolated environment(Optional, otherwise you can just install to the global environment)
python3 -m venv .venv
source .venv/bin/activate

# Install the Hugging Face download command to download and install model weights
python -m pip install huggingface_hub
hf download nikhilnavas/branchshine-cr --local-dir branchshine-cr

# Install CPU PyTorch, then BranchShine-CR
python -m pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
python -m pip install ./branchshine-cr

# Inference!
branchshine-transcribe recording.wav --model ./branchshine-cr --device cpu

I've given a sample clip for counting from one to ten in the sample folder.

# Inference using sample clip
branchshine-transcribe "/home/nik/Downloads/sample.wav" --model ./branchshine-cr --device cpu

It should give you an output with the ipa transcription like:

wʌntuθɹifɔɹfaɪvsɪkssɛvəneɪtnaɪntɛn

Which is:         wʌn tu θɹi fɔɹ faɪv sɪks sɛvən eɪt naɪn tɛn
Which stands for: one two three four five six seven eight nine ten

The first run downloads config.json, model.safetensors and vocab.json into your Hugging Face cache. The output is one path IPA line per file, --output writes a JSON line per file with the symbols, their emission times and a confidence value. Other flags: --device cpu|cuda, --pad-seconds (default 1.0), --chunk-seconds (default 30), --batch-size.

In Python:

from branchshine_cr import load_model, transcribe

model, tokenizer = load_model("nikhilnavas/branchshine-cr")   # device="cpu" to force CPU
result = transcribe(model, tokenizer, "recording.wav")        # also arrays (sample_rate=) and lists
print(result.text)                    # 'aɪɛndʒɔɪɪntwɪθmaɪfneɪm'
print(result.symbols, result.times)   # per-symbol emission times in seconds

BranchShineCR.from_pretrained("nikhilnavas/branchshine-cr") also works, with CharTokenizer.from_file(hf_hub_download(repo, "vocab.json")) for the vocabulary. The raw forward pass model(waveforms, lengths) returns logits of shape (batch, frames, 117) at 25 frames per second, lengths, and the two intermediate auxiliary logits.

Preparing text and audio

The output is a continuous string of IPA characters: no spaces, punctuation, stress or tone marks, because the training transcriptions have none. Each token is one Unicode code point, so diacritics are separate tokens (ɛ̃ is two). The voiced velar stop is ASCII g, not IPA ɡ (U+0261). The 114 characters are:

a b c d e f g h i j k l m n o p q r s t u v w x y z
æ ç ð ø ħ ŋ œ ǀ ǁ ǃ ɐ ɑ ɒ ɓ ɔ ɕ ɖ ɗ ɘ ə ɛ ɜ ɞ ɟ ɠ ɢ ɣ ɤ ɥ ɦ ɧ ɨ ɪ ɬ ɭ ɮ ɯ ɰ ɱ ɲ ɳ ɴ ɵ ɶ ɸ ɹ ɻ ɽ ɾ
ʀ ʁ ʂ ʃ ʄ ʈ ʉ ʊ ʋ ʌ ʍ ʎ ʏ ʐ ʑ ʒ ʔ ʕ ʙ ʝ ʰ ʲ ʷ ʼ ː ˞ ˠ ˤ β θ χ ᶑ
combining marks: ̃ (U+0303 nasal)  ̚ (U+031A no release)  ̥ (U+0325 voiceless)  ̩ (U+0329 syllabic)
                 ̪ (U+032A dental)  ̴ (U+0334 velarized)  ̺ (U+033A apical)

The conventions are those of IPAPack++, whose transcripts were generated with grapheme-to-phoneme tools and normalized (Zhu et al., 2025). The model reproduces each source corpus's convention.

Silence margins: the tools add one second of digital silence before and after each clip by default (--pad-seconds at inference, --eval-pad-seconds during fine-tuning). On a sibling BranchShine-CR checkpoint, tightly cropped recordings caused severe under-transcription and the margins removed it. State the setting when you report numbers, and use the same setting for every system you compare.

Clips longer than 30 s are split into chunks and concatenated at inference. The base model was trained on utterances of 1 to 24 s, so you can prepare fine-tuning data in that range too.

Fine-tuning

Manifest

Write a JSON Lines manifest, one utterance per line:

{"audio": "clips/0001.wav", "text": "həloʊ", "language": "en"}

audio is any decodable file (relative paths resolve from the manifest's folder), text is the target, one token per character, spaces become <space>, language, duration and id are optional. Keep utterances between 1 and 25 s. Rows whose target needs more CTC frames than the audio provides (one per 40 ms, plus one per repeated adjacent character) are skipped with a message. Ideally, make a separate dev manifest from other speakers so the trainer can select the best checkpoint.

Vocabulary

--vocab pretrained keeps the 117 symbols and stops on unknown characters, extend appends new characters (existing head rows, biases and conditioning columns are kept), new builds the vocabulary from your text and copies weights for shared characters, for orthographic or phoneme-set targets. --normalize nfc|nfd normalizes the text first.

Run

branchshine-finetune --train train.jsonl --dev dev.jsonl --output-dir runs/my_task \
  --model nikhilnavas/branchshine-cr --vocab extend --epochs 20

The trainer uses the pretraining recipe: speed perturbation, two SpecAugment views, CTC on the final and intermediate heads, symmetric KL consistency between the views, AdamW with warmup and cosine decay. Defaults: --lr 1e-4, --warmup-steps 500, --epochs 10, --max-seconds 128 padded audio per micro-batch (3.1 GB of GPU memory with two views, 1.7 GB with --consistency-weight 0), --aux-weight 0.3, --consistency-weight 0.2, --eval-every 500. --freeze encoder trains only the head, the conditioning projection and the norms (about 60k parameters), --freeze blocks --freeze-blocks 6 also trains blocks 7-12. --resume runs/my_task/last.pt continues an interrupted run.

Outputs

Outputs in runs/my_task/: best/ (config.json, model.safetensors and vocab.json of the best dev-CER checkpoint, loadable with load_model and uploadable with hf upload), best.pt and last.pt (full training state), events.jsonl (loss, learning rate, dev CER per language, throughput, GPU memory) and per-evaluation predictions. Untuned starting points: the defaults for tens of hours of data, --freeze blocks --lr 5e-5 for under an hour, --vocab new --lr 2e-4 for a new alphabet. If the targets contain spaces the model learns <space> within a few hundred updates, --eval-ignore-spaces scores symbols only.

Evaluate and export

branchshine-evaluate --manifest dev.jsonl --model <repo or folder> --output-dir results/ reports micro CER overall and per language and writes predictions. branchshine-export --checkpoint last.pt --output dir converts a training checkpoint to the three-file layout.

Encoder features

For encoder features, model.encode(waveforms, lengths) returns the final hidden states (256-dimensional, 25 per second) with the intermediate conditioning applied and without the output projection. Training only the head with --freeze encoder is parameter-efficient adaptation, not a fixed-feature probe: the shared head and the conditioning projection act inside the encoder at blocks 6 and 10.

About the base model

The base checkpoint scores 5.28 % character error rate on the IPAPack++ development split. Its architecture and training are described in the BranchShine-CR paper cited below. branchshine_cr/model.py and losses.py are byte-identical to the code the model was trained with, so fine-tuning uses the same model and objective.

Files

config.json, model.safetensors and vocab.json are the base model. branchshine_cr/ is the package with the four command-line tools (branchshine-transcribe, branchshine-finetune, branchshine-evaluate, branchshine-export). The original training state and data pipeline are not included. The vocab.json sha256 is 427cbbd6..., matching the frozen training vocabulary.

Limitations

Conventions come from G2P-derived transcriptions, so tone, stress and allophonic detail are not represented. Tightly cropped clips may be under-transcribed without margins. Not validated for pronunciation or literacy assessment, or for children's, pathological or far-field speech. Timestamps and confidence are unvalidated. The checkpoint was selected on the development set.

Citation and license

Weights and code: CC BY-NC-SA 4.0. Training data: IPAPack++, CC BY 4.0, please credit it. Feel free to cite the BranchShine-CR paper for the method and this model card for the checkpoint.

Generative AI declaration

Generative AI was used to refine the quality of the above description as well as assistance with refractoring and debugging the codebase. The authors have tried our best and verified all related outputs manually. In the rare case of indiscrepancies, or even if you would just like to have a chat with us fellow humans, please feel free to contact us through our official channels. Have fun the with the model. Cheers!

@misc{navas2026branchshinecr,
  title         = {BranchShine-CR: Compact Multilingual IPA Transcription with Self-Conditioned CTC and Consistency Regularization},
  author        = {Navas, Nikhil and Chevtchenko, Sergio and Damiao, Talisson and Afshar, Saeed},
  year          = {2026},
  eprint        = {2609.29069},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2609.29069}
}

@misc{navas2026branchshinecr_fullcorpus,
  title  = {BranchShine-CR trained on the full IPAPack++ corpus (model card and weights)},
  author = {Navas, Nikhil and Chevtchenko, Sergio and Damiao, Talisson and Afshar, Saeed},
  year   = {2026},
  url    = {https://huggingface.co/nikhilnavas/branchshine-cr}
}

@inproceedings{zhu2025zipa,
  title     = {ZIPA: A family of efficient models for multilingual phone recognition},
  author    = {Zhu, Jian and Samir, Farhan and Chodroff, Eleanor and Mortensen, David R.},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL)},
  year      = {2025},
  url       = {https://arxiv.org/abs/2505.23170}
}

BranchShine-CR builds on E-Branchformer, rotary position embeddings and self-conditioned CTC. Developed at the International Centre for Neuromorphic Systems, Western Sydney University. Questions: open a discussion on this page.