romaji2ja
Romaji → Japanese conversion (ローマ字→日本語変換). A compact Qwen3-style causal LM plus a hybrid IME-style fast path that resolves most inputs from dictionaries/segmentation in well under a millisecond and only falls back to the neural model for genuinely novel input.
Code in the companion GitHub repository is MIT-licensed. Model weights and model artifacts are published under CC BY-SA 4.0 because the training corpus is derived from Japanese Wikipedia.
What this is
- Task: transduce romaji (incl. wapuro/kunrei/Hepburn, typos, IME small-tsu, case/space/hyphen noise) into Japanese (kanji/kana).
- Model:
Qwen3ForCausalLM, hidden 1024 · 20 layers · 16 heads / 4 KV (GQA) · intermediate 3584 · vocab 8000 · context 1024 · tied embeddings (~0.3B, fp32). Trained on rule-generated romaji↔Japanese pairs from a Japanese corpus. - Published checkpoint: accepted A75 interpolation checkpoint, SHA-256
97cd23ebb00582d461c04c0e8ed3ebd140395b298aa80f98f73cdfb5aa67b332. - Special tokens:
U+EE00romaji-input start ·U+EE01Japanese-output start ·U+EE02reserved context ·<|endoftext|>EOS/PAD. The model is prompted asU+EE00 {romaji} U+EE01and generates the Japanese continuation.
Hybrid fast path (recommended for product use)
infer_fast.py routes each input through, in order:
- input normalization (NFKC, lowercase, strip spaces / soft separators / zero-width)
- learned choice feedback → exact lexicon → SQLite cache
- exact lexicon segmentation
- generic romaji/kana phrase fallback
- whole-input fuzzy + fuzzy/weighted/dense segmentation routes
- general-phrase rescue — a generic noisy-romaji recovery stage:
bidirectional romaji style canonicalization (wapuro
syudan↔ Hepburnshudanboth match the dictionary), IME small-tsu (moxtute→もって) and run-collapse (syyyuu→syuu) canonicalization, then an anchor-and-fill beam Viterbi over a general reading lexicon + reusable colloquial layer with a confidence gate. It runs only when every earlier route declines and strictly before the neural model, abstaining (→ neural model) on low-confidence / ambiguous input rather than emitting a wrong answer. - neural model fallback
Verified accuracy (acceptance gate)
Practical Windows hybrid fast path — python src/check_acceptance_98.py,
12/12 required gates pass, evidence SHA-256 verified:
| Gate | Result |
|---|---|
| fixed gold | 99/99 (100%) |
| typo stress | 1980/1980 (100%) |
| realworld noise | 594/594 (100%) |
| expanded realworld case/space/hyphen | 1188/1188 (100%) |
| full-v1 expanded case/space/hyphen | 9432/9432 (100%) |
| large realworld case/space/hyphen (95 gate) | 5675/5675 (100%) |
| large full-v1 case/space/hyphen (95 gate) | 45598/45598 (100%) |
| long composition | 800/800 (100%) |
| fresh case/space/hyphen | 2358/2358 (100%) |
| strict neural full-v1 (normalized) | 771/786 (98.09%) |
| strict neural fresh case/space/hyphen (normalized) | 2313/2358 (98.09%) |
| post-shared-normalization fresh rerun | 2358/2358 (100%) |
Caveat (honest boundary): the 98% claim is for the practical hybrid path and
for normalized strict-neural input. Raw, un-normalized strict-neural
case/space/hyphen input is not a 95% claim (measured ~93.4%). Product input
should go through the hybrid path, which normalizes before routing. Full evidence
and per-file hashes are in acceptance/summary.json.
Usage
Neural model directly (normalized input recommended):
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("limoXD/romaji2ja")
model = AutoModelForCausalLM.from_pretrained("limoXD/romaji2ja")
bos_in, bos_out = chr(0xEE00), chr(0xEE01) # romaji-start, japanese-start
prompt = bos_in + "kyouhaiitenkidesu" + bos_out
ids = tok(prompt, return_tensors="pt", add_special_tokens=False)
ids.pop("token_type_ids", None) # Qwen-style model does not consume this field
out = model.generate(**ids, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))
# 今日はいい天気です
Hybrid fast path (clone the repo for code/ + lexicon/):
python code/infer_fast.py --model . --lexicon lexicon/romaji2ja_typo_95.json \
--general-lexicon lexicon/general_reading_lexicon.json "syudan no hani nara"
Repository layout
config.json, model.safetensors, tokenizer.json, ... # the model (load at repo root)
code/ infer_fast.py, general_phrase.py, romaji_kana.py, normalization.py, infer.py, eval_general_phrase.py
lexicon/ romaji2ja*.json, general_reading_lexicon.json, candidate_feedback.jsonl, adversarial_piece_aliases.json
acceptance/ summary.json, gate_report.json # 12/12 gate evidence with SHA-256 manifest
release_manifest.json # exact file hashes and checkpoint identity
Companion projects: source code · interactive Space.
Limitations
- Tuned for Windows-local use (CPU / DirectML); fp32 weights.
- Severe multi-error corruption or genuine homophone ambiguity is intentionally left to the neural fallback (or abstained) rather than guessed.
- Context window 1024 tokens.