sion_translate v1 Korean–Japanese Translation

KJ-X is a custom PyTorch encoder-decoder model for bidirectional Korean↔Japanese translation. The uploaded release contains inference exports and its SentencePiece tokenizer; training datasets and training checkpoints are not distributed.

v1 numeric inference update — runtime 0.1.2

Existing v1 users: reinstall the updated inference code. Existing verified v1 weights and tokenizer can be reused; no retraining or weight replacement is needed. The follow-up preserves spaced signs, repairs recognized measurement-unit suffixes (including compound units and powers), protects previously unrecognized numeric suffixes, and keeps ordinary uppercase acronyms visible during translation.

Actual v1 EMA execution now preserves ± 0.05, 250mL, 3,850km², 15mile, and 40mile in the previously failing cases. The five identified acronym regressions are also corrected. This remains a partial inference correction: arbitrary units, word-written numbers, arithmetic, and full numerical semantics are not guaranteed. Some numerical omissions still cause an error instead of a translation.

The focused follow-up attempted 238 inputs, returned 236 outputs, and blocked 2. It includes 72 authored numeric stress inputs, 16 existing diagnostics, and every affected FLORES input. Average chrF++ on the rerun no-digit subset increased in both directions; individual regressions are still possible. This was not a fresh full FLORES rerun. Related regression tests: 98 passed.

한국어 안내: 부호·단위·숫자 인식 누락·약어 회귀를 재학습 없이 보완했습니다. 기존 사용자는 추론 코드를 다시 받으세요. 기존 가중치는 재사용 가능합니다. 숫자 처리가 완벽하다는 의미는 아니며, 일부 값 누락은 오류로 차단됩니다.

Runtime commit: 45254b8cb84e.

See details and evidence.

Files

• model_ema.pt: recommended FP32 EMA inference export
• model_int8.pt: smaller CPU-only dynamic INT8 export
• kjx.model, kjx.vocab, token_features.npz: tokenizer files

The model uses custom code from gaon12/sion_translate and is not a Transformers AutoModel checkpoint.

Quick start

python -m pip install --upgrade "sion-translate @ git+https://github.com/gaon12/sion_translate.git@45254b8cb84e7725aa7b58709f060317ff4e2ab6"
python -m pip install huggingface_hub
hf download gaon12/sion_translate \
  model_ema.pt kjx.model kjx.vocab token_features.npz inference_v1.py v1_numeric_hotfix.json \
  --local-dir models/sion_translate

from sion_translate.inference import Translator

translator = Translator(
    "models/sion_translate/model_ema.pt",
    "models/sion_translate/kjx.model",
    device="cuda",  # Use "cpu" when CUDA is unavailable.
)
print(translator.translate(["안녕하세요."], target_language="ja", num_beams=4)[0])
print(translator.translate(["こんにちは。"], target_language="ko", num_beams=4)[0])

Translator automatically finds token_features.npz beside the model/tokenizer. The model is custom code, so transformers.AutoModel and hosted Transformers inference are not supported.

The numeric update was verified with model_ema.pt on CPU. Use that EMA file for the safe weights-only path; the legacy pickled INT8 download is not covered by this validation.

You can also run python models/sion_translate/inference_v1.py --text "17시 30분" --target-language ja. The helper verifies the downloaded EMA, tokenizer, and feature-file hashes.

Reproducible smoke test

After downloading the files above, run this from a clean environment to exercise both directions:

python - <<'PY'
from sion_translate.inference import Translator

translator = Translator("models/sion_translate/model_ema.pt", "models/sion_translate/kjx.model")
for text, target in [("안녕하세요.", "ja"), ("こんにちは。", "ko")]:
    output = translator.translate([text], target_language=target, num_beams=1)[0]
    assert isinstance(output, str) and output.strip(), output
    print(f"{text} -> {output}")
PY

Evaluation

The GitHub repository includes a small, independently authored diagnostic JSONL covering both directions and provides a common scorer for KJ-X, LibreTranslate, Papago, Google Translation, DeepL, M2M100 418M, and NLLB-200. No universal quality claim is made from that small set.

Limitations

• Machine translations can omit, add, or reverse meaning.
• Proper nouns, short context-dependent utterances, specialist terminology, and long inputs need human review.
• Do not use unreviewed output for medical, legal, safety-critical, or certified translation.
• The custom architecture requires the repository code; hosted Transformers inference is not available out of the box.

License

The original KJ-X code and uploaded model release are provided under the MIT License. This does not grant rights to user-provided inputs, generated service outputs, third-party software, or third-party models used only as comparison baselines.