Bashkir Diacritics Restorer v1.2

A compact, local-first ONNX model that restores Bashkir-specific letters in partially undiacritized text while deliberately preserving foreign and technical tokens.

Overview

This is a character-level Bashkir diacritics restorer for the nine Bashkir letters: Ә Ғ Ҙ Ҡ Ң Ө Ҫ Ү Һ. It combines a 1.18M-parameter ConvBiGRU model with a conservative safety gate. The neural model is self-contained and runs with only model.onnx, vocab.json and substitution_map.json. The default runtime does not require a lexicon, KenLM, an internet connection, or a GPU.

Contents

Files and Configurations

The public package intentionally excludes the derived lexicon.json and the optional phrase-level KenLM model. This keeps the release self-contained and avoids redistributing a corpus-derived lexical resource. A research/full variant with optional components may be requested from the author through the Hugging Face profile.

Model Architecture

The model predicts a character distribution at each input position. It uses parallel 1D convolution features (kernel sizes 3 and 5) followed by a two-layer bidirectional GRU with hidden size 192. A substitution whitelist permits only valid base-letter to Bashkir-letter changes; it cannot invent arbitrary symbols.

Examples

Method

raw input text
  ├── 1. token safety gate (skip URLs, emails, Latin, Roman numerals & Russian stopwords)
  ├── 2. preserve already typed Bashkir-specific letters intact
  ├── 3. character-level neural sequence predictions via ConvBiGRU ONNX
  ├── 4. apply calibrated probability threshold & substitution_map constraints
  └── 5. restore Bashkir diacritics into surface text

Benchmark

The original experimental held-out evaluation reported sentence exact match of 93.42% and Bashkir-specific-letter F1 of 99.67%. A reproducible check of the public standalone release (without lexicon.json or KenLM) on the first 1,000 rows of the current local test file gives 83.6% restoration exact match and 97.6% untouched clean input. The experimental runtime, with its lexicon disabled on those same rows, gives 83.8%. An earlier 87.8% public-only figure could not be reproduced because its exact protocol was not recorded. The test has automatically generated targets, so individual disagreements can include noisy or historical source forms. See META.json for the file hash and machine-readable measurements.

Component Smoke Check

A historical 15-item behavioural check, ordered from simple word restoration through mixed-script input, casing, ambiguity and protected foreign text, was run with three local configurations. This is a transparent integration check, not a representative benchmark.

The context-free form кул is deliberately retained in all modes: it can mean more than one Bashkir word, and this release does not guess without a phrase-level language model.

On the rechecked sample the lexical component changes the overall result by 5.2 percentage points. It is most useful for known forms and mixed-script keyboard input; its impact on real-world usage should be measured separately with a human-checked test set.

Quality and Use

This release is suitable for local text cleanup, input-method suggestions and post-processing of Bashkir text. Its small ONNX model is practical on CPU-only systems. For a keyboard, call it only for the current token or short composing text, and present a candidate rather than silently replacing uncertain text.

Limitations

• This is a restoration model, not a general spellchecker or text generator.
• A text with no Bashkir lexical evidence may be intentionally left unchanged.
• Without the optional context language model, homographs can remain ambiguous.
• The model is trained for Bashkir Cyrillic; code-mixed, dialectal, novel and OCR-noisy forms need application-level review.
• Quality scores do not establish linguistic normativity for every output.

Related Resources

• Bashkir FastText Embeddings — compact subword embeddings for similar-form lookup and Bashkir lexical signals.
• Bashkir Word N-gram Index — phrase-level statistics for optional context-aware candidate ranking in keyboards.

Usage

Install the two runtime dependencies:

pip install -r requirements.txt

Run the included example from this release directory:

python example.py

Or use the runtime directly:

from pathlib import Path
from runtime import BashkirDiacriticsRestorer

restorer = BashkirDiacriticsRestorer(Path("path/to/bashkir-diacritics-restorer"))
print(restorer.restore("мин башкортса хойлэшэм"))
# мин башҡортса һөйләшәм

The default threshold is 0.40. Raise it to make neural substitutions more conservative, for example restorer.restore(text, min_confidence=0.60).

License

Distributed under the CC BY 4.0 license. The release contains model artifacts and derived substitution rules, not source texts. Upstream source licenses and attribution requirements still apply to the underlying materials.

Citation

@software{failed09_bashkir_diacritics_restorer_2026,
  title = {Bashkir Diacritics Restorer v1.2},
  author = {failed09},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/failed09/bashkir-diacritics-restorer},
  note = {Compact ONNX model for safe Bashkir diacritics restoration}
}

Open Bashkir Data and Sources 🐝

This release is part of an open-source effort to support the development, preservation and practical use of the Bashkir language. Other related models, datasets and tools are available on the author's Hugging Face profile.

The author does not claim ownership or authorship of the source texts or other materials used to derive this release; rights and licensing remain with the original authors, publishers and dataset providers. Source texts are not redistributed in this repository, so users should follow the licenses and attribution requirements of the relevant upstream resources.