BashkirRoBERTa

A Bashkir masked language model for fill-mask suggestions and further fine-tuning.

Overview

BashkirRoBERTa predicts masked SentencePiece tokens from context. It can provide fill-mask suggestions and serve as an encoder for further fine-tuning, including the related person-name model. The release includes Transformers weights and FP32, FP16 and INT8 ONNX graphs. Its custom Pre-LayerNorm architecture requires the included model code when loaded with Transformers.

Contents

Files and Configurations

Model Architecture

The output embedding matrix is tied to the input word embeddings. Token IDs are fixed: <pad> 0, <unk> 1, <s> 2, </s> 3, [CLS] 4, [SEP] 5 and [MASK] 6.

Examples

Outputs from the INT8 ONNX model on CPU:

Method

The model was pretrained with dynamic masked-language modelling on a monolingual Bashkir-language dataset assembled from encyclopedic, periodical and literary sources. The source texts are not distributed in this repository.

raw Bashkir text (monolingual corpus)
  ├── 1. SentencePiece unigram tokenization (vocab size: 32,000)
  ├── 2. dynamic masked-language pretraining (15% subword masking rate)
  ├── 3. export PyTorch Transformers checkpoint
  └── 4. export optimized ONNX graphs: FP32, FP16 & dynamic INT8

Benchmark

On a held-out Bashkir encyclopedic evaluation set the project reports 24.7% top-1 and 54.0% top-5 accuracy for masked subword prediction. These are diagnostic MLM results, not a general-purpose language-understanding score: a mask may represent a whole word or a SentencePiece subword fragment. The FP32 ONNX graph was checked against the source PyTorch model on multiple sequence lengths and batch sizes, with matching top-token predictions.

Quality and Use

Use the Transformers checkpoint for fill-mask experiments or fine-tuning and the INT8 ONNX graph for compact CPU inference. Choose FP32 ONNX when a full-precision ONNX reference is needed. Predictions are token suggestions that need review in context, especially when the mask represents only part of a word.

Limitations

• Diagnostic MLM accuracy only; not fine-tuned for any downstream task.
• A mask may correspond to a partial subword, not always a full word.
• Predictions reflect the training corpus and may prefer frequent or encyclopedic phrasing.
• No training texts are redistributed; provenance or removal requests go through the maintainer.

Related Resources

• BashkirRoBERTa NER fine-tunes this encoder to find person names and return their text spans. Use it when the task is person-name extraction rather than masked-token prediction.

Usage

pip install transformers torch huggingface_hub

PyTorch (Transformers), which requires trust_remote_code=True because of the custom Pre-LayerNorm architecture:

from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "failed09/bashkir-roberta"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(repo_id, trust_remote_code=True)

inputs = tokenizer("Мин башҡорт телен [MASK].", return_tensors="pt")
logits = model(**inputs).logits
mask_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
prediction_id = logits[0, mask_index].argmax().item()
print(tokenizer.decode([prediction_id]))  # яратам

ONNX Runtime for CPU and edge deployment:

import numpy as np
import onnxruntime as ort
import sentencepiece as spm
from huggingface_hub import hf_hub_download

model_path = hf_hub_download("failed09/bashkir-roberta", "onnx/model_int8.onnx")
sp_path = hf_hub_download("failed09/bashkir-roberta", "spm_bashkir_bert_16k.model")

session = ort.InferenceSession(model_path, providers=["CPUExecutionProvider"])
sp = spm.SentencePieceProcessor(model_file=sp_path)

tokens = [2] + sp.encode("Мин башҡорт телен ") + [6] + sp.encode(".") + [3]
mask_idx = tokens.index(6)
logits = session.run(None, {"input_ids": np.array([tokens], dtype=np.int64)})[0][0, mask_idx]
top_tokens = np.argsort(logits)[::-1][:5]
print([sp.decode([int(t)]) for t in top_tokens])  # ['яратам', 'беләм', 'өйрәнә', ...]

For full-precision ONNX inference, change the downloaded filename to onnx/model_fp32.onnx; the input and output names are the same.

License

The model weights, tokenizer and release code are distributed under the Apache-2.0 license. Training texts are not redistributed; their rights remain with their respective owners. For provenance or removal requests, contact the maintainer through the Hub.

Citation

@software{failed09_bashkir_roberta_2026,
  title = {BashkirRoBERTa},
  author = {failed09},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/failed09/bashkir-roberta},
  note = {Masked language model for Bashkir}
}

Open Bashkir Data and Sources 🐝

This release is part of an open-source effort to support the development, preservation and practical use of the Bashkir language. Other related models, datasets and tools are available on the author's Hugging Face profile.

The author does not claim ownership or authorship of the source texts or other materials used to derive this release; rights and licensing remain with the original authors, publishers and dataset providers. Source texts are not redistributed in this repository, so users should follow the licenses and attribution requirements of the relevant upstream resources.