Racka-4B Model Card

Racka (Regionális Adatokon Célzottan Kialakított Alapmodell) is a continually pretrained large language model designed to bridge the resource gap between Hungarian and high-resource languages. It employs parameter-efficient continual pretraining via Low-Rank Adaptation (LoRA) on a Qwen3-4B (reasoning/instruct) backbone.

The model was trained on a mixture of 160B tokens (44% Hungarian, 24% German, 21% English, 11% Code) on the Komondor HPC. To better match the training distribution, Racka uses an adapted tokenizer that achieves substantially improved tokenization fertility for Hungarian while maintaining competitive performance in English and German.

For additional details, please see our paper in the MSZNY2026 proceedings: https://rgai.inf.u-szeged.hu/sites/rgai.inf.u-szeged.hu/files/mszny2026.pdf

To learn more about the details of our project, please check our FAQ.

Model Details

• Developed by: ELTE Faculty of Humanities (Dept. of Digital Humanities) & ELTE Faculty of Informatics (Dept. of Artificial Intelligence)
• Backbone Model: Qwen/Qwen3-4B (Reasoning/Instruct version)
• Language(s): Hungarian (primary), English, German, Code
• License: cc-by-nc-sa-4.0
• Architecture: Transformer with LoRA adapters (Rank=64, Alpha=128)
• Training Context Length: 4,096 tokens (with sequence packing)
• Context Length (Inference): 32,768 natively and 131,072 tokens with YaRN
• Intended use case: Low-complexity edge tasks, optimized few-shot pipelines, task-specific Hungarian fine-tuning.

Usage

For edge/mobile inference, check out our quantized models in the GGUF format: Racka-4B-GGUF

Hugging Face Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig

model_name = "elte-nlp/Racka-4B"

# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

messages = [
    {"role": "system", "content": "You are a helpful Hungarian assistant."},
    {"role": "user", "content": "Magyarázd el a gépi tanulás lényegét óvodásoknak egy mondatban!"}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True # Switches between thinking and non-thinking modes. Default is True.
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)


generation_config = GenerationConfig(
    do_sample=True,
    temperature=0.6,
    top_p=0.8,
    top_k=50,
    repetition_penalty=1.1,
    presence_penalty=1.1,
)

# conduct text completion
generated_ids = model.generate(
    input_ids = model_inputs["input_ids"],
    attention_mask = model_inputs["attention_mask"],
    max_new_tokens=32768,
    generation_config=generation_config
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() 

# parsing thinking content
try:
    # rindex finding 151668 (</think>)
    index = len(output_ids) - output_ids[::-1].index(151668)
except ValueError:
    index = 0

thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip("\n")
content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")

print("thinking content:", thinking_content)
print("content:", content)

vLLM

vllm serve elte-nlp/Racka-4B --tokenizer elte-nlp/Racka-4B --dtype float16 --rope-scaling '{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}' --max-model-len 131072 --reasoning-parser qwen3

Technical Details

Training Data

The model was trained on a 160B token corpus designed to mitigate catastrophic forgetting via data replay:

Tokenizer Adaptation

The vocabulary was extended by 32,000 new Hungarian tokens initialized via VIPI (Vocabulary Initialization with Partial Inheritance). This reduced Hungarian subword fertility by ~47%. This fertility reduction results in proportional processing time reduction.

Training Configuration

• Infrastructure: Komondor HPC (64 \( \times \) NVIDIA A100 40GB).
• Training time: 287 hours (total GPU time: 2.1 years)
• Strategy: Distributed Data Parallel (DDP).
• Parameters:
  • LoRA Rank: 64, Alpha: 128, Dropout: 0.1.
  • Learning Rate: \( 1\times10^{-4} \) (LoRA), \( 5\times10^{-5} \) (Non-LoRA).
  • Batch Size: 2 per GPU (Effective batch size: 512).
  • Steps: 326,357.

Evaluation

The following tables present the performance of Racka-4B compared to its base models (Qwen3-4B and Qwen3-4B-Base) and the SOTA 8B Hungarian model PULI-LlumiX-Llama-3.1 8B.

1. HULU Benchmark (Fine-tuned)

Performance on the Hungarian Language Understanding (HULU) benchmark suite. Results represent the average of multiple runs, taking the best result between LoRA and full fine-tuning.

2. OpenHuEval

Evaluation on Hungarian reading comprehension, generation, and reasoning tasks. Qwen and Racka models use a patched implementation of OpenHuEval for compatibility.

3. LM-Eval-Harness (Hungarian)

Few-shot evaluation on standard benchmarks translated to Hungarian. Best results are kept (with chat template for Racka-4B and without for others).

Limitations

• The model is capable of both instruction following chat and English reasoning using the original Qwen settings, this is a preserved capability with no direct training targetting this functionality.
• The model has not been aligned and is unsafe for use with end-users.
• This model is only to be used for research purposes, commercial or for-profit usage is not permitted.

Team

In alphabetical order:

• Zsolt Csibi (ELTE-IK, AI Dept.)
• Bence Gortka (ELTE-BTK, DH-Lab)
• Natabara Gyöngyössy (ELTE-IK, AI Dept.)
• Kornél Nagy (ELTE-BTK, DH-Lab)
• Dávid Nemeskey (ELTE-BTK, DH-Lab)
• Gábor Palkó (ELTE-BTK, DH-Lab)
• Martin Sallai (ELTE-BTK, DH-Lab)
• András Simonyi (ELTE-IK, AI Dept.)
• András Szekeres (ELTE-BTK, DH-Lab)

Acknowledgements

We acknowledge the Digital Government Development and Project Management Ltd. for awarding us access to the Komondor HPC facility based in Hungary.

This research was supported by the EKÖP-24 University Excellence Scholarship Program of the Ministry for Culture and Innovation, funded by the National Research, Development and Innovation Fund.

The authors acknowledge the support of the National Laboratory for Digital Heritage. Project no. 2022-2.1.1-NL-2022-00009 has been implemented with the support provided by the Ministry of Culture and Innovation of Hungary from the National Research, Development and Innovation Fund, financed under the 2022-2.1.1-NL funding scheme.

We would like to thank Levente Szabados for the name idea and initial informal discussions.

Citation

@inproceedings{csibi2026racka,
  title     = {Racka: Efficient Hungarian LLM Adaptation on Academic Infrastructure},
  author    = {Csibi, Zsolt and Gortka, Bence Gy{\"o}rgy and Gy{\"o}ngy{\"o}ssy, Natabara and Nagy, Korn{\'e}l and Nemeskey, D{\'a}vid M{\'a}rk and Palk{\'o}, G{\'a}bor and Sallai, Martin and Simonyi, Andr{\'a}s and Szekeres, Andr{\'a}s M{\'a}rk},
  booktitle = {XXII. Magyar Sz{\'a}m{\'\i}t{\'o}g{\'e}pes Nyelv{\'e}szeti Konferencia (MSZNY 2026)},
  year      = {2026},
  address   = {Szeged, Hungary},
  pages     = {17--38},
  url       = {https://rgai.inf.u-szeged.hu/sites/rgai.inf.u-szeged.hu/files/mszny2026.pdf}
}

FAQ

Q: Is this model intended for use as a general-purpose chat agent?

A: No. While Racka-4B currently stands as one of the highest-scoring small/edge models on Hungarian benchmarks, it remains a compact, edge-oriented model. Conducting a full language adaptation for larger architectures would demand computing infrastructure beyond the Komondor HPC (AI training is just really expensive on the large scale).

Our team is actively working on adapting larger models; in the meantime, we recommend utilizing models of at least 27B–30B parameters for complex or resource-heavy reasoning tasks.

Q: Why is this a gated-access model with a restricted license?

A: Due to Hungarian and EU regulations, as well as the licensing terms of our source data, certain components of our training corpus cannot be used to generate derivative works for commercial or for-profit activities.

The volume of globally available, high-quality Hungarian text data remains limited. Restricting our training data exclusively to open, unrestricted corpora would not have yielded a sufficient dataset for effective language adaptation. Consequently, we require users to fill in an access agreement confirming the model will be used strictly for non-profit purposes. We believe the academic and non-profit communities will still derive value from this work.

Q: What kind of financial funding did this project receive?

A: As noted in our acknowledgements, our team consists of AI researchers and enthusiasts who volunteered their time. We did not receive dedicated financial funding or grants to build this model. Instead, the project was made possible through institutional and operational support:

• Worktime Allocations & Academic Grants: Several ELTE researchers and PhD students were permitted to dedicate official university hours and existing fellowship time to this initiative.
• Compute & HPC Access: Our academic project proposal was accepted by the Komondor HPC team (this is a public opportunity available to all Hungarian academics), granting us vital GPU time. The facility's engineering team also provided essential, specialized HPC support.
• Voluntary Contribution: Driven by a shared passion for open-source NLP, project members frequently dedicated their personal time to bring this model to completion.

Q: How would you improve on your model?

A: The current model has undergone continual pre-training during language adaptation, but it has not been through any SFT or alignment steps, which are required for highly capable models. Our aim is to translate, create and synthetize higher complexity datasets as part of our next projects. This will allow us to train models that are more capable in everyday use cases.