NOESIS / AMAImedia
Released as part of the NOESIS Professional Multilingual Dubbing Automation Platform (framework: DHCF-FNO — Deterministic Hybrid Control Framework for Frozen Neural Operators).
- Founder: Ilia Bolotnikov
- Organization: AMAImedia.com
- X (Twitter): @AMAImediacom
- LinkedIn: Ilia Bolotnikov
- Telegram: @djbionicl
- NOESIS version: v16.1
- Release date: 2026-09-18
Kimi-K3-0.40B Kazakh CPT — Step 200
An experimental Kazakh continued-pretraining checkpoint based on inference-optimization/Kimi-K3-0.40B.
This is an early smoke-test checkpoint created to validate a complete Kazakh language-model pretraining pipeline on Kaggle.
Important limitation
The base checkpoint is a tiny architecture-development version of Kimi K3. Its weights were initialized from scratch and trained only on a toy copypasta dataset. It is not a distilled or compressed version of the trained 2.8T Kimi K3 model.
This checkpoint is therefore an educational research artifact, not a production language model.
Training
- Dataset:
kz-transformers/multidomain-kazakh-dataset - Objective: causal language modeling
- Optimizer steps: 200
- Tokens processed: 1,638,400
- Sequence length: 512
- Effective batch: 8,192 tokens
- Initial learning rate: 3e-4
- Hardware: 1× NVIDIA T4
- Runtime: approximately 12.7 minutes
- Tokenization: native Kimi tokenizer accelerated with Gigatoken
Results
| Metric | Before | After |
|---|---|---|
| Validation loss | 16.7615 | 4.9539 |
| Validation perplexity | 19,030,254 | 141.72 |
Example generations
The model has started learning Kazakh Cyrillic, morphology, punctuation, and sentence structure, but its generations remain semantically incoherent and contain invented words. This is expected after only 1.64M training tokens.
Usage
Install the required libraries:
pip install -U torch transformers
Run a minimal Kazakh text-generation example:
import torch
from transformers import AutoModel, AutoTokenizer
model_id = "AMAImedia/Kimi-K3-0.40B-Kazakh-CPT-step200"
tokenizer = AutoTokenizer.from_pretrained(
model_id,
trust_remote_code=True,
)
model = AutoModel.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.float16,
device_map="auto",
)
prompt = "Қазақстан —"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
output = model.language_model.generate(
**inputs,
max_new_tokens=100,
do_sample=True,
temperature=0.8,
top_p=0.95,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
For reproducible comparisons, use a fixed random seed and report the prompt, decoding parameters, Transformers version, and hardware. Because this is an intermediate checkpoint, outputs should be treated as experimental and inspected manually.
Intended use
This checkpoint is intended for studying Kazakh language adaptation, inspecting Kimi K3's small architecture-development configuration, testing dataset and pretraining pipelines, and comparing intermediate language-acquisition checkpoints.
It should not be used for factual, safety-critical, medical, legal, financial, or production tasks. The checkpoint has not been instruction-tuned or safety-aligned for general deployment.
License
This repository is released under the MIT License. Review the base model and dataset terms before redistribution or commercial use.