alemllm

Description:

AlemLLM is a large language model customized by Astana Hub to improve the helpfulness of LLM generated responses in the Kazakh language.

Evaluation Metrics

Model evaluations were conducted using established benchmarks, employing a systematic process to test performance across various cognitive and technical tasks.

Kazakh Leaderboard

Russian Leaderboard

English Leaderboard

Model specification

Architecture: Mixture of Experts Total Parameters: 247B Activated Parameters: 22B Tokenizer: SentencePiece Quantization: BF16 Vocabulary Size: 100352 Number of Layers: 56 Activation Function: SwiGLU Positional Encoding Method: ROPE Optimizer: AdamW

Run in Docker mode

• Ubuntu 24.04
• NVIDIA-SMI 535.247.01
• Driver Version: 535.247.01
• CUDA Version: 12.2

docker run -it --runtime nvidia -d \
  --restart=unless-stopped \
  --gpus all \
  -e OMP_NUM_THREADS=1 \
  -e NVIDIA_VISIBLE_DEVICES=all \
  -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \
  -p 8000:8000 \
  -v shm:/dev/shm \
  -v /alemllm/tmp/:/tmp \
  -v /alemllm/tmp/:/root/.cache \
  -v /alemllm/tmp/:/root/.local \
  -v /alemllm/weights:/alemllm/weights/ \
  astanahubcloud/alemllm:latest \
  python3 -m vllm.entrypoints.openai.api_server \
  --model=/alemllm/weights/ \
  --trust-remote-code \
  --tokenizer-mode=slow \
  --disable-log-requests \
  --max-seq-len-to-capture=131072 \
  --gpu-memory-utilization=0.98 \
  --tensor-parallel-size=8 \
  --port=8000 \
  --host=0.0.0.0 \
  --served-model-name  astanahub/alemllm

Run in Huggingface mode

• ubuntu22.04
• cuda 12.1
• python 3.11
• pytorch==2.1.0
• transformers==4.40.1

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "/path/to/alemllm"

model = AutoModelForCausalLM.from_pretrained(
  model_name,
  torch_dtype="auto",
  device_map="auto",
  rope_scaling=None,
  trust_remote_code=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)
prompt = "Give me a short introduction to large language model."

messages = [
  {"role": "user", "content": prompt}
]

text = tokenizer.apply_chat_template(
  messages,
  tokenize=False,
  add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
  **model_inputs,
  max_new_tokens=16384
)

generated_ids = [
  output_ids[len(input_ids):] for input_ids, output_ids in
  zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

Run in TuringInfer mode

• ubuntu22.04
• cuda 12.4
• pytorch==2.6.0
• transformers==4.51.0

python -m turing_serving.launcher \
  --model-path /path/to/alemllm \
  --model-name alemllm \
  --host 0.0.0.0 \
  --port 9528 \
  --solver server_solver \
  --backend vllm \
  --tensor-parallel-size 8 \
  --worker-timeout-seconds 7200 \
  --skip-authorizationcheck \
  --engine-args tokenizer-mode=slow disable-log-requests=__NULL__ trustremote-code=__NULL__ kv-cache-dtype=fp8 quantization=fp8 max-seq-len-tocapture=131072 gpu_memory_utilization=0.98

License

The model is licensed under Apache 2.0. The License prohibits unlawful, harmful, or abusive uses.

Attribution

Developed with technical support from 01.AI.

Intended Use & Limitations

Intended Use: Research and development in line with Kazakhstan's AI initiatives.

Limitations: The model may generate inaccurate, biased, or unsafe content; users must apply responsible use practices.

Safety & Compliance: Publication is subject to applicable laws, export control, and cybersecurity regulations.