teach-aws Qwen3.5-2B — Malay AWS QA
Answers AWS questions in Bahasa Malaysia at reference-document register.
91.0% strict accuracy on 2,661 unseen questions (completely-correct binary LLM judge; base model: 4.1%; previous release: 87.3%). Eval set and judge are fully reproducible — see the Colab notebook and the experiment walkthrough.
Recipe
unsloth LoRA on unsloth/Qwen3.5-2B (text-only): full MLP modules, all 24 layers, r=16,
α=32, lr 4e-4, 2 epochs, completion-only loss, trained on 28,148 rows =
khursanirevo/teach_aws (18,455 paraphrase-linked QA) + 8,433 teacher-paraphrased answers
- 1,260 hard-service variants targeting the hallucination-prone services (vpc, amplify, appsync, snowball, ecs, fargate). Two-seed validated: 91.2% / 91.0%.
What it does well
- Paraphrased AWS questions it has never seen, answered at doc-quality register (91% strict-correct)
- Short questions get short answers; correct Malay + code-switched AWS terminology
- Hard-service hallucination rate cut: vpc FALSE 16%→9%, amplify 11%→0% vs the previous release
What it does badly (read this)
- Invented features on thin-coverage topics. Example: it will confidently describe a non-existent CloudFormation "Memory and Cache Behavior" (correct answer: deletion policies). Most of its failures are fluent, confident text. Pair with the grounding guardrail (58% hallucination catch at 0.7% overblock on this model's full-set outputs) or retrieval.
- Rigid phrasing: most outputs reproduce the reference nearly verbatim. This is a deliberate accuracy-over-flexibility trade; the full frontier analysis is in the walkthrough.
- No knowledge outside its corpus (AWS-only, Malay/English mix as trained).
Usage
from vllm import LLM, SamplingParams
llm = LLM(model="khursanirevo/teach-aws-qwen3.5-2b", max_model_len=4096,
gpu_memory_utilization=0.85, max_num_seqs=256, enforce_eager=True)
tok = llm.get_tokenizer()
q = "Bolehkah S3 Transfer Acceleration melengkapi AWS Direct Connect?"
p = tok.apply_chat_template([{"role": "user", "content": q}],
tokenize=False, add_generation_prompt=True)
out = llm.generate([p], SamplingParams(temperature=0.0, max_tokens=2048))
print(out[0].outputs[0].text)
Comparison (same judge, same unseen eval)
| Model | Strict acc |
|---|---|
| this model (H4) | 91.0% (n=2,661) |
| previous release (H3) | 87.3% (n=2,661) |
| PixelSpaceAI/Malaysian-Qwen2.5-7B-AWS-Malay-LoRA | 15.0% (n=60) |
| mesolitica/Malaysian-Qwen2.5-7B-Instruct | 8.3% (n=60) |
The 7B community adapter is excellent for register/voice and recommends RAG for facts — this model is the opposite trade: fact recall first.
Acknowledgements
Trained with unsloth. Data from khursanirevo/teach_aws (built on PixelSpaceAI/aws-malay-qa sources). Judge: Qwen3-4B-Instruct-2507.