T3Q-LLM-MG-v1.0
Model Developers Chihoon Lee(chihoonlee10), T3Q
Python code
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
MODEL_DIR = "chihoonlee10/T3Q-LLM-MG-v1.0"
model = AutoModelForCausalLM.from_pretrained(MODEL_DIR, torch_dtype=torch.float16).to("cuda")
tokenizer = AutoTokenizer.from_pretrained(MODEL_DIR)
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
s = "한국의 수도는 어디?"
conversation = [{'role': 'user', 'content': s}]
inputs = tokenizer.apply_chat_template(
conversation,
tokenize=True,
add_generation_prompt=True,
return_tensors='pt').to("cuda")
_ = model.generate(inputs, streamer=streamer, max_new_tokens=1024)
hf (pretrained=chihoonlee10/T3Q-LLM-MG-v1.0), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
| Task | Version | Metric | Value | Stderr | |
|---|---|---|---|---|---|
| kobest_boolq | 0 | acc | 0.9523 | ± | 0.0057 |
| macro_f1 | 0.9523 | ± | 0.0057 | ||
| kobest_copa | 0 | acc | 0.7740 | ± | 0.0132 |
| macro_f1 | 0.7737 | ± | 0.0133 | ||
| kobest_hellaswag | 0 | acc | 0.4980 | ± | 0.0224 |
| acc_norm | 0.5920 | ± | 0.0220 | ||
| macro_f1 | 0.4950 | ± | 0.0223 | ||
| kobest_sentineg | 0 | acc | 0.7254 | ± | 0.0224 |
| macro_f1 | 0.7106 | ± | 0.0234 |
T3Q-LLM/T3Q-LLM-sft1.0-dpo1.0
| Task | Version | Metric | Value | Stderr | |
|---|---|---|---|---|---|
| kobest_boolq | 0 | acc | 0.9387 | ± | 0.0064 |
| macro_f1 | 0.9387 | ± | 0.0064 | ||
| kobest_copa | 0 | acc | 0.7590 | ± | 0.0135 |
| macro_f1 | 0.7585 | ± | 0.0135 | ||
| kobest_hellaswag | 0 | acc | 0.5080 | ± | 0.0224 |
| acc_norm | 0.5580 | ± | 0.0222 | ||
| macro_f1 | 0.5049 | ± | 0.0224 | ||
| kobest_sentineg | 0 | acc | 0.8489 | ± | 0.0180 |
| macro_f1 | 0.8483 | ± | 0.0180 |