DeepSeek-R1-ReDistill-Llama3-8B-v1.1

This is a version of the DeepSeek-R1-Distill-Llama3-8B model re-distilled for better performance.

Performance

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
compute_dtype = torch.bfloat16
device   = 'cuda'
model_id = "mobiuslabsgmbh/DeepSeek-R1-ReDistill-Llama3-8B-v1.1"

model     = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=compute_dtype, attn_implementation="sdpa", device_map=device)
tokenizer = AutoTokenizer.from_pretrained(model_id)

prompt  = "What is 1.5+102.2?"
chat    = tokenizer.apply_chat_template([{"role":"user", "content":prompt}], tokenize=True, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(chat.to(device), max_new_tokens=1024, do_sample=True) 
print(tokenizer.decode(outputs[0]))

Output:

<｜begin▁of▁sentence｜><｜User｜>What is 1.5+102.2?<｜Assistant｜><think>
To solve 1.5 plus 102.2, I'll start by adding the two numbers together.

First, I'll add the whole numbers: 1 plus 102 equals 103.

Then, I'll add the decimal parts: 0.5 plus 0.2 equals 0.7.

Finally, I'll combine the results: 103 plus 0.7 equals 103.7.

Therefore, 1.5 plus 102.2 is 103.7.
</think>

To find the sum of \(1.5\) and \(102.2\), follow these steps:

1. **Align the decimal points:**

   \[
   \begin{array}{r}
     1.5 \\
   +102.2 \\
   \hline
   \end{array}
   \]

2. **Add the numbers:**

   - Add the whole numbers: \(1 + 102 = 103\)
   - Add the decimal parts: \(0.5 + 0.2 = 0.7\)
   - Combine the results: \(103 + 0.7 = 103.7\)

3. **Final Answer:**

   \[
   \boxed{103.7}
   \]<｜end▁of▁sentence｜>

HQQ

Run ~3.5x faster with HQQ. First, install the dependencies:

pip install hqq

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from hqq.models.hf.base import AutoHQQHFModel
from hqq.core.quantize import *

#Params
device        = 'cuda:0'
backend       = "torchao_int4" 
compute_dtype = torch.bfloat16 if backend=="torchao_int4" else torch.float16
model_id      = "mobiuslabsgmbh/DeepSeek-R1-ReDistill-Llama3-8B-v1.1"

#Load
tokenizer = AutoTokenizer.from_pretrained(model_id)
model     = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=compute_dtype, attn_implementation="sdpa")

#Quantize
quant_config = BaseQuantizeConfig(nbits=4, group_size=64, axis=1)
AutoHQQHFModel.quantize_model(model, quant_config=quant_config, compute_dtype=compute_dtype, device=device)

#Optimize
from hqq.utils.patching import prepare_for_inference
prepare_for_inference(model, backend=backend, verbose=False)

############################################################
#Generate (streaming)
from hqq.utils.generation_hf import HFGenerator
gen = HFGenerator(model, tokenizer, max_new_tokens=4096, do_sample=True, compile='partial').warmup()

prompt = "If A equals B, and C equals B - A, what would be the value of C?" 
out    = gen.generate(prompt, print_tokens=True)

############################################################
# #Generate (simple)
# from hqq.utils.generation_hf import patch_model_for_compiled_runtime
# patch_model_for_compiled_runtime(model, tokenizer, warmup=True)

# prompt = "If A equals B, and C equals B - A, what would be the value of C?" 
# chat    = tokenizer.apply_chat_template([{"role":"user", "content":prompt}], tokenize=True, add_generation_prompt=True, return_tensors="pt")
# outputs = model.generate(chat.to(device), max_new_tokens=8192, do_sample=True) 
# print(tokenizer.decode(outputs[0]))