Refact-1.6B-base
Finally, the model we started training with our blog post is ready 🎉 The model might contain some problems, especially with the FIM format
It Works As a Chat
The primary application of this model is code completion (infill) in multiple programming languages. But it works as a chat quite well.
Example
Fill-in-the-middle uses special tokens to identify the prefix/middle/suffix part of the input and output:
# pip install -q transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "smallcloudai/Refact-1_6B-fim"
device = "cuda" # for GPU usage or "cpu" for CPU usage
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, trust_remote_code=True).to(device)
prompt = '<fim_prefix>def print_hello_world():\n """<fim_suffix>\n print("Hello world!")<fim_middle>'
inputs = tokenizer.encode(prompt, return_tensors="pt").to(device)
outputs = model.generate(inputs, max_length=100, temperature=0.2)
print("-"*80)
print(tokenizer.decode(outputs[0]))
Chat Format
The same model works as chat (experimental).
prompt_template = "<empty_output>SYSTEM {system}\n" \
"<empty_output>USER {query}\n" \
"<empty_output>ASSISTANT"
prompt = prompt_template.format(system="You are a programming assistant",
query="How do I sort a list in Python?")
Architecture
As described in more detail in the blog post, we used:
- ALiBi based attention
- LayerNorm instead of RMSNorm
- Multi Query Attention
We also used LiON, flash attention, early dropout. It's not that innovative that you can't run it, in fact you can -- see an example below.
Training
For the base model, we used our own dataset that contains code with permissive licenses only, and open text datasets. Filtering is the key to success of this model:
- We only used text in English
- Only topics related to computer science
- Applied heavy deduplication
The text to code proportion was 50:50, model trained for 1.2T tokens.
We don't release the base model, because its Fill-in-the-Middle (FIM) capability likes to repeat itself too much, so its practical use is limited. But if you still want it, write us a message on Discord.
Limitations and Bias
The Refact-1.6B model was trained on text in English. But it has seen a lot more languages in code comments. Its performance on non-English languages is lower, for sure.
Model Stats
- Architecture: LLAMA-like model with multi-query attention
- Objectives Fill-in-the-Middle, Chat
- Tokens context: 4096
- Pretraining tokens: 1.2T
- Finetuning tokens: 40B
- Precision: bfloat16
- GPUs 64 NVidia A5000
- Training time 28 days
License
The model is licensed under the BigScience OpenRAIL-M v1 license agreement
Citation
If you are using this model, please give a link to this page.