Table of Contents
Model Summary
SantaCoderPack is an pre-trained model with the same architecture of SantaCoder on <a href=https://huggingface.co/datasets/bigcode/commitpack>CommitPack using this format:
<commit_before>code_before<commit_msg>message<commit_after>code_after
- Repository: bigcode/octopack
- Paper: OctoPack: Instruction Tuning Code Large Language Models
- Languages: Python, JavaScript, Java, C++, Go, Rust
- SantaCoderPack:
Use
Intended use
The model follows instructions provided in the input. We recommend prefacing your input with "<commit_before>def has_close_elements(numbers: List[float], threshold: float) -> bool:\n for idx, elem in enumerate(numbers):\n for idx2, elem2 in enumerate(numbers):\n if idx != idx2:\n distance = elem - elem2\n if distance < threshold:\n return True\n\n return False<commit_message>Fix bugs in has_close_elements.<commit_after>"
Feel free to share your generations in the Community tab!
Generation
# pip install -q transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "bigcode/santacoderpack"
device = "cuda" # for GPU usage or "cpu" for CPU usage
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
inputs = tokenizer.encode("Q<commit_before>def has_close_elements(numbers: List[float], threshold: float) -> bool:\n for idx, elem in enumerate(numbers):\n for idx2, elem2 in enumerate(numbers):\n if idx != idx2:\n distance = elem - elem2\n if distance < threshold:\n return True\n\n return False<commit_message>Fix bugs in has_close_elements.<commit_after>", return_tensors="pt").to(device)
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))
Training
Model
- Architecture: GPT-2 model with multi-query attention
- Steps: 250k pretraining
- Pretraining tokens: 131B
- Precision: bfloat16
Hardware
- Pretraining:
- GPUs: 32 Tesla A100
- Training time: 15 days
Software
- Orchestration: Megatron-LM/Transformers
- Neural networks: PyTorch
Citation
@article{muennighoff2023octopack,
title={OctoPack: Instruction Tuning Code Large Language Models},
author={Niklas Muennighoff and Qian Liu and Armel Zebaze and Qinkai Zheng and Binyuan Hui and Terry Yue Zhuo and Swayam Singh and Xiangru Tang and Leandro von Werra and Shayne Longpre},
journal={arXiv preprint arXiv:2308.07124},
year={2023}
}