Model Card for tinymistral-v2-pycoder-instruct-248m
This modelcard is for tinymistral-v2-pycoder-instruct, a python-specific code generation model on top of Locutusque/TinyMistral-248M-v2-Instruct.
Model Details
This instruct model follows the original in using ChatML format.
An empty prompt will return various information from the base model, but using the instruct format will deliver python code of varying quality.
Model Description
Model is in active development, base model is in active development, and all should be treated with caution.
- Developed by: [Locutusque and M4ai]
- Funded by: [Lint from a corner pocket]
- Shared by: jtatman
- Model type: MistralForCausalLM
- License: [MIT]
- **Finetuned from model Locutusque/TinyMistral-248M-v2
Uses
Generate python code.
Direct Use
Probably could be fine tuned with a more comprehensive dataset. Experiments are in progress.
How to Get Started with the Model
Use the prompt format below to get started with the model.
<|im_start|>user Write a function for multiplying two numbers, from variables 'a' and 'b'.<|im_end|> <|im_start|>assistant
Training Details
Training Data
Custom formatted existing python data from:
- jtatman/python-code-dataset-500k
- jtatman/python-github-code-instruct-filtered-5k
- jtatman/pile_python_instruct_format
Training Procedure
Repeat training depending on compute budget.
Preprocessing
Conversion to alpaca/instruct format.
Training Hyperparameters
- Training regime: fp16, merge of parameter fine-tune adapters when necessary and helpful.
Evaluation
Metrics
Latest metrics:
- epoch: 4.87
- global_step: 220
- learning_rate: 0.00006713780918727916
- loss: 2.3736