Model Card for Model ID
This is just a test project - don't really use it!
This is a quantized version of TinyLlama-1.1B-Chat-v1.0
It was tested for performance with these results:
| Metric | Base Model | Quantized Model | Change |
|---|---|---|---|
| hellaswag acc | 0.485 | 0.495 | unchanged |
| hellaswag acc_norm | 0.64 | 0.64 | unchanged |
| eval time (CPU) - seconds | 970 | 751 | -23% |
Hellaswag accuracy was essentially maintained in the quantized version of the model, while time to complete the hellaswag eval decreased by ~23% (in a run on a low end CPU, with no access to a GPU).
For expediency a limited version of hellaswag was run with just 1% of samples evaluated - if this was going to be used for real we'd want to run the full eval set.
Note that all of the work to quantize and test the model were done using only CPU.
Model Details
Model Description
A quantized version of TinyLlama-1.1B-Chat-v1.0
- Developed by: Eric
- Model type: GGUF quantized 4 bit
Quantization Approach
The model was quantized to 4-bits (using llama.cpp and the Q4_K_M method), i.e. using commands like the following.
Convert original model to gguf:
python ./llama.cpp/convert_hf_to_gguf.py ./llama.cpp/models/TinyLlama-1.1B-Chat-v1.0/
Quantize gguf to 4-bit Q4_K_M:
./llama.cpp/build/bin/llama-quantize ./llama.cpp/models/TinyLlama-1.1B-Chat-v1.0/TinyLlama-1.1B-Chat-v1.0-F16.gguf q4_k_m