h2o-danube3-4b-chat-GGUF
- Model creator: H2O.ai
- Original model: h2oai/h2o-danube3-4b-chat
Description
This repo contains GGUF format model files for h2o-danube3-4b-chat quantized using llama.cpp framework.
Table below summarizes different quantized versions of h2o-danube3-4b-chat. It shows the trade-off between size, speed and quality of the models.
| Name | Quant method | Model size | MT-Bench AVG | Perplexity | Tokens per second |
|---|---|---|---|---|---|
| h2o-danube3-4b-chat-F16.gguf | F16 | 7.92 GB | 6.43 | 6.17 | 479 |
| h2o-danube3-4b-chat-Q8_0.gguf | Q8_0 | 4.21 GB | 6.49 | 6.17 | 725 |
| h2o-danube3-4b-chat-Q6_K.gguf | Q6_K | 3.25 GB | 6.37 | 6.20 | 791 |
| h2o-danube3-4b-chat-Q5_K_M.gguf | Q5_K_M | 2.81 GB | 6.25 | 6.24 | 927 |
| h2o-danube3-4b-chat-Q4_K_M.gguf | Q4_K_M | 2.39 GB | 6.31 | 6.37 | 967 |
| h2o-danube3-4b-chat-Q3_K_M.gguf | Q3_K_M | 1.94 GB | 5.87 | 6.99 | 1099 |
| h2o-danube3-4b-chat-Q2_K.gguf | Q2_K | 1.51 GB | 3.71 | 9.42 | 1299 |
Columns in the table are:
- Name -- model name and link
- Quant method -- quantization method
- Model size -- size of the model in gigabytes
- MT-Bench AVG -- MT-Bench benchmark score. The score is from 1 to 10, the higher, the better
- Perplexity -- perplexity metric on WikiText-2 dataset. It's reported in a perplexity test from llama.cpp. The lower, the better
- Tokens per second -- generation speed in tokens per second, as reported in a perplexity test from llama.cpp. The higher, the better. Speed tests are done on a single H100 GPU
Prompt template
<|prompt|>Why is drinking water so healthy?</s><|answer|>