Cathallama

Awesome model, my new daily driver.

Edit: I am seeing a lot of token generations pointing to unknown unicode addresses that didn't show up during testing for this model, so I have stopped using it and I am working on a new version.

Notable Performance

• 9% overall success rate increase on MMLU-PRO over LLaMA 3.1 70b at Q4_0
• Strong performance in MMLU-PRO categories overall
• Great performance during manual testing

Creation workflow

Models merged

• meta-llama/Meta-Llama-3.1-70B-Instruct
• turboderp/Cat-Llama-3-70B-instruct
• Nexusflow/Athene-70B

flowchart TD
    A[Nexusflow_Athene] -->|Merge with| B[Meta-Llama-3.1]
    C[turboderp_Cat] -->|Merge with| D[Meta-Llama-3.1]
    B -->| | E[Merge]
    D -->| | E[Merge]
    E[Merge] -->|Result| F[Cathallama]

Testing

Hyperparameters

• Temperature: 0.0 for automated, 0.9 for manual
• Penalize repeat sequence: 1.05
• Consider N tokens for penalize: 256
• Penalize repetition of newlines
• Top-K sampling: 40
• Top-P sampling: 0.95
• Min-P sampling: 0.05

LLaMAcpp Version

• b3527-2-g2d5dd7bb
• -fa -ngl -1 -ctk f16 --no-mmap

Tested Files

• Cathallama-70B.Q4_0.gguf
• Nexusflow_Athene-70B.Q4_0.gguf
• turboderp_Cat-Llama-3-70B-instruct.Q4_0.gguf
• Meta-Llama-3.1-70B-Instruct.Q4_0.gguf

Tests

Manual testing

Note: See sample_generations.txt on the main folder of the repo for the raw generations.

MMLU-PRO

Note: MMLU-PRO Overall tested with 100 questions. Categories testes with 20 questions from each category.

PubmedQA

Request

If you are hiring in the EU or can sponsor a visa, PM me :D

PS. Thank you mradermacher for the GGUFs!