a small model is all you need

We celebrate three years of experiments with the release of palmer-006, our flagship and extremely small language model for local experimentation, edge hardware, small SBCs, and low-power devices. Note that this model serves primarily as a language model and has not been validated nor finetuned as a general-purpose chat assistant. Thus further finetuning is required to properly work as an instruction-tuned model.

updates

• 🐦‍🔥 A wild palmer appeared, go catch it!
• No longer promoting arithmark, after doing proper evals, palmer-006 is barely better than the base model used to train. Model is #5 on open slm leaderboard
• Second spot on BananaMindBench Leaderboard!!🎉🎉

● details

We first lightly fine-tuned the base model on a diverse set of highly curated data points. We then “palmerized” it by merging models, followed by another light fine-tuning round. Finally, we adjusted Mamba for maximum token speed. With only 90M parameters, this turned the model into a competitive baseline against models above 125M parameters.

As seen in the benchmark, we are getting closer to the performance of bigger models with our current technique, even on models of this size. Note: We used acc_norm whenever available.

● quick test

You can load model.gguf model into a llama-server, then run the script inside this repo as follows:

python completion.py prompt.txt

● license

Open research, education, hobby use, modification, and redistribution are permitted. Commercial deployment, internal business use, paid products, hosted APIs, and client work require a separate commercial license from appvoid. For commercial licensing: nosoyhackercodigo@gmail.com

👋 But hey! If you already have a donation subscription, you can claim access to a free commercial license through the email above. You will keep the rights as long as the subscription is active.

● upstream attribution

palmer-006 is built using artificial intelligence technology from the Technology Innovation Institute.

● fun facts

• the new logo represents palmer's eye, pointing how small this new model is
• palmer models still exists to some extent due to inspiration from Karpathy's idea of a self-contained "cognitive core".
• we just like light finetuning because we prefer starting "real" experiments with the strongest baseline possible
• we have an org with early access checkpoints