Prodigy SM Base v0.1
In our latest endeavour, we performed continued pre-training of a large language model (Mistral-7b-v0.1) to understand and generate text in new languages, including Serbian, Bosnian and Croatian using an innovative approach.
Rather than depending only on extensive datasets in the target language, our method utilizes a more compact set of both synthetic and human-curated data along with some mixture of CC Web data, which is implemented in two strategic phases:
- Establishing a comprehensive demonstration of all grammatical and orthographic rules pertinent to the language.
- Supplying a diverse array of examples that not only reinforce these rules but also integrate a wide range of linguistic nuances.
While our approach is uniquely tailored to our objectives, we have drawn some inspiration from recent advancements in language model training. Specifically, the conceptual strategies discussed in the paper ADAPTING LARGE LANGUAGE MODELS VIA READING COMPREHENSION provided valuable insights, though our methods diverge significantly in practice. By adopting this inspired approach, we aim to efficiently teach the model new languages with a balanced blend of accuracy and linguistic diversity.
So... Did it work?!
Yes!
See the benchmark results, or even better, download the model and try it yourself. As you know by now, there's no better benchmark than a quick 'try it yourself' vibe check. :)
Here, we demonstrate results of benchmark that is not frequently performed, yet equally important: how adapting the model for a new language impacted its original English-only performance.
*All evals are performed in zero shot manner.
*Also bear in mind that llama-2-7b, llama-3-8b and mistral-7b models compared to Prodigy SM base aren't trained on extensive Serbian language datasets, and these benchmarks demonstrate that primarily English models can be adapted to other languages.
So, as you can see, we successfully improved the original model's performance for Serbian language use cases while retaining or even slightly improving its performance for English language.
Training results
Training results of continued pre-training of mistral-7b-v0.1
As last experimental step we merged produced model with Mistral-7B-v0.1 and two earlier checkpoints from prodigy-sm-base using Model Stock method.
Notes
As this is base model, there is no chat template or strict chat following capabilities, this model is best candidate for further pre-train on Serbian language as there is a lot more room for improvement (you can hit sweet spot), or next step in the pipeline, such as some form of chat or instruct tuning.
If you want model that is already instruction tuned we did that too, check Prodigy SM Instruct v0.1
Prodigy SM Instruct v0.1
🚀prodigy-sm-instruct COMING SOON
And stay tuned for:
prodigy-sm-base (llama-3.1) COMING SOON
prodigy-sm-instruct (llama-3.1) COMING SOON
📢 Also we are excited to announce that iskon.ai will soon launch an API platform featuring advanced Prodigy series of models, advanced AI tools and much more! 🚀
Thanks
- gordicaleksa/serbian-llm-eval and his community for curating translations and adaptation of lm-evaluation-harness that we used to perform benchmarks.
- jondurbin for amazing airoboros framework
- teknium for various insights shared on discord and twitter aka x.com
- Eric for various insights shared on discord and twitter aka x.com
- mergekit for model merging tools
Huge thanks to Redmond.ai for generous DGX cloud credits redmond.ai