Goku-8x22B-v0.1 (Goku 141b-A35b)

A fine-tuned version of v2ray/Mixtral-8x22B-v0.1 model on the philschmid/guanaco-sharegpt-style dataset. This model has a total of 141b parameters with 35b only active.

How to use it

Use a pipeline as a high-level helper:

from transformers import pipeline

pipe = pipeline("text-generation", model="MaziyarPanahi/Goku-8x22B-v0.1")

Load model directly:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("MaziyarPanahi/Goku-8x22B-v0.1")
model = AutoModelForCausalLM.from_pretrained("MaziyarPanahi/Goku-8x22B-v0.1")

Load via Adapter:

You can also use PEFT to just load the adapter if you already have one of these models downloaded: v2ray/Mixtral-8x22B-v0.1 or mistral-community/Mixtral-8x22B-v0.1 (they are the same)

# assuming you have already downloaded the 
# resizing the vocab
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id="v2ray/Mixtral-8x22B-v0.1"
peft_model_id = "~/.cache/huggingface/hub/models--MaziyarPanahi--Goku-8x22B-v0.1/adapter"

tokenizer = AutoTokenizer. from_pretrained (peft_model_id)
model = AutoModelForCausalLM. from_pretrained (model_id)
# I have added 2 new tokens for ChatML template
# this step is required if you are using PEFT/Adapter
model.resize_token_embeddings (len (tokenizer))
model.load_adapter(peft_model_id)

# you can even have TextStreamer and a text-generation pipeline with your adapter
streamer = TextStreamer(tokenizer)

pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=750,
    temperature=0.6,
    do_sample=True,
    top_k=50,
    top_p=0.95,
    repetition_penalty=1.1,
    return_full_text=False,
    add_special_tokens=False,
    streamer=streamer
)

Examples

Goku-8x22B-v0.1 has been tested in generating text, answering questions based on long context, coding, and some reasoning. In the next version I will use more math and coding related datasets.

This is a sample story written by MaziyarPanahi/Goku-8x22B-v0.1

Goku had heard a commotion from his house but when he went to check he saw nothing. He thought to himself, "I'll let it go, it was probably just a bird or something. I'm sure it will be fine." But that was when he heard the commotion again, so he went outside and this time he saw two figures on the horizon. One of the figures was a giant pinkish-purple creature, while the other was small, pink, ball-shaped thing.

As the figures approached, Goku realized the large creature was his former enemy, the powerful Majin Buu. And the smaller creature was Kirby, a powerful Star Warrior from the planet Popstar. Goku couldn't believe his eyes.

The two creatures approached Goku menacingly. "Kirby and I have teamed up," said Majin Buu. "We're going to destroy the world!"

Goku was taken aback by the statement. He had never considered the possibility of these two powerful creatures joining forces. He knew he had to put a stop to them, before they could cause any more damage.

He took a deep breath and faced the two creatures. "You two won't get away with this," Goku said firmly. "I won't let you destroy the world."

Majin Buu scoffed, "You can't stop us! Kirby and I are too powerful!"

Goku quickly formed an energy ball in his hands and faced the two creatures. "We'll see about that," he said.

The battle that ensued was intense. The two creatures worked together, using their powerful energy attacks to try to overcome Goku. But Goku kept fighting, using his own powerful energy attacks to counter their moves.

After what seemed like an eternity, Goku managed to get the upper hand. He used a powerful energy attack to defeat the two creatures. After they were defeated, Goku looked around and saw the damage that had been caused by the battle. He knew he still had a lot of work ahead of him in order to prevent any further destruction, but he was determined to do his best.

He summoned all of his power and focused it into a powerful energy attack. The energy spread throughout his body and he felt his power grow stronger. With a battle cry, he launched the attack at the two creatures.

The energy hit them both, sending them flying back, stunned for a moment. Goku continued to pressure them with his energy attacks, but they soon recovered and began to counter-attack with their own energy blasts.

Goku knew he had to act quickly if he was going to defeat them. He focused his energy into one powerful attack, and launched it at Kirby. The attack hit and the Star Warrior was sent flying away.

Goku then focused his attention on Majin Buu. He launched a series of energy attacks, using his signature technique, the Kamehameha, and managed to defeat the powerful creature.

After the battle, Goku looked around at the destruction that had been caused by the two creatures. He knew he still had a lot of work ahead of him in order to prevent any further destruction, but he was determined to do his best.

With the two creatures defeated, Goku knew he still had a job to do. He took a deep breath and set out to repair the damage that had been caused by the two powerful creatures. He worked for hours, using his energy to put everything back in order and ensuring that the world was safe once again.

Goku's journey was long and hard but, in the end, he was successful. He defeated two powerful enemies and saved the world from destroyers. Thanks to his hard work, the world was able to heal and once again become a place of peace and prosperity.