OPT-Alpaca-125M

This model is a fine-tuned version of facebook/opt-125m on the tatsu-lab/alpaca dataset. This is a test to check if the concept of Databrick's databricks/dolly-v1-6b can be applied to smaller models.

Model description

This is an LLM trained to follow instructions, similar to ChatGPT, except much much smaller.

Sample Use

from transformers import pipeline

generator = pipeline('text-generation', model='linkanjarad/OPT-Alpaca-125M')

prompt = '''### Instruction:
What is the capital of Australia?

### Response:'''

output = generator(prompt, do_sample=True, min_length=10, max_length=50, temperature=0.5)
print(output[0]['generated_text'])

Intended uses & limitations

The model works best when given input in the format of the dataset it was trained on. Here is an example of the format of the tatsu-lab/alpaca dataset:

### Instruction:
What are the three primary colors?

### Response:
The three primary colors are red, blue, and yellow.

Here are a few sample model outputs (temperature=0.7):

### Instruction:
What is the capital of Australia?

### Response:
The capital of Australia is Canberra.

### Instruction:
How do rainbows form?

### Response:
Rainbows form when the light of a sun enters the earth's atmosphere. The sun's rays are absorbed by the atmosphere, creating rainbows. The sun's light is then used to create a rainbow, which is a result of the Earth's rotation.

### Instruction:
Write me a poem about pancakes in the style of Dr. Seuss.

### Response:
Pancakes are like a wonderful miracle. They give us a beautiful and delicious breakfast.
Pancakes go from day to day,
But they also bring us pleasure.

I'm so glad I made the time to make my own
Pancakes are a delight for me and my family.
I'm going to make a special treat for you

### Instruction:
If you put an egg on the tip of a pencil, what will happen?

### Response:
An egg will start to move around the pencil without a trace.

### Instruction:
Write me code in Python that prints out the first ten numbers in the Fibonacci sequence.

### Response:
def number_primes(fibonacci):
    for i in range(10):
       for j in range(10):
          print(i)
    return i

### Instruction:
Write an article about card games with SEO.

### Response:
Card games are a popular game that allow players to play the game of cards with friends. The game is a puzzle game with players finding the correct cards that will be used to win a game. Players can use a variety of strategies to reach the right goal and win the game. Card games provide a lot of fun, and are a great way of providing a fun and engaging experience.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 64
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.15
• num_epochs: 3

Framework versions

• Transformers 4.27.4
• Pytorch 1.13.0
• Datasets 2.1.0
• Tokenizers 0.13.2