Llama 3 DreamGen Opus

🚨 WARNING 🚨

This model has issues, please use the following preview models instead:

• New train on top of Llama 3 8B Base
• New train on top of Llama 3 8B Instruct

Make sure to read this discussion if the model won't stop generating output.

Models for (steerable) story-writing and role-playing. All Opus V1 models, including quants.

Resources

• Opus V1 prompting guide with many (interactive) examples and prompts that you can copy.
• Google Colab for interactive role-play using opus-v1.2-7b.
• Python code to format the prompt correctly.
• Join the community on Discord to get early access to new models.

Prompting

<|im_start|>system
(Story description in the right format here)
(Typically consists of plot description, style description and characters)<|im_end|>
<|im_start|>user
(Your instruction on how the story should continue)<|im_end|>
<|im_start|>text names= Alice
(Continuation of the story from the Alice character)<|im_end|>
<|im_start|>text
(Continuation of the story from no character in particular (pure narration))<|im_end|>
<|im_start|>user
(Your instruction on how the story should continue)<|im_end|>
<|im_start|>text names= Bob
(Continuation of the story from the Bob character)<|im_end|>

The Opus V1 extension is the addition of the text role, and the addition / modification of role names.

Pay attention to the following:

• The text messages can (but don't have to have) names, names are used to indicate the "active" character during role-play.
• There can be multiple subsequent message with a text role, especially if names are involved.
• There can be multiple names attached to a message.
• The format for names is names= {{name[0]}}; {{name[1]}}, beware of the spaces after names= and after the ;. This spacing leads to most natural tokenization for the names.

While the main goal for the models is great story-writing and role-playing performance, the models are also capable of several writing related tasks as well as general assistance.

Here's how you can prompt the model for the following tasks

Steerable Story-writing and Role-playing:

• Story-writing prompt guide
• Role-playing prompt guide
• Input:
  • System prompt: You provide story / role-play description, which consists of:
    • Plot description
    • Style description
    • Characters and their descriptions
  • Conversation turns:
    • Text / message turn: This represents part of the story or role play
    • Instruction: This tells the model what should happen next
• Output: Continuation of the story / role-play.

Story plot summarization

• Prompt guide
• Input: A story, or a few chapters of a story.
• Output: A description of the story or chapters.

Examples:

One day, Malygris decides to summon Nylissa's spirit from the grave using his necromantic powers. He performs the necessary rituals and incantations, and Nylissa appears before him as a ghostly apparition. However, as Malygris gazes upon her, he becomes increasingly disillusioned. He notices subtle changes in her appearance and behavior that do not match his memories of her. He begins to question whether this is truly the same Nylissa he once knew, or if it is just another illusion conjured by his magic.

As Malygris grapples with this uncertainty, he becomes increasingly despondent and disillusioned with life itself. He realizes that even his magic cannot bring back the past or make him truly happy. In the end, he dismisses Nylissa's spirit and returns to his lonely tower, resigned to his fate as an old and weary sorcerer.

Story character description

• Prompt guide
• Input: A story, or a few chapters of a story, set of characters.
• Output: A description of the characters.

Examples:

Story style description

• Prompt guide
• Input: A story, or a few chapters of a story.
• Output: A description the style of the story.

Story description to chapters

• Prompt guide
• Input: A brief plot description and the desired number of chapters.
• Output: A description for each chapter.

And more...

Sampling params

For story-writing and role-play, I recommend "Min P" based sampling with min_p in the range [0.01, 0.1] and with temperature in the range [0.5, 1.5], depending on your preferences. A good starting point would be min_p=0.1; temperature=0.8.

You may also benefit from setting presence, frequency and repetition penalties, especially at lower temperatures.

Dataset

The fine-tuning dataset consisted of ~100M tokens of steerable story-writing, role-playing, writing-assistant and general-assistant examples. Each example was up to 31000 tokens long.

All story-writing and role-playing examples were based on human-written text.

Running the model

The model is should be compatible with any software that supports the base model, but beware of prompting and tokenization.

I recommend using these model versions:

• 7B: no quant (opus-v1.2-7b)
• 34B: no quant (opus-v1-34b) or awq (opus-v1-34b-awq)
• 34B: no quant (opus-v1.2-70b) or awq (opus-v1.2-70b-awq)

Running on DreamGen.com (free)

You can run the models on dreamgen.com for free — you can use the built-in UI for story-writing & role-playing, or use the API.

Running Locally

• Make sure your prompt is as close as possible to the Opus V1
  • Regardless of which backend you use, it's important that you format your prompt well and that the tokenization works correctly.
  • Read the prompt guide
  • Read the prompt formatting code
  • Make sure <|im_start|> and <|im_end|> are tokenized correctly
• vLLM
  • Google Colab: This is a simple interactive Google Colab to do role-play with the 7B model, it should fit on the T4 GPU.
  • Code: This is simple script for interactive chat for one hard-coded scenario.
• SillyTavern
  • Official SillyTavern documentation for DreamGen -- applies to both the API an local models
  • SillyTavern (staging) comes with built-in DreamGen preset for RP
    • Other presets can be found here, v2 kindly provided by @MarinaraSpaghetti
    • Make sure to unselect Skip special tokens, otherwise it won't work
    • This is just an attempt at approximating the Opus V1 prompt, it won't be perfect
  • Character cards specifically rewritten for the built-in DreamGen preset:
    • Seraphina (based on the default Seraphina card)
    • Lara Lightland (based on the card by Deffcolony)
• LM Studio
  • Config
  • Just like ChatML, just changed "assistant" to "text" role.
  • There's a bug in LM Studio if you delete a message or click "Continue", see here for details.
• HuggingFace
  • Chat template
  • Just like ChatML, just changed "assistant" to "text" role.

Known Issues

• 34B repetition:
  • The 34B sometimes gets stuck repeating the same word, or synonyms. This seems to be a common problem across various Yi 34B fine-tunes.
• GGUF:
  • The tokenization might be messed up. Some users reported that <|im_start|> and <|im_end|> are tokenized as multiple tokens. Also llama.cpp may not tokenize correctly (the Yi tokenizer is subtly different from the Llama 2 tokenizer).

License

• This model is intended for personal use only, other use is not permitted.