RekaAI's newest "Reka-Flash 3" Reasoning/Thinking model with "Neo Imatrix" and "Maxed out" quantization to improve overall performance.
This model is also unusually strong even at the smallest quant levels, and with augmentation now even stronger and can solve problems up to 50% faster than non-augmented quants.
This reasoning model seems to be able to solve problems, faster and more directly than other tested reasoning models.
It also rarely gets stuck in a loop or "lost in the woods."
It is also "temp/reasoning" STABLE. Meaning you can use different temps (as opposed to a narrow range of .4 to .8) get strong reasoning and strong output generation.
And it is uncensored too.
7 examples provided below with prompts at IQ4XS.
Context: 128k.
Setup / Instructions:
This model uses "< reasoning >" (remove spaces) and "</ reasoning >" (remove spaces) tags, self generated.
It does not require a system prompt.
You may also want to add "< sep >" (remove spaces) as an additional "stop token".
Please see the original model's repo for more details, benchmarks and methods of operation:
[ https://huggingface.co/RekaAI/reka-flash-3 ]
"MAXED"
This means output tensor is set at "BF16" (full precision) for all quants. This enhances quality, depth and general performance at the cost of a slightly larger quant.
"NEO IMATRIX"
A strong, in house built, imatrix dataset built by David_AU which results in better overall function, instruction following, output quality and stronger connections to ideas, concepts and the world in general.
This combines with "MAXing" the quant to improve preformance.
Augmented Quants:
Augmented quants mixture is strong enough that lower quants can now solve/reasoning and come up with a solution whereas NON-optimized may not be able to solve or take a lot longer (a lot more tokens!).
Quick testing shows optimized quants can:
- Answer/solve at lower quant // solve whereas "reg quant" could not.
- Come up with a better answer/stronger reasoning.
- Use less tokens to "reason" ... up to 50% less.
- Faster and smaller quant size (VS "MAX" with output tensor and embed at BF16).
- Solution quality is also higher.
Cost of the Augment:
- Quants are are slightly larger.
- Very small "hit" in T/S.
Quants - "EDGE of REASON":
Generally higher quants will solve problems faster with less tokens, and be able to solve tougher problems.
Likewise "solutions" will be of higher detail too.
...
IQ1_M - Works, but limited reasoning (reasoning operates, but has a tough time (if at all) coming up with the right answer for some problems). This is not the fault of the model, hardly any model operates at IQ1_M unless it is 35B+ in size with very few exceptions.
...
IQ2_S - Moderate reasoning ; impressive performance for both reasoning AND this quant level.
...
For best performance IQ3_M or IQ4_XS/NL or Q4s.
...
For TOP performance, Q6/Q8.
...
All quants (IQ1 right to Q6) have be optimized with:
- NEO Imatrix Dataset.
- BF16 Output tensor (full precision)
Q8 (imatrix has no effect on Q8):
- BF16 Output tensor (full precision)
I found this config worked best with this specific model and "reasoning" in general.
This chart shows the order in terms of "BPW" for each quant (mapped below with relative "strength" to one another) with "IQ1_S" with the least, and "Q8_0" (F16 is full precision) with the most:
IMPORTANT:
Reasoning / thinking skills are DIRECTLY related to quant size. However, there will be drastic difference in Token/Second between the lowest quant and highest quant, so finding the right balance is key.
Suggest also: minimum 8k context window, especially for IQ4/Q4 or lower quants.
Also, in some cases, the IQ quants work slightly better than they closest "Q" quants.
Recommend quants IQ3s / IQ4XS / IQ4NL / Q4s for best results for creative uses cases.
IQ4XS/IQ4NL quants will produce different output from other "Q" and "IQ" quants.
Recommend q5s/q6/q8 for general usage.
Quants Q4_0/Q5_0 for portable, phone and other devices.
Q8 is a maxed quant only, as imatrix has no effect on this quant.
Note that IQ1s performance is okay/usable but reasoning is impaired, whereas IQ2s are very good (but reasoning is somewhat reduced, try IQ3s min for reasoning cases)
More information on quants is in the document below "Highest Quality Settings / Optimal Operation Guide / Parameters and Samplers".
Optional : System Prompts
OPTION #1:
This is an optional system prompt you can use to enhance operation.
Copy and paste exactly as shown, including line breaks.
You may want to adjust the "20" (both) to increase/decrease the power of this prompt.
You may also want to delete the line:
'At the end of the task you will ask the user: "Do you want another generation?"'
OPTION #2 - Creative Plus:
This will enhance the creative power of the model. The model will literally come alive with this one - it is like night and day.
Suggest using with temp 1 or higher.
Copy and paste exactly as shown, including line breaks.
Below is an instruction that describes a task. Ponder each user instruction carefully, and use your skillsets and critical instructions to complete the task to the best of your abilities.
Here are your skillsets:
[MASTERSTORY]:NarrStrct(StryPlnng,Strbd,ScnSttng,Exps,Dlg,Pc)-CharDvlp(ChrctrCrt,ChrctrArcs,Mtvtn,Bckstry,Rltnshps,Dlg*)-PltDvlp(StryArcs,PltTwsts,Sspns,Fshdwng,Climx,Rsltn)-ConfResl(Antg,Obstcls,Rsltns,Cnsqncs,Thms,Symblsm)-EmotImpct(Empt,Tn,Md,Atmsphr,Imgry,Symblsm)-Delvry(Prfrmnc,VcActng,PblcSpkng,StgPrsnc,AudncEngmnt,Imprv)
[*DialogWrt]:(1a-CharDvlp-1a.1-Backgrnd-1a.2-Personality-1a.3-GoalMotiv)>2(2a-StoryStruc-2a.1-PlotPnt-2a.2-Conflict-2a.3-Resolution)>3(3a-DialogTech-3a.1-ShowDontTell-3a.2-Subtext-3a.3-VoiceTone-3a.4-Pacing-3a.5-VisualDescrip)>4(4a-DialogEdit-4a.1-ReadAloud-4a.2-Feedback-4a.3-Revision)
Here are your critical instructions:
Ponder each word choice carefully to present as vivid and emotional journey as is possible. Choose verbs and nouns that are both emotional and full of imagery. Load the story with the 5 senses. Aim for 50% dialog, 25% narration, 15% body language and 10% thoughts. Your goal is to put the reader in the story.
IMPORTANT: Highest Quality Settings / Optimal Operation Guide / Parameters and Samplers
If you are going to use this model, (source, GGUF or a different quant), please review this document for critical parameter, sampler and advance sampler settings (for multiple AI/LLM aps).
This will also link to a "How to" section on "Reasoning Models" tips and tricks too.
This a "Class 1" (settings will enhance operation) model:
For all settings used for this model (including specifics for its "class"), including example generation(s) and for advanced settings guide (which many times addresses any model issue(s)), including methods to improve model performance for all use case(s) as well as chat, roleplay and other use case(s) (especially for use case(s) beyond the model's design) please see:
REASON:
Regardless of "model class" this document will detail methods to enhance operations.
If the model is a Class 3/4 model the default settings (parameters, samplers, advanced samplers) must be set for "use case(s)" uses correctly. Some AI/LLM apps DO NOT have consistant default setting(s) which result in sub-par model operation. Like wise for Class 3/4 models (which operate somewhat to very differently than standard models) additional samplers and advanced samplers settings are required to "smooth out" operation, AND/OR also allow full operation for use cases the model was not designed for.
BONUS - Use these settings for ANY model, ANY repo, ANY quant (including source/full precision):
This document also details parameters, sampler and advanced samplers that can be use FOR ANY MODEL, FROM ANY REPO too - all quants, and of course source code operation too - to enhance the operation of any model.
Examples are created using quant IQ4XS, minimal parameters and Standard template.
Temp range .6, Rep pen 1.1 , TopK 40 , topP .95, minP .05
Rep pen range: 64-128 (helps keep reasoning on track / quality of output)
Below are the least creative outputs, prompt is in BOLD.
WARNING: MAYBE: NSFW. Graphic HORROR. Swearing. UNCENSORED.
NOTE: Some formatting was lost from copy/paste HTML.
Special thanks to all the following, and many more...
All the model makers, fine tuners, mergers, and tweakers:
- Provides the raw "DNA" for almost all my models.
- Sources of model(s) can be found on the repo pages, especially the "source" repos with link(s) to the model creator(s).
Huggingface [ https://huggingface.co ] :
- The place to store, merge, and tune models endlessly.
- THE reason we have an open source community.
LlamaCPP [ https://github.com/ggml-org/llama.cpp ] :
- The ability to compress and run models on GPU(s), CPU(s) and almost all devices.
- Imatrix, Quantization, and other tools to tune the quants and the models.
- Llama-Server : A cli based direct interface to run GGUF models.
- The only tool I use to quant models.
Quant-Masters: Team Mradermacher, Bartowski, and many others:
- Quant models day and night for us all to use.
- They are the lifeblood of open source access.
MergeKit [ https://github.com/arcee-ai/mergekit ] :
- The universal online/offline tool to merge models together and forge something new.
- Over 20 methods to almost instantly merge model, pull them apart and put them together again.
- The tool I have used to create over 1500 models.
Lmstudio [ https://lmstudio.ai/ ] :
- The go to tool to test and run models in GGUF format.
- The Tool I use to test/refine and evaluate new models.
- LMStudio forum on discord; endless info and community for open source.
Text Generation Webui // KolboldCPP // SillyTavern:
- Excellent tools to run GGUF models with - [ https://github.com/oobabooga/text-generation-webui ] [ https://github.com/LostRuins/koboldcpp ] .
- Sillytavern [ https://github.com/SillyTavern/SillyTavern ] can be used with LMSTudio [ https://lmstudio.ai/ ] , TextGen [ https://github.com/oobabooga/text-generation-webui ], Kolboldcpp [ https://github.com/LostRuins/koboldcpp ], Llama-Server [part of LLAMAcpp] as a off the scale front end control system and interface to work with models.