Specialized quants (including "MXFP4_MOE.gguf") for new OpenAI 20B MOE - Mixture of Experts Model at 80+ t/s.
These are DUEL NEO Imatrix / CODE IMATRIX GGUFs, NEO dataset by DavidAU with "Code" dataset compiled by DavidAU.
NEO dataset improves overall performance.
CODER dataset is specifically for coding performance.
Example generation provided at bottom of this page.
DUEL ("DI")-> Separate Imatrix datasets ("NEO" and "CODER" - generated separately per model) are co-joined to create a new Imatrix dataset, which is then applied to the quants.
Model also passed "hard" coding test too (4 experts); no issues (IQ4_NL).
TEST: Forcing the model to create code with no dependencies and limits of coding short cuts, with multiple loops, and in real time with no blocking in a language that does not support it normally.
(example prompt, and generation below)
Due to quanting issues with this model (which result in oddball quant sizes / mixtures), only TESTED quants will be uploaded (at the moment).
Currently that means IQ4_NL, and MXFP4_MOE (a new OpenAI specific quant type) are available.
NEO/Coder datasets performance improvements will show the most in the IQ4_NL quants.
There are TWO "IQ4_NL" quants:
- OpenAI-20B-NEO-CODE-DIMAT-IQ4_NL.gguf : DI Imatrix applied.
- OpenAI-20B-NEO-CODE-DIMAT-2-IQ4_NL.gguf : DI Imatrix applied, including output tensor (also imatrixed), and embed tensor at IQ4_NL.
There are THREE NEO MXFP4_MOE quants:
- OpenAI-20B-NEO-CODE-DIMAT-MXFP4_MOE2.gguf : Output tensor Q5_1 (DI Imatrix applied)
- OpenAI-20B-NEO-CODE-DIMAT-MXFP4_MOE3.gguf : Output tensor IQ4_NL (DI Imatrix applied)
- OpenAI-20B-NEO-CODE-DIMAT-MXFP4_MOE4.gguf : Output tensor IQ4_NL (DI Imatrix applied) AND Embed at IQ4_NL - this makes this quant the smallest version.
MXFP4_MOE quants vastly outperform (at the moment) all other quants, except IQ4_NL, Q5_1 and Q8_0 due to odd issues compressing OpenAI's 20B model due to odd "tensor" dimensions.
( IQ4_NL, Q5_1 and Q8_0 quants are compatible with OpenAI's tensor structure as of this writing. )
MXFP4_MOE quant was specifically designed for (what appears to be) loss less quantization relative to source model. Jury still out on this.
Suggested Settings (tested in Lmstudio, Beta Branch 0.3.21 ; 4 ):
- Context: 8k min.
- Temp 1.2+ for creative. Temp .6 (or so) for coding/general.
- Rep pen 1.1, topk 40, topp .95, min p 0.05
- Experts 4-8 depending on use case. (higher than 8 MAY lower quality)
Model Supports:
- 128k context
- up to 24 experts
- Tools use, browsing, etc
For my help docs, SETTING NUMBER OF EXPERTS, and other see below.
See more about this model here:
https://huggingface.co/openai/gpt-oss-20b
Please refer to their model card, especially to control "thinking" levels.
CHANGE THE NUMBER OF ACTIVE EXPERTS:
See this document:
https://huggingface.co/DavidAU/How-To-Set-and-Manage-MOE-Mix-of-Experts-Model-Activation-of-Experts
Settings: CHAT / ROLEPLAY and/or SMOOTHER operation of this model:
In "KoboldCpp" or "oobabooga/text-generation-webui" or "Silly Tavern" ;
Set the "Smoothing_factor" to 1.5
: in KoboldCpp -> Settings->Samplers->Advanced-> "Smooth_F"
: in text-generation-webui -> parameters -> lower right.
: In Silly Tavern this is called: "Smoothing"
NOTE: For "text-generation-webui"
-> if using GGUFs you need to use "llama_HF" (which involves downloading some config files from the SOURCE version of this model)
Source versions (and config files) of my models are here:
OTHER OPTIONS:
-
Increase rep pen to 1.1 to 1.15 (you don't need to do this if you use "smoothing_factor")
-
If the interface/program you are using to run AI MODELS supports "Quadratic Sampling" ("smoothing") just make the adjustment as noted.
Highest Quality Settings / Optimal Operation Guide / Parameters and Samplers
This a "Class 1" model:
For all settings used for this model (including specifics for its "class"), including example generation(s) and for advanced settings guide (which many times addresses any model issue(s)), including methods to improve model performance for all use case(s) as well as chat, roleplay and other use case(s) please see:
You can see all parameters used for generation, in addition to advanced parameters and samplers to get the most out of this model here:
Using default settings (coder), 4 experts and no system prompt.
Prompt:
Write a JavaScript for fetch api using 4 ais, all using streaming that does NOT use "promise.all", and processes all streams separately but in real time - that means all streams need to be processed at the same time and not interrupt each other.
Also include sending a payload to each of the 4 ais with content, system prompt, temp, topk, topp, minp, repeat penalty ; all of these parameters will be in an array for each of the four ais.
Record the entire response - chunks - from each AI in an array - text.
Also, send the chunks in real time to 4 separate text boxes on an HTML page so real time readout from each AI can be seen by the user.
IMPORTANT: All streams must be processed at the same time, not in sequence. IMPORTANT: Do not use any dependencies, nor import any module(s).
OUTPUT:
(note: Pretty print, color and some formatting lost on copy/paste)
<!-- Start button --> <button id="startBtn">Start All 4 AIs</button>
<script src="ai-stream.js"></script> <!-- the script below --> </body> </html>
/* ------------------------------------------------------------- * 2. Storage for the full responses (array of strings) * ------------------------------------------------------------- */ const allResponses = [[], [], [], []]; // one array per AI
/* -------------------------------------------------------------
* 3. Helper – read a stream and push chunks to UI & storage
* ------------------------------------------------------------- */
async function streamFromAi(aiIndex) {
const cfg = aiConfigs[aiIndex];
const outputBox = document.getElementById(output${aiIndex});
const decoder = new TextDecoder('utf-8');
try { // 3.1 Send the request const response = await fetch(cfg.url, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(cfg.payload) });
if (!response.ok) {
throw new Error(HTTP ${response.status} – ${response.statusText});
}
// 3.2 Get the reader const reader = response.body.getReader();
// 3.3 Read loop – runs until stream ends while (true) { const { value, done } = await reader.read(); if (done) break; // no more data
// Convert Uint8Array to string const chunkText = decoder.decode(value, { stream: true });
// Store the chunk in the array allResponses[aiIndex].push(chunkText);
// Push it live into the textarea outputBox.value += chunkText; }
// Finalise – flush any remaining buffered text outputBox.value += decoder.decode(); // empty decode to flush
} catch (err) {
console.error(AI #${aiIndex+1} error:, err);
outputBox.value += \n[Error] ${err.message}\n;
}
}
/* -------------------------------------------------------------
* 4. Kick‑off – start all four streams concurrently
* ------------------------------------------------------------- */
function startAllAIs() {
// Reset UI and storage
for (let i = 0; i < 4; ++i) {
document.getElementById(output${i}).value = '';
allResponses[i] = [];
}
// Start each stream – the async function is called but not awaited, // so they run in parallel. for (let i = 0; i < 4; ++i) { streamFromAi(i); // <-- no await } }
/* ------------------------------------------------------------- * 5. Wire up the button * ------------------------------------------------------------- */ document.getElementById('startBtn').addEventListener('click', startAllAIs);