OUI-1 GGUF
This model is a quantization of:
https://huggingface.co/thesysdev/OUI-1
It was quantized using Unsloths's fork of llama.cpp b10840-mix-d5c17a0.
How to use
The easiest way to serve this model is to install Unsloth Desktop:
https://unsloth.ai/docs/desktop
Enable API endpoint
Enable the openai compatible endpoint:
-
Download a GGUF, run unsloth desktop and load the model.
-
Go to settings > api
-
Generate token and note it down.
-
Scroll down to lan access > start (you can customize the port)
-
Now you can go to https://github.com/thesysdev/openui and test out the demos while pointing it to your own endpoint.