Atlas
Local Atlas from the game No Man's Sky.
Serving GGUF & AWQ Models with vLLM
Examples for running GGUF (via llama.cpp) and AWQ (via vLLM).
GGUF Version (via llama.cpp)
# pip install llama-cpp-python
from llama_cpp import Llama
llm = Llama.from_pretrained(
repo_id="Knixee/Atlas-nms-v2",
filename="Atlas-2.0-Q4_K_M.gguf", # Or Q4_K_S, Q8_0, BF16
)
output = llm("User: Hello!\nAssistant:", max_tokens=512)
print(output["choices"][0]["text"])
AWQ Version (via vLLM)
To launch 4bit AWQ version you need to download this folder
# pip install vllm
vllm serve Knixee/Atlas-nms-v2 \
--subfolder Atlas-2.0-AWQ-4bit \
--served-model-name Atlas-nms-v2 \
--quantization awq \
--max-model-len 8192 \
--gpu-memory-utilization 0.88 \
--max-num-seqs 6 \
--port 8148
API Request Example:
curl http://localhost:8148/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "Atlas-nms-v2", "messages": [{"role": "user", "content": "Hello!"}]}'