Sophea-Titan-1-mlx

MLX conversions of KIEFERSA/Sophea-Titan-1 — a Greek/English multimodal (vision + video) language model from the Qwen3.5-VL family (model_type: qwen3_5, with a qwen3_5_vision tower and qwen3_5_text decoder). These builds run natively on Apple Silicon via mlx-vlm.

Three floating-point formats are provided, each in its own subfolder:

fp8 and fp4 use MLX's true micro-scaling floating-point quantization (mxfp8 / mxfp4), not integer affine quantization.

Requirements

pip install -U mlx mlx-vlm

Tested with mlx==0.31.2, mlx-vlm==0.6.3. Runs on Apple Silicon (M-series).

Memory (unified RAM / "VRAM") required

Apple Silicon uses unified memory shared between CPU and GPU, so "VRAM" = your Mac's total RAM. macOS reserves memory for the OS and, by default, caps the GPU working set (Metal recommendedMaxWorkingSetSize) at roughly 75% of total RAM (e.g. ~48 GB on a 64 GB Mac, ~96 GB on a 128 GB Mac). The "Min Mac RAM" column already accounts for that cap.

• Figures are anchored on a measured fp4 peak of 15.7 GB for short text generation, and line up with community MLX reports for Qwen3-VL-class 30B models (4-bit ≈ 18–20 GB, 8-bit ≈ 30 GB).
• Images, video, and long contexts add more (vision activations + KV cache), so pick a tier with headroom above the minimum if you feed large inputs.
• fp16 does not fit a 64 GB Mac at default settings (~55–60 GB peak vs the ~48 GB default GPU cap); 96 GB+ is recommended. A 64 GB Mac can only attempt fp16 by raising the wired limit and closing everything else, which is tight.
• On a Mac that's close to the minimum, you can raise the GPU memory cap:

  # allow the GPU to use more of unified memory (value in MB), e.g. 28 GB
  sudo sysctl iogpu.wired_limit_mb=28672
  

Usage

Because the three formats live in subfolders, download the one you want, then point mlx-vlm at the local folder.

from huggingface_hub import snapshot_download

# Pick a format: "fp4", "fp8", or "fp16"
path = snapshot_download(
    "KIEFERSA/Sophea-Titan-1-mlx",
    allow_patterns="fp4/*",
    token="<your_hf_token>",   # private repo
)
model_dir = f"{path}/fp4"

Then generate (text-only or with an image / video):

# text-only
python -m mlx_vlm generate --model "$model_dir" \
  --prompt "Γεια σου! Πες μου μια πρόταση στα ελληνικά." --max-tokens 128

# with an image
python -m mlx_vlm generate --model "$model_dir" \
  --image path/to/image.jpg \
  --prompt "Περίγραψε την εικόνα." --max-tokens 256

Or from Python:

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load(model_dir)
config = model.config
messages = [{"role": "user", "content": "Περίγραψε την εικόνα."}]
prompt = apply_chat_template(processor, config, messages, num_images=1)
out = generate(model, processor, prompt, ["path/to/image.jpg"], max_tokens=256, verbose=True)
print(out)

Serving (OpenAI-compatible HTTP server)

mlx-vlm ships an OpenAI-compatible server. Start it on a downloaded format folder:

python -m mlx_vlm.server --model "$model_dir" --host 0.0.0.0 --port 8080
# e.g. --model ./fp8   (or fp4 / fp16)

Then call it like the OpenAI Chat Completions API:

# text-only
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Γεια σου! Πες μου μια πρόταση στα ελληνικά."}],
    "max_tokens": 128
  }'

# with an image (URL or data: URI)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": [
      {"type": "text", "text": "Περίγραψε την εικόνα."},
      {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}
    ]}],
    "max_tokens": 256
  }'

Or with the OpenAI Python client:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="sophea-titan-1",  # any string; the server uses the preloaded --model
    messages=[{"role": "user", "content": "Γράψε ένα σύντομο ποίημα για τη θάλασσα."}],
    max_tokens=128,
)
print(resp.choices[0].message.content)

Useful flags: --max-tokens, --vision-cache-size, --prefill-step-size, and KV-cache quantization (--kv-bits, --max-kv-size) for long contexts. See python -m mlx_vlm.server --help.

Conversion

Converted from the bf16 source with mlx-vlm:

# fp16
python -m mlx_vlm convert --hf-path <src> --mlx-path fp16 --dtype float16

# fp8 (MXFP8)
python -m mlx_vlm convert --hf-path <src> --mlx-path fp8 \
  -q --q-bits 8 --q-mode mxfp8 --q-group-size 32 --dtype float16

# fp4 (MXFP4)
python -m mlx_vlm convert --hf-path <src> --mlx-path fp4 \
  -q --q-bits 4 --q-mode mxfp4 --q-group-size 32 --dtype float16

Note: the source ships a unified processor_config.json; mlx-vlm expects preprocessor_config.json, so it was aliased before conversion (both are included here).

Notes & limitations

• fp16 is the reference-quality build; fp8 is near-lossless at ~half the size; fp4 is the smallest and fastest to load, with some quality trade-off (expected for 4-bit).
• Smoke-tested: the fp4 build loads and produces fluent, correctly-accented Greek (~19 tokens/sec, ~15.7 GB peak) on an Apple Silicon laptop.
• Inherits the capabilities, license, and any limitations of the base model KIEFERSA/Sophea-Titan-1. Refer to the base model card for training details and intended use.