bge-reranker-v2-m3, 8-bit MLX (Swift)

BAAI/bge-reranker-v2-m3 (revision 953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e), quantized to 8-bit for MLX Swift: 0.64 GB instead of about 2.2 GB in fp32.

Stilltone, a local-first meeting-notes app for macOS, uses it as the multilingual cross-encoder reranker (is a new item really about an existing one?). It runs entirely on the Mac.

How it was made and checked

• Made: the original fp32 weights were loaded by Stilltone's own MLX Swift loader and quantized in memory (MLXNN.quantize, affine, 8-bit, group size 64, its default Linear and Embedding layers). The resulting module parameters were saved unchanged as model.safetensors. quantization.json records the scheme.
• Checked: Stilltone's memory-diff benchmark (four packs: English, Czech and a multilingual parliament pack) produced byte-identical results with these files and with the original fp32 weights. That benchmark exercises the embedder, the reranker and the NLI model together.

Format: read this before using it elsewhere

These files are MLX Swift weights, not a Transformers checkpoint. Parameter names follow MLXEmbedders' BertModel after its sanitize step, and quantized layers store weight / scales / biases. To load them, build the same module, call quantize(model:groupSize:64,bits:8,mode:.affine) on it first, then load the arrays. That's what Stilltone does when quantization.json is present. transformers and sentence-transformers cannot load them directly; use the original repo for those.

config.json and the tokenizer files are unchanged from the source (listed below).

Files

• model.safetensors: the 8-bit parameters.
• quantization.json: {"bits": 8, "groupSize": 64, "mode": "affine"}.
• config.json, tokenizer.json, tokenizer_config.json and special_tokens_map.json are copied from the source revision.

License and attribution

Apache License 2.0, the same as the original BAAI/bge-reranker-v2-m3; the text is in LICENSE. All credit for the model goes to BAAI. The change here is quantization only, described above.