prompt2model-reference-onnx

This is a toy reference artifact, not a production model. It is the ONNX classification export that the public prompt2model-demo Space's conformal-abstain browser evaluates. Its purpose is to give people a real, small, fast-loading model file to poke at while looking at how Prompt2Model's factory pipeline (prompt to config to training to metrics to ONNX export to report) behaves end to end, not to demonstrate classification accuracy on any task that matters.

This card also documents a second, separate finding further down: compressing Dhi's actual production edge models with the same factory machinery (see "A separate finding" section below). That section is about different models entirely, on a different date; no production weights are published here.

What it is

• Task: image classification, 3 synthetic classes: red_square, blue_circle, green_triangle.
• Backbone: mobilenet_v3_small, ImageNet-pretrained (the smoke test path hardcodes a pretrained backbone as of PR #18).
• Parameters: 1,520,931 (about 1.52M).
• Input: 96x96 RGB images.
• Export format: ONNX, opset 17.
• Training data: a tiny synthetic toy set (colored shapes), 2 epochs, batch size 8, low_light augmentation tag, generated by the same run that produced this export. Not a natural-image dataset. See prompt2model-examples for the matching example fixtures.

Measured metrics (verbatim, this run only)

This is the deterministic smoke run after PR #18 (seeded RNGs, stratified splits, pretrained backbone), the same artifact the demo Space evaluates.

{
  "accuracy": 1.0,
  "macro_f1": 1.0,
  "latency_ms": 78.61008350737393,
  "fps": 12.721014345522176,
  "parameter_count": 1520931,
  "calibration": {
    "calibrated": true,
    "temperature": 0.05,
    "alpha": 0.1,
    "conformal_threshold": 0.004888,
    "ece_before": 0.3864,
    "ece_after": 0.0014,
    "val_samples": 7
  }
}

• Chance level on this 3-class set is 0.33. Accuracy 1.0 / macro F1 1.0 is expected and not impressive: the toy set is tiny (a handful of synthetic images per class), the task is trivially separable (three fixed colored shapes), and the split is small enough that a perfect score carries very little statistical weight. Treat these numbers as "the pipeline plumbing works end to end," not "this model classifies well."
• Latency/FPS were measured locally on a development machine (Apple Silicon Mac, ONNX Runtime CPU execution provider) via the same onnxruntime.InferenceSession the model ships in, not a production or edge-device benchmark.
• Calibration numbers are from the pipeline's conformal-abstain stage (temperature scaling + conformal threshold), which is what the demo Space's "abstain browser" actually visualizes.

What this is not

• Not a production or deployment-ready classifier.
• Not evidence of real-world classification quality: the label set, images, and split are all synthetic and tiny.
• Not tuned, ablated, or compared against any baseline beyond the trivial chance rate.

A separate finding: compressing Dhi's actual production edge models (2026-07-28)

Everything above is about the toy reference model. This section is a different, later experiment: applying the same factory compression machinery (prompt2model.compression.quantize_onnx + decide_gate) to Dhi's real production edge models (object/pose/PPE/face detectors and violence classifiers from aixavier/models), not synthetic fixtures. No weights from Dhi's production models are published in this repo or anywhere on this org; these are proprietary production artifacts. Only the methodology and measured results below are published, reproduced from a fresh run of scripts/compress_production_models.py and saved as b1_production_compression_summary.json.

The metric is output agreement, explicitly not accuracy. There is no labeled ground-truth eval set for these detector/pose/PPE/face/violence models available to this run (a public crowd and occlusion benchmark lives on a separate GPU box and was never touched here; aixavier ships no boxed annotations for its PPE/pose/face models either, only unlabeled demo videos and a handful of face crops). So instead of fabricating an accuracy number, every model is graded on mean cosine similarity between the FP32 and INT8-quantized ONNX graph's raw outputs on the same real, unlabeled frames pulled from Dhi's own demo videos and test fixtures. This is a fidelity check ("did compression change what the model sees"), never relabeled as accuracy, fed through the same never-softened accuracy-floor gate (relative floor 0.98) that the toy classification path above uses.

Every artifact passed the output-fidelity gate (all held at or above the 0.98 relative floor against their own FP32 self-fidelity of 1.0), so nothing was refused here. weights/face/model.pth (an mmdetection 2.7 training checkpoint, not the artifact the runtime actually loads) was skipped outright and is not in the table; its runtime-loaded equivalent, scrfd_34g_gnkps.onnx, is the face_detector_scrfd34g row instead.

Why the split is so stark: an honest negative result. The compression gate deliberately restricts dynamic INT8 quantization to MatMul/Gemm nodes only, because Conv quantization previously collapsed a real classifier to chance accuracy in this same codebase, a decision this run does not soften. Measured node counts explain the whole pattern: yolo11n has 88 Conv nodes and only 2 MatMul; ppe_detection has 64 Conv and zero MatMul/Gemm; scrfd_34g has 161 Conv and zero MatMul/Gemm; pose_yolov11n and object_yolo26n are the same Conv-heavy YOLO family. With no (or almost no) MatMul/Gemm nodes to touch, these graphs are essentially unquantized and their size does not meaningfully change (sometimes a few hundred bytes larger, from added quantization metadata). FaceLiVT (49 MatMul/Gemm nodes, 93 Conv) and VideoMAE (96 MatMul + 1 Gemm, essentially 1 Conv, i.e. almost entirely attention) are the two architectures with real MatMul/Gemm mass to quantize, and that is exactly where the size drops land.

The honest takeaway: if the product goal is "materially shrink the object/pose/PPE/face-detector engines," dynamic MatMul/Gemm-only quantization is not the lever for that. Static (activation-calibrated) quantization that also covers Conv, or actual pruning/distillation, would be needed, and static Conv quantization is exactly the mode this codebase already measured collapsing a real classifier's accuracy, so it would need its own from-scratch collapse-safety validation before being trusted on these detectors. That is documented future work, not something faked here.

Eval samples per model were small (6 to 16 frames or images, pulled from Dhi's own demo videos and test fixtures), enough to detect gross behavior changes from quantization but not a statistically rigorous held-out set.

Reproduce with (against a local copy of aixavier/models):

.venv/bin/python scripts/compress_production_models.py \
    --models-root /path/to/aixavier/models --output-dir output/production_compression

Provenance

• Repo: DHI-Technologies-Inc/Prompt2Model-Language-Guided-Vision-Model-Factory
• Commit: ba62c2fd721115f996812af61079ad339a8c5b80 ("fix(training): seed RNGs, stratify splits, use pretrained backbone in smoke test (#18)")
• Command: .venv/bin/python -m prompt2model.cli smoke-test --output-dir output/smoke
• Date: 2026-07-10

Provenance, production-compression section only

• Repo: DHI-Technologies-Inc/Prompt2Model-Language-Guided-Vision-Model-Factory, commit 7e9c710 ("feat(scripts): compress Dhi's production edge models with B1's gate").
• Command: scripts/compress_production_models.py --models-root /path/to/aixavier/models --output-dir output/production_compression, re-run fresh against a local copy of aixavier/models.
• Output file: b1_production_compression_summary.json (per-model size, cosine-similarity, and gate-decision report for all 8 in-scope production artifacts).
• Date: 2026-07-28.

Links

• Example dataset: Dhi-Technologies/prompt2model-examples
• Live demo: Dhi-Technologies/prompt2model-demo

License and access

Released under cc-by-nc-4.0 and gated for non-commercial research and evaluation only. No redistribution. Commercial licensing via dhi-tech.com.

Commercial licensing

Research and evaluation use is free. Production and commercial use is licensed self-serve with published prices.

• Get a license: https://dhi-tech.com/buy/prompt2model
• All published prices: https://dhi-tech.com/pricing/