Rarestry-Moondream2-Pokemon-Grading-v0.1.0

Rarestry-Moondream2-Pokemon-Grading-v0.1.0 is an edge-optimized, fine-tuned Vision-Language Model tailored specifically for physical Pokémon trading card analysis and automated grade prediction.

Trained on synthetic multi-era trading card distributions, this checkpoint maps raw visual card crops directly into 100% schema-compliant JSON, evaluating centering ratios, corner wear, edge chipping/whitening, surface condition, and predicted PSA tiering.

The repository includes both the full merged Hugging Face safetensors weights and a 4-bit quantized CoreML MLPackage (MoondreamViT_int4) engineered to run directly on the Apple Neural Engine (ANE) within tight mobile memory budgets (sub-400 MB active footprint on devices like the iPhone SE 3).

Model Architecture & Mobile Strategy

Moondream2 utilizes a compact 0.5B-parameter dual architecture consisting of an image ViT projection network and an autoregressive language decoder. Because physical sub-millimeter measurements require extreme mathematical precision, Rarestry pairs this model with a hybrid pipeline:

                  [Raw Camera Frame / Scan]
                             │
                             ▼
                 [OpenCV Perspective Warp]
                             │
            ┌────────────────┴────────────────┐
            ▼                                 ▼
   [Deterministic Math]              [Moondream2 VLM / CoreML]
   - Sub-pixel border offsets        - Edge whitening & chipping
   - Corner radius detection         - Holographic silvering & scratches
   - Ratio bounds (50/50, 60/40)     - Surface indentation / creasing
            │                                 │
            └────────────────┬────────────────┘
                             ▼
             [Structured Grading JSON Output]

Apple Neural Engine (ANE) Optimizations (MoondreamViT_int4)

• Rank-3 Unfold Conversion: Bypasses CoreML's maximum tensor rank constraint ($rank \le 5$) by replacing raw $6\text{D}$ ViT patch slicing with rank-3 torch.nn.functional.unfold(kernel_size=14, stride=14).
• K-Means INT4 Palettization: The vision encoder is compressed down to ~140 MB, running at zero memory pressure under iOS jetsam limits (~2.2 GB ceiling).
• Target Resolution: Fixed static crop of $378 \times 378$ ($27 \times 27$ patches of size $14 \times 14$).

Target Output Schema

The model reliably outputs valid JSON without prefix conversational text or markdown code blocks:

{
  "centering": {
    "score": 9.5,
    "horizontal_ratio": "53.6/46.4",
    "vertical_ratio": "51.8/48.2",
    "psa_tier": "Gem Mint (54/46)"
  },
  "corners": {
    "score": 10.0,
    "flaws": []
  },
  "edges": {
    "score": 10.0,
    "flaws": []
  },
  "surface": {
    "score": 10.0,
    "flaws": []
  },
  "overall_grade": 9.5
}

Multi-Era Coverage

Fine-tuned and verified against card borders across Pokémon TCG history:

• Vintage Wizards of the Coast (Base Set – Gym Challenge): Yellow border symmetry, holo foil scratching, vintage cardstock whitening.
• e-Reader Era (Expedition, Aquapolis, Skyridge): Wide asymmetric bottom and left borders.
• EX, Diamond & Pearl, Platinum Era: Classic borders with micro-frame layout variance.
• Modern Silver & Scarlet / Violet Era: Low-contrast silver border ratios and micro-chipping.
• Full Art, Secret Rare, & Trainer Gallery: Borderless and ultra-rare texture grading.

Quickstart (Python / PyTorch)

import json
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "economyofdreams/Rarestry-Moondream2-Pokemon-Grading-v0.1.0"

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model.eval()

# Load and prepare cropped card image
image_path = "charizard_base_set.jpg"
image = Image.open(image_path).convert("RGB")

# Grading prompt
prompt = "Analyze Pokémon card centering and physical edges. Output structured JSON."

# Inference
with torch.no_grad():
    image_embeds = model.encode_image(image)
    raw_response = model.answer_question(image_embeds, prompt, tokenizer)

# Parse output
grading_result = json.loads(raw_response.strip())
print(json.dumps(grading_result, indent=2))

iOS Deployment (CoreML on Apple Neural Engine)

Download coreml/MoondreamViT_int4.zip from the files tab, extract MoondreamViT_int4.mlpackage, and drag it directly into your Xcode project:

import CoreML
import Vision
import UIKit

final class CardVisionPipeline {
    private let visionModel: MoondreamViT_int4

    init() {
        let configuration = MLModelConfiguration()
        configuration.computeUnits = .all // Route ViT execution directly to the ANE
        self.visionModel = try! MoondreamViT_int4(configuration: configuration)
    }

    func extractVisualEmbeddings(from pixelBuffer: CVPixelBuffer) -> MLMultiArray? {
        do {
            let input = MoondreamViT_int4Input(card_crop: pixelBuffer)
            let output = try visionModel.prediction(input: input)
            return output.visual_tokens // Shape: [1, 729, 1152]
        } catch {
            print("Vision feature extraction failed: \(error)")
            return nil
        }
    }
}

Training Details

• Base Model: vikhyatk/moondream2 (Revision 2024-08-26 / 0.5B parameters)
• Dataset Size: 330 synthetically augmented card samples mapping perspective-corrected high-resolution scans to rigorous PSA grading schemas.
• Hardware: 1x NVIDIA L4 (Google Cloud)
• Epochs: 3
• Final Converged Loss: ~0.005–0.012
• Optimizer: AdamW (LR: $2\times 10^{-5}$, Linear Warmup)

Repository Files

Ecosystem Integration

Part of the Rarestry card digitization and grading suite. Developed by TehWiz Productions.