Latent tool-memory: trained probes and surfacing triggers (Qwen3-8B)
Trained readout probes and memory-surfacing triggers from the five-experiment
latent tool-memory line (MATS). All artifacts read
Qwen/Qwen3-8B residual-stream hidden states. Code and usage live in
Sid-MB/mats-gf-latent-tool-memory;
the training/eval data is in
siddharthmb/mats-gf-latent-tool-memory-data.
Hidden-state convention throughout: hidden_states index = layer, with 0 =
embeddings; features are the residual at the stated layer, L2-normalized
unless a config says otherwise.
exp2/probes/ — pre-generation tool-selection readout (experiment 2)
Trained on the 840-prompt BFCL v3 manifest (exp2/ in the data repo),
features = unit-normalized last-prompt-token residuals.
| File | What it is |
|---|---|
probe_a_best.joblib | tool-needed logistic head, layer 28 (held-out AUROC 1.0) |
probe_b_slot_best.joblib | which-slot logistic head, layer 29 (0.717 vs 0.417 TF-IDF) |
probe_b_retrieval_best.joblib | ridge map to centered tool embeddings (unseen-tool retrieval; bounded at its shuffled floor) |
steering_directions_layer29.pt | slot-pair difference-of-means directions, keys "i->j" |
Load the .joblib heads with joblib.load; they are scikit-learn estimators.
exp4/ — surfacing triggers, sensor-fleet task (experiment 4)
Decode-time triggers deciding when to surface a memory-store lookup. Features: L2-normalized layer-20 residual at each decode step.
triggers/— need-labeled probe (probe.npz:w [4096],b,layer), FLARE logprob threshold (flare.json), adaptive classifier (adaptive.npz), P′ identity readout (pprime.npz; reads at chance, a recorded negative), chosen operating points (chosen.json).triggers_p2/— the retrieval-readiness probe (same file layout;chosen.jsonhas τ=0.75): fires when the generated text can name the needed store entry. Beats FLARE on every slice (0.741 vs 0.598 overall).triggers_km/— text-side key-match trigger:km_config.json(embedder, window, θ) +km_clf.npz(logistic on[top1, margin, delta3, wlen]).
import numpy as np
d = np.load("triggers_p2/probe.npz") # w, b, layer (=20)
score = 1 / (1 + np.exp(-(h20_norm @ d["w"] + d["b"]))) # fire if > tau
exp5/triggers/ — in-domain retrained triggers, BFCL v3 multi-turn (experiment 5)
The frozen experiment-4 probe transfers at chance (AUROC 0.4996); these are the cheap in-domain retrains. Same layer-20 feature convention.
probe.npz+probe_chosen.json— retrained readiness probe (τ=0.998, dev event-F1 0.634; test P 0.66 / R 0.58; trained on 359 episodes).probe_frac10.npz/probe_frac25.npz(+_chosen.json) — label-efficiency variants trained on 10% / 25% of episodes.km_clf.npz+km_chosen.json— retrained key-match classifier.flare.json,retau.json— FLARE threshold and the dev-recalibrated frozen-probe threshold (both recorded baselines).labels.parquet,summary.json,chosen.json— training labels and selection records.
Not included: the 49 GB of training-time step-table caches and floor-rollout intermediates behind these files, and all raw activation captures — not published; regenerable from the episodes + model with the repo code, or available on request.
Provenance
Base model Qwen/Qwen3-8B (Apache 2.0); data from BFCL v3 (Apache 2.0).