Llama-3.2-1B-DelaySentinel

Building and evaluating AI on logistics data. Built with Llama.

This checkpoint is a full-parameter fine-tune of meta-llama/Llama-3.2-1B-Instruct, published with the data split, evaluation results, and behavioral tests used to inspect it. The project connects a working training-and-inference pipeline with a retrospective audit of what its score demonstrates.

The checkpoint and a depth-2 decision tree both score 100% on the same historical 200-row split. A rule using two supplied fields reproduces every label in the 1,000-row source table. Prompt rewrites then show what moves the answer: an appended note about "Mr. Delayed" flips 84 of 84 negatives, and Early and Light, which contain neither rule value, keep every row at 1. These findings make the checkpoint useful for studying evaluation and label leakage.

GitHub project · Case study · Full evaluation · Frozen dataset · Reproduce

Model details

The frozen split came from an unseeded shuffle and is verified against recorded hashes; see split provenance.

Evaluation summary

Source: results/eval.json, model.test_metrics_greedy and baselines_split_v0. The target satisfies Shipment_Status == "Delayed" OR Traffic_Status == "Heavy" on all 1,000 source rows, and the four predictors that see those two fields all score 1.000. Without them, gradient boosting falls to 0.500 accuracy and 0.452 AUROC. The table measures agreement with that dataset label.

The project's leakage scanner recovers the rule from the raw table and runs on any CSV with a 0/1 target column: python -m delaysentinel.leakage_audit --csv <your.csv> --target <binary column>.

What changes the answer

Appending "Note: the depot supervisor is Mr. Delayed" to the 84 negative prompts, with every logistics field unchanged, flips 84 of 84 answers to 1. The same note with "Mr. Heavy" flips 0 of 84. Counterfactual field edits are exact: 261 of 261 targeted rule-field edits flip the answer as the dataset rule predicts, and 3,200 rewrite attempts on the other 13 fields change no prediction.

Rewriting the rule values themselves gives the sharpest contrast. Not Delayed, Late and Early each keep 73 of 73 Delayed rows at 1, and Not Heavy and Light keep 66 of 66 Heavy rows at 1. Postponed and all 12 unrelated control words leave 23 of 73 at 1, the rows whose Heavy traffic satisfies the other rule clause. In the traffic field, Congested and 11 of the same 12 control words leave 23 of 66, the rows that are also Delayed; Meadow is the lone exception at 33 of 66. Late, Early and Light contain neither rule value, yet every row still returns 1. No single string explains that set: what the checkpoint actually responds to is the project's open question. All 104 probe sets are in the complete probe tables; the rewrite sets all draw on the same 200 historical prompts.

A separate study on real orders

The separate Olist study takes the same evaluation questions to real orders with classical models: logistic regression on checkout-time features reaches 0.6908 AUROC on 37,702 later orders, where the late-delivery rate is 0.0722.

Intended use and scope

Use this checkpoint to reproduce a label-leakage case study, inspect behavior under controlled input rewrites, or teach baseline comparison and evaluation design. The repository provides recorded outputs and executable checks for these purposes.

It has not been validated for operational delay forecasting, risk scoring, or dispatch decisions. It provides hard labels rather than calibrated risk estimates. Tested invalid or incomplete inputs can still receive a label instead of an abstention. Examples and probes are specific to this checkpoint and prompt format.

How to use

The example reproduces an original-format response. Decode greedily, pin the template date, and parse with an anchored regex; treat a non-match as unparsable, never as 0.

# Run end to end with transformers 4.55.4 and torch 2.8.0, the evaluation's pinned versions.
# return_dict=True also matches the transformers 5.x default, where apply_chat_template
# returns a BatchEncoding.
import re
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "Yuchiwang02/Llama-3.2-1B-DelaySentinel"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype=torch.bfloat16, device_map="auto")

SYSTEM = ("Assume you are a supply chain analyst. Based on the following information, "
          "output the result for Logistics_Delay, where 1 represents a delay and 0 represents no delay.")
# First row of data/test.jsonl (gold label 0). Same 15 columns, same order, as in training.
USER = """Timestamp: 2024-07-18 12:19:20
Asset_ID: Truck_10
Latitude: 20.2969
Longitude: 124.1885
Inventory_Level: 298
Shipment_Status: In Transit
Temperature: 18.6
Humidity: 74.4
Traffic_Status: Detour
Waiting_Time: 41
User_Transaction_Amount: 248
User_Purchase_Frequency: 1
Logistics_Delay_Reason: None
Asset_Utilization: 86.3
Demand_Forecast: 283"""

messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": USER}]
enc = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_dict=True, return_tensors="pt", date_string="04 Sep 2025"
).to(model.device)
out = model.generate(
    **enc,
    do_sample=False,
    max_new_tokens=8,
    pad_token_id=tok.convert_tokens_to_ids("<|finetune_right_pad_id|>"),  # never add a pad token / resize embeddings
)
text = tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True).strip()
m = re.match(r"^\s*Logistics_Delay:\s*([01])\b", text)
label = int(m.group(1)) if m else None  # None = unparsable
print(text, label)                      # -> Logistics_Delay: 0  0

Or, with the package: from delaysentinel.model import Scorer and Scorer(REPO).predict([USER]).

Training and data

The source is Kaggle's Smart Logistics Supply Chain Dataset by ziya07, listed as CC0 when recorded by the project. It contains 1,000 timestamped records, 15 supplied fields, and the target. A deterministic target rule and geographically dispersed coordinates are consistent with a constructed table, and the Kaggle page documents no collection process.

The original run used 30 epochs, 1,500 steps, bf16, batch size 4 with gradient accumulation 4, learning rate 2e-5, cosine scheduling, and assistant-only loss. The logged training loss first rounds to 0.0000 at step 50. Reading the upstream trainer explained the learning-rate log: learning_rate_overshoot = 1.15 plans a 1,724-step cosine, which reproduces all 300 logged learning rates, so when training ends at step 1,500 the rate is still 8.3e-07.

Training used an adaptation of acon96/home-llm with three local changes and an attribution header. The published weights are run sc904: the local sha256 equals the Hub LFS hash. Recovered settings and preserved artifacts are described in run provenance and the technical report. The script is available in the GitHub source.

Limitations and open questions

• Open question: what does the checkpoint respond to? The field name Logistics_Delay_Reason carries "Delay" in all 200 prompts and never triggers anything. Two experiments would separate the candidate explanations: the same rewrites on the base Llama-3.2-1B-Instruct, and ranking the tested values by embedding similarity to Delayed and Heavy. What changes the answer has the full contrast.
• The source label is reconstructible from supplied inputs, and the table carries no promised or actual delivery dates.
• With no pad token defined, the historical collator padded with <|eot_id|> and masked it, which also hid the genuine system and user end-of-turn tokens from attention. Its effect on the trained behavior is open; a retrained comparison would measure it.
• The earliest surviving checkpoint is step 1,300 of 1,500, so when the final behavior arose during training is open.
• Recorded bf16 margins can vary in their final digits across batch composition and environments.

The evaluation report contains the complete methods, tables, probe conditions, and evidence boundaries. The reproduction guide covers CPU checks, full model execution, and the separate Olist study.

Attribution and licenses

Built with Llama. The weights are subject to the Llama 3.2 Community License and Acceptable Use Policy, with attribution in NOTICE. The full model name is Llama-3.2-1B-DelaySentinel; the original Hub name redirects to this repository.

The training script derives from acon96/home-llm under MIT, with Stanford Alpaca portions under Apache-2.0. The preserved copy has three local changes relative to the identified upstream revision; see third-party attribution and the Apache license.

Project code and documentation are distributed under MIT. The Olist aggregate result and the figure drawn from it are distributed under CC BY-NC-SA 4.0. Implementation and documentation use AI assistance; linked source records and the attribution file describe the project materials and reused work.

Citation

@misc{wang2026delaysentinel,
  title   = {Llama-3.2-1B-DelaySentinel: Building and evaluating AI on logistics data},
  author  = {Wang, Yuchi},
  year    = {2026},
  version = {1.0.3},
  note    = {Weights trained September 2025; retrospective evaluation and behavioral audit added September 2026},
  url     = {https://huggingface.co/Yuchiwang02/Llama-3.2-1B-DelaySentinel}
}

中文案例 · Related project: BizHallu