Regulatory Alert Triage — Qwen3-4B (v1)

Reads a UK regulatory publication and returns a structured triage record: what kind of document it is, how urgent it is, which bank functions own the response, and whether it creates an obligation. It is a working first release trained on openly-licensed UK sources, published so the approach can be reviewed and built on — not a finished product.

Fine-tuned by Comply2Reg from mlx-community/Qwen3-4B-Instruct-2507-4bit with LoRA, then fused back into standalone 4-bit weights. Runs on Apple Silicon through MLX.

Read the macro-F1, not the accuracy. The test set is 54% one class, so accuracy flatters any model. The accuracy is a range because the test labels were written by a model rather than a person, and a second labeller scores it differently. Limitations below are specific and worth reading before you use this for anything.

Quickstart

Requires Apple Silicon and pip install mlx-lm.

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load("Comply2Reg/regulatory-alert-triage-qwen3-4b-v1")

SYSTEM = (
    "You triage UK financial-services regulatory publications for a bank. "
    "Return a JSON triage record with exactly these keys: alert_class (A1-A14), "
    "alert_class_confidence, priority (P1-P3), primary_functions, "
    "secondary_functions, lines_of_defence, summary, key_dates, applicability, "
    "obligations_present, thought_trace."
)
USER = """You are the regulatory-change desk of a UK bank. Read the publication and
return a triage record: alert_class, alert_class_confidence, priority,
primary_functions, secondary_functions, lines_of_defence, summary, key_dates,
applicability, obligations_present.

Regulator: CMA
Publication type: publications
Title: Proposed steering conduct requirement for Google's mobile platform
Published: 2026-08-14

<the first ~1,500 characters of the document text>"""

prompt = tokenizer.apply_chat_template(
    [{"role": "system", "content": SYSTEM}, {"role": "user", "content": USER}],
    tokenize=False,
    add_generation_prompt=True,
)
text = generate(model, tokenizer, prompt=prompt, max_tokens=400,
                sampler=make_sampler(temp=0.0), verbose=False)

# The model opens every answer with an empty reasoning block; take the JSON.
import json
record = json.loads(text[text.find("{"): text.rfind("}") + 1])

The model opens every answer with an empty <think></think> block — inherited from the base model, not from the training data, which contained none. It did so on 104/104 test documents, and the JSON that follows parsed every time. Take the substring from the first { to the last }.

Worked example

A real test document, and this model's actual output for it:

Regulator: CMA
Publication type: publications
Title: Proposed steering conduct requirement for Google's mobile platform
Published: 2026-08-14

CONSULTATION:
STEERING CONDUCT REQUIREMENT
Google's Mobile Platform
30 June 2026
...

{
  "alert_class": "A3",
  "alert_class_confidence": "high",
  "priority": "P2",
  "primary_functions": ["Product", "Legal"],
  "secondary_functions": ["Compliance", "Regulatory change", "Risk", "Board"],
  "lines_of_defence": [1, 2],
  "summary": "The CMA is consulting on a proposed steering conduct requirement for Google's mobile platform, including whether Google should provide fair and transparent access to its Play Store, app ranking, data and APIs. ...",
  "key_dates": [],
  "applicability": ["Google's mobile platform", "Play Store", "app developers", "..."],
  "obligations_present": false,
  "thought_trace": "This is a CMA consultation, so it is a proposed rule/requirement rather than final guidance or a binding instrument. ..."
}

Output schema

The label space

Alert classes

Fourteen document kinds, from the taxonomy Comply2Reg's crawler applies across UK regulators.

Train counts the fine-tuning examples; Test the held-out documents; Correct how many of those this model classified correctly.

Functions and lines of defence

The model routes each publication to the functions that own it, and reports which of the three lines of defence those sit in.

Priority

• P1: binding or imminent — legislation or final rules with an in-force or compliance date, enforcement against a firm type the reader could be, sanctions changes, reporting deadlines.
• P2: action likely but not yet binding — consultations with a closing date, supervisory guidance or letters setting expectations, thematic reviews.
• P3: awareness only — speeches, research, press releases, market notices with no direct obligation.

Intended use

Built to sit in front of a regulatory-change desk: sort an incoming feed of publications, route each to the right function, and flag the ones that carry a deadline or a duty. The structured output is meant to be reviewed, not executed.

Out of scope. This is not legal advice and not a compliance decision. It does not replace a compliance officer's reading of the source document. It is not calibrated for non-UK regulators, for material published before 2022, or for the FCA and PRA — whose publications were deliberately excluded from training, as explained below. Nothing it produces carries any official or endorsed status.

Training data

1,683 training, 218 validation and 104 test documents, drawn from a corpus of 6,642 UK regulatory documents published since 2022-01-01 that Comply2Reg's crawler collected and Comply2Reg labelled.

Only openly-licensed sources were used: HM Treasury, the Competition and Markets Authority, the Information Commissioner's Office and legislation.gov.uk, all published under the Open Government Licence v3.0. Publications from the FCA, PRA, Bank of England, Ofcom and the DRCF sit under terms that do not clearly permit this use, so they were held back pending a licensing review — which is the single largest constraint on the model's quality, and the reason several classes below have almost no training data.

Labels. Every document was labelled by OpenAI's gpt-5.4-mini against the taxonomy, one call per document, cached by content hash. A second model labelled the same corpus and the disagreements were sampled against a stronger judge, which sided with the chosen labeller in 18 of 20 cases. The labels are therefore model-generated and consistent, not human-adjudicated, and some are certainly wrong.

Preparation. Documents were cleaned of page furniture (running headers, page numbers, tables of contents, boilerplate addresses) before anything else, so stored offsets and hashes refer to the cleaned text — 1,863,690 characters removed across 5,812 documents. Exact duplicates were removed by content hash (394) and near-duplicates by MinHash within a regulator and document type (185).

Splits are chronological, not random. The test set is the latest 2 months (2026-07-01 to 2026-09-18); validation is the next slice back; training runs from 1948-01-01 to 2026-07-02. So the reported scores measure generalisation forward in time, which is the way the model is actually used.

Stratification. Training aimed at a fixed mix of document kinds and fell short on three of them, because the open-licence sources do not publish much guidance or enforcement material:

Input format. The model sees the regulator, publication type, title, date and the first ~1,500 characters of the document — not the full text. Median training prompt 820 tokens, 95th percentile 1034, budget 2048, nothing truncated.

The dataset is not published. The pipeline that builds it is open, in the repository linked below.

Training procedure

LoRA on 16 of 36 layers, rank 8, scale 20.0, loss on the assistant turn only. That trains 7.340M of 4022.468M (0.182%) of the model's parameters.

Validation loss at iterations 1, 300, 600, 900, 1200, 1500, 1700: 3.075 → 0.961 → 0.936 → 0.860 → 0.748 → 0.873 → 0.805.

The adapter was then fused into the base weights with mlx_lm fuse, with no de-quantisation: the adapter was trained against the 4-bit weights, so those are the weights it should be measured on.

Fusing is not lossless at 4 bits. It dequantizes each layer, adds the LoRA delta and re-quantizes, so the published model is a numerically different object from base-plus-adapter. Measured on the same test set, the two agree on 93 of 104 documents (89.4%), the fused model scoring 0.644 against 0.635. The differences fall entirely among the classes the model never learned; every figure reported here was measured on the published weights, not inherited from the adapter.

Evaluation

104 held-out documents, greedy decoding, scored field by field.

Two things in that table are worth stating plainly.

Fine-tuning bought the format. The untuned base model produced no parseable triage record at all on a 10-document probe — it answers in prose. Everything downstream depends on the JSON being there, so this is the change that makes the model usable.

Lower validation loss did not mean a better model. The iteration-1200 checkpoint had the best validation loss of the run (0.748 against 0.805 at the end) and scored worse on every task metric, because it had collapsed toward predicting the majority class. The final adapter was selected on task metrics, not loss.

How much of that is label noise

The test labels were written by a model, not a person, so the figures above measure agreement with one labeller rather than correctness. All 104 test documents were re-labelled by a stronger model and the same saved predictions were scored again.

And the two labellers against each other, on the same documents:

So the accuracy is not "64% correct". Two capable models, given the same taxonomy and the same documents, agree on the alert class only 70.2% of the time. That is roughly the ceiling any model can reach against either of them, because a third of the remaining disagreement is in the labels rather than in the answer. Read the headline as a band between 0.567 and 0.644 against a ceiling near 0.70.

Three fields scored higher under the stronger labeller, not lower: priority, obligations and especially function assignment. On those the model generalised past the teacher it was trained on. Only class assignment is anchored to the training labeller's idiosyncrasies, which is what one would expect of the field with the largest label space and the noisiest labels.

Fixing this properly needs human adjudication, which has not been done.

Benchmarks

No public benchmark scores regulatory alert triage, so none of these measure the task. What they measure is whether fine-tuning preserved the general legal and structured-output ability the base model had. Each task was run on this model and on the untouched base model, over the same seeded sample, so the change is the result and the absolute numbers should not be quoted as leaderboard scores.

Tasks: jsonschema, ledgar, obliqa, unfair_tos.

The model answers every prompt with a triage record, which costs it marks wherever a task wants a different shape. That is a real property, not a scoring artefact, so it is measured rather than prompted away: where a task has a fixed label set, the scorer looks inside any returned JSON for a label before counting the answer wrong, and reports how often a usable answer appeared at all.

Limitations

It has not learned most of the taxonomy. Macro-F1 is 0.244. Of the classes present in the test set, the model gets a non-zero score on 4. Classes with roughly twenty training examples — A9, A13, A14 — score zero. Four further classes have one or two test documents each, too few to say anything about.

One class dominates. A11 is 54% of the test set and 40% of training. An always-A11 baseline scores 0.538 accuracy, so the +0.106 the model adds is the honest measure of the accuracy figure.

Train and test come from different regulators. Training is mostly HM Treasury and legislation.gov.uk; the test window is mostly CMA. That makes the evaluation harder than the training distribution, and it means the scores say little about regulators the model barely saw.

It is not really a financial-services model yet. The open-licence sources are cross-sector: Treasury policy, competition cases, data protection, statutory instruments. The FCA and PRA conduct and prudential material a UK bank most cares about is exactly what is missing.

The labels are model-generated. No human adjudicated them. Errors in the labeller are reproduced in the model, and the evaluation shares that bias because its labels came from the same source.

No calibration. alert_class_confidence is what the labelling model said and the fine-tune imitated. It has not been checked against observed accuracy — do not threshold on it.

It fails silently on regulation it was not trained on. Given 198 passages from a non-UK financial regulator's rulebook, the model produced well-formed triage JSON 99% of the time and then assigned A11 to 198 of 198 of them, flagging an obligation in 0.5%. That class means "intelligence: speech, blog, press release"; the passages are rulebook text and obligation-bearing by construction. The summaries were often a correct reading of the text while every structured field contradicted them, so nothing downstream could detect the failure from the output alone. Outside UK open-licence sources, evaluate before trusting it.

Fine-tuning cost general legal ability. On a 100-class contract-provision benchmark the base model scores 0.610 and this model 0.418. Refusing to answer in the requested format explains part of that, but not all: counting only the items where both produced a label, accuracy still falls by 0.128. This is the known cost of LoRA fine-tuning on a single narrow task without replay data.

Licence, attribution and provenance

The weights are released under Apache 2.0, inherited from Qwen3-4B-Instruct-2507.

Training text came from UK public bodies under the Open Government Licence v3.0:

Contains public sector information licensed under the Open Government Licence v3.0.

The Open Government Licence requires that the use not suggest official status or endorsement. This model is not endorsed by, affiliated with or connected to any UK regulator or government body, and its output is not an official interpretation of anything.

Training labels were generated with OpenAI models under terms that assign the output to the customer.

Citation

@misc{comply2reg_regulatory_alert_triage_qwen3_4b_v1,
  title  = {Regulatory Alert Triage — Qwen3-4B (v1)},
  author = {Comply2Reg},
  year   = {2026},
  url    = {https://huggingface.co/Comply2Reg/regulatory-alert-triage-qwen3-4b-v1}
}

Changelog

v1 — 2026-09-25. First release. GREEN-tier (Open Government Licence) sources only, 1,683 training documents, LoRA fused into 4-bit weights. Known gap: no FCA or PRA material, and seven alert classes with too little data to learn.

Built by Comply2Reg. Pipeline and training code: ComplianceGPT.