Qwen3.5 Standalone 4L Classification Base

This repository publishes a headless, multilingual-input, classification-ready four-layer Qwen3.5 text backbone. The root model has no task labels and no classification head. It is a compact starting point for user-trained single-label or multilabel classifiers, not a chat model or a ready-made universal classifier.

The repository follows one integrated 1-1-1 layout:

1. Repository root: task-agnostic four-layer base trained on unlabeled general text.
2. models/semeval-propaganda/: ready-to-use SemEval-derived propaganda classifier preserved as a separate task-specific lineage.
3. distillation/: reusable 24L→8L→6L→4L platform for user-owned data.

Model at a glance

Load the headless base

from transformers import AutoModel, AutoTokenizer

model_id = "mp-juuuns/qwen35-standalone4l-classification-base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)
backbone = AutoModel.from_pretrained(model_id, trust_remote_code=False)

The published root weight file contains no score.weight. A fresh task head and labeled fine-tuning are required before outputs have class meaning.

Make your own classifier

git clone https://huggingface.co/mp-juuuns/qwen35-standalone4l-classification-base
cd qwen35-standalone4l-classification-base/distillation
python -m pip install -e .

qwen35-distill finetune \
  --checkpoint .. \
  --train examples/multilabel_train.jsonl \
  --labels examples/labels.json \
  --mode multilabel \
  --output my-classifier-4l

For a full task-specific compression run, fine-tune a 24-layer teacher, then repeat materialize-classifier and distill-classifier for 24to8, 8to6, and 6to4. Exact JSONL formats, split rules, manifests, and commands are in distillation/README.md.

How the root weights were made

The text backbone was retained from the upstream-derived 4L model. The 128k tokenizer was selected from the verified English-focused classifier lineage, and embedding rows were remapped by token string rather than by row position. This release is a vocabulary-remapped artifact, not a new training run. Students were trained sequentially through 24→8→6→4 using hidden-boundary and final-representation matching on unlabeled text.

The task-free training set was exactly 4,096 deterministically shuffled, non-empty rows from the first locally cached WikiText-103 raw training shard. No SemEval examples, labels, logits, thresholds, classification heads, or evidence/span objectives were read while training the root weights.

These are training-objective values, not downstream quality scores. Exact parameter counts, BF16 file sizes, hashes, and stage values are in benchmark/compression_ladder.csv and provenance/task_agnostic_stages/.

The chart reports the actual unsharded safetensors parameter counts and BF16 file sizes. It does not imply that downstream quality scales monotonically with layer count.

Why 128k vocabulary trimming?

The original root used 248,320 input-token rows. This update retains 128,000 token IDs and remaps each retained token to the corresponding row of the original embedding. The goal is to reduce edge storage and memory for the English-focused classification deployment path; it is not a claim that the model remains equally suitable for every language represented by the upstream tokenizer.

The remapping manifest is v128k_remap_manifest.json. It records the source hashes, token-string mapping hash, output hash, and parameter count.

Quality and efficiency evidence

The new root is headless, so its downstream quality is NR until a new task head is attached and evaluated. Existing evidence is shown for context:

• 248k task-free root transfer probe: Macro-F1 mean 0.587476 across three seeds on the previously opened SemEval-derived test.
• 128k specialized classifier lineage: Macro-F1 mean 0.587568 across three seeds on the same opened task.
• These two values are not a controlled vocabulary ablation: the models differ in task-specific head/training lineage. They must not be interpreted as proof that vocabulary trimming preserves quality.

The direct efficiency result of this release is the storage/parameter reduction above. Existing CUDA resource measurements also show the 128k specialized lineage at lower peak allocation than the 248k task-free lineage, but that comparison is confounded by the separate classifier lineage and is therefore descriptive only. Raw inputs and limitations are in benchmark/figures/v128k_quality_efficiency.csv and benchmark/figures/manifest.json.

Downstream benchmark contract

Because the root is headless, it cannot be scored as a classifier without adaptation. Its task-appropriate benchmark is therefore a controlled transfer probe:

• attach a fresh randomly initialized 14-label classification head;
• train all parameters for five epochs on the same SemEval-derived train split;
• select one global threshold on the epoch-5 calibration split;
• evaluate once per seed on the already-opened 55-article test split;
• compare against the same four-layer structural copy without task-free KD.

This derived task measures article-level presence of 14 propaganda techniques. It is not the official SemEval span- or fragment-level task. Results are exploratory because the public test split had already been opened.

Frozen protocol

Overall results by seed

Three-seed summary

Values are mean ± sample SD across the three training seeds. The SD is not a confidence interval.

The task-agnostic KD initialization improved same-seed mean Macro-F1 by +0.06939 over the structural-copy control. This is an observed comparison under this protocol, not a significance or universal-superiority claim.

Bars are three-seed means and whiskers are sample SD. The specialized 4L point is a separate historical lineage rather than a fine-tuned child of the root base.

Task-agnostic base: per-label test results

Precision, recall, and F1 are three-seed means. Support is the fixed number of positive test articles, not a three-seed sum.

Per-seed values and sample SDs for all labels and all arms are available in benchmark/per_label_metrics.csv and benchmark/per_label_summary.csv.

Labels are ordered by their three-seed mean F1. Support remains visible because several classes have only 7–14 positive test articles.

Efficiency and resource measurements

These numbers belong to the fresh-head transfer checkpoints, not a standalone headless-root latency benchmark.

The timing is specific to the recorded RTX 5070 Ti, BF16, batch-1, sliding-window protocol and includes article aggregation. It is not a claim about CPU, mobile, UNO Q, or other deployment performance. Full seed-level resource fields are in benchmark/resource_metrics.csv.

These plots compare the recorded fresh-head transfer checkpoints only. Apparent differences must not be generalized to the headless root on untested hardware.

Machine-readable benchmark bundle

• benchmark/BENCHMARK_CARD.md: generated benchmark report and interpretation
• benchmark/summary.json: complete aggregate
• benchmark/seed_metrics.csv: overall metrics per arm and seed
• benchmark/arm_summary.csv: three-seed means and sample SDs
• benchmark/per_label_metrics.csv: per-label values per arm and seed
• benchmark/per_label_summary.csv: per-label means and sample SDs
• benchmark/resource_metrics.csv: parameters, CUDA memory, wall time, and test timing
• benchmark/reports/: normalized full frozen reports, including article-level probability vectors
• benchmark/figures/: Matplotlib PNG/PDF figures, per-figure source CSVs, and a hash manifest

Multilingual scope

The root retains the upstream Qwen3.5 tokenizer and multilingual architecture. The official Qwen3.5-0.8B model card states expanded support for 201 languages and dialects and reports multilingual upstream benchmarks such as MMMLU, MMLU-ProX, NOVA-63, INCLUDE, Global PIQA, PolyMATH, WMT24++, and MAXIFE. This is why this repository carries the Hugging Face multilingual language and model tag.

However, this four-layer root was distilled on English WikiText and its only downstream transfer probe is English SemEval-derived data. Tokenizer coverage and inherited architecture support do not establish retained classification quality in all 201 languages. No multilingual downstream score is reported for this root release. Users should fine-tune and evaluate on each intended language and domain before making performance claims.

Separate SemEval model

The ready-to-use propaganda classifier is under models/semeval-propaganda/. It preserves the published seed-41/42/43 checkpoints and links to the original repository at immutable revision e7e0ff1.

That specialized model is not claimed to descend from this task-free base. It has its own SemEval task-specific shrink/distillation and fine-tuning history and a reduced 128k vocabulary. The updated root now uses the token-string- remapped 128k tokenizer documented above. Its complete reports remain under models/semeval-propaganda/benchmark/.

Limitations

• A classification head and labeled training are required before root-model predictions have task meaning.
• Task-free KD used only 4,096 rows from one English WikiText shard.
• Only one downstream task and one hardware/software setting were measured.
• The 55-article test split was previously opened; results are exploratory.
• Per-label support ranges from 7 to 45 articles and rare-label estimates are unstable.
• Three seeds describe run-to-run variation; they are not a confidence interval.
• Multilingual input support has not been validated as multilingual downstream classification quality for this four-layer root.
• Default layer maps are documented structural choices, not universal optima.
• Long-context, calibration, robustness, fairness, and production safety have not been established.
• Do not use either model as a fact checker, safety oracle, or autonomous decision maker.

License and attribution

The released code and model artifacts use Apache-2.0, subject to upstream model and data terms. Qwen3.5 is attributed to Qwen. Transformers, PyTorch, Hugging Face Hub, WikiText, and SemEval remain the work of their respective authors. No WikiText or SemEval source records are redistributed.

The WikiText page currently has a license wording discrepancy: metadata lists CC BY-SA 3.0 and GFDL, while prose says CC BY-SA 4.0. Users should inspect Salesforce/wikitext directly.