Qiushi-Engine-Frontier-Advancement
Full research project on GitHub · Oxelra AI · Questions & feedback
Explore the English and Chinese reports, methods, training and evaluation code, controlled experiments, results and research notes in the GitHub project. We welcome questions, independent replication and discussion of the findings.
Stage I model in Qiushi Engine's BabyLM Strict-Small research program.
Frontier Advancement -> Principle Discovery -> Principle-Guided Frontier Advancement.
This English masked language model pairs source passages with concise rewrites, allowing more source passages within a fixed 10-million-word corpus. It uses a DeBERTa-v2 encoder with residual adapters: small trainable layers added to the encoder. A second adapter is added after pretraining and trained during a short continuation.
The two releases connect three research stages: training a limited-data language model, studying how it learns from paired texts and retains learned behavior, and using those findings to develop a continuation method tested against matched controls.
Companion model: Qiushi-Engine-Principle-Guided-Frontier-Advancement.
Model
| Property | Value |
|---|---|
| Architecture | DeBERTa-v2 with two sequential residual adapter paths |
| Parameters, including masked-language-model head | 36,458,592 |
| Encoder parameters | 36,210,368 |
| Second adapter parameters | 995,584 |
| Layers / hidden size / attention heads | 8 / 480 / 8 |
| Adapter bottleneck / scales | 128 / 1.75 and 0.75 |
| Tokenizer | 16,384-entry byte-level BPE |
| Corpus budget | 10,000,000 counted words in 64,740 rows |
| Recorded cumulative word exposure | 86,005,295 |
| Local full-evaluation Overall | 42.0240 |
Use
from transformers import AutoModelForMaskedLM, AutoModel, AutoTokenizer
repo = "leslie721007/Qiushi-Engine-Frontier-Advancement"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True)
encoder = AutoModel.from_pretrained(repo, trust_remote_code=True)
The custom code is required: both adapter paths are part of the model. The encoder
entry point supports downstream fine-tuning. The included requirements.txt records
the tested model-loading environment. For a fixed experiment, use the same Hub commit
for tokenizer, model and encoder.
Full Evaluation
Local results from the full BabyLM evaluation. SuperGLUE uses the custom encoder with both adapters and fine-tuning seed 42.
| Component | Score |
|---|---|
| BLiMP | 68.5100 |
| Supplement | 63.6400 |
| EWoK | 50.0200 |
| Entity | 28.3200 |
| COMPS | 52.0500 |
| SuperGLUE | 68.9457 |
| GlobalPIQA | 38.5650 |
| Reading | 8.1650 |
| AoA | 0.0000 |
| Overall | 42.0240 |
Overall is the arithmetic mean of the nine components. Age of Acquisition (AoA) was evaluated across 18 checkpoints and scored 0.
Research Comparison
| Method | Continuation seed 62064 | Continuation seed 62065 |
|---|---|---|
| Stage I parent | 42.0240 | 42.0240 |
| Ordinary continuation | 42.0926 | 42.1159 |
| Dense input masking, sparse target supervision | 42.2025 | 42.1789 |
| With ordinary-input preservation | 42.2464 | 42.2317 |
Both continuation runs start from the same Stage I model. The final method improves Overall over ordinary continuation in each run; individual task scores vary. METHOD.md explains the training changes and what the comparisons measure.
Artifacts
EVALUATION.json: exact local component scores and model identity.METHOD.md,TRAINING.md,DATA.md: design, recorded training and data provenance.CHECKPOINTS.json: shared intermediate checkpoints and actual exposure counts.evaluation/: complete final prediction files, AoA measurements and Fast metrics.submission/: full and full-plus-Fast official-format prediction bundles.VALIDATION.json: package loading and save/reload tests.
Intended uses are masked-token prediction, sentence scoring and encoder fine-tuning. The experiments cover English language modeling under the BabyLM Strict-Small budget.
License
Model weights and original code: Apache-2.0. Data sources retain their original licenses; see DATA.md.