Qwen3.5-0.8B Medical — First 5 MLP Layers
Model Description
This repository contains an experimental medical-domain adaptation based on the Qwen3.5-0.8B model family.
The experiment focuses specifically on adaptation involving the first five MLP layers of the model architecture. It was created as part of an investigation into how targeted adaptation of a limited subset of transformer components affects medical reasoning and language-model behavior.
The model is intended primarily for:
- research into parameter-efficient or targeted model adaptation;
- medical-language-model experimentation;
- layer-level and module-level ablation studies;
- comparison of adaptation strategies across transformer layers;
- evaluation of medical reasoning behavior;
- investigation of compute-efficient fine-tuning techniques.
This repository should be treated as a research artifact rather than a clinical model.
Repository
Repository: adityag-india/qwen3p5-0p8b-medical-mlp-first5-layers
Maintainer: adityag-india
Experimental Objective
Modern transformer models contain multiple attention and feed-forward/MLP components distributed throughout their layers.
Instead of treating every model layer identically, this experiment investigates whether concentrating adaptation on a small, predefined portion of the model can produce useful domain specialization.
The specific focus of this repository is the:
First 5 MLP layers
The broader research question is whether early MLP-layer adaptation contributes meaningfully to medical-domain specialization and how its behavior compares with models adapted using different layer selections.
This repository may therefore be useful alongside complementary experiments such as:
- last-N-layer MLP adaptation;
- full-MLP adaptation;
- attention-module adaptation;
- standard LoRA/QLoRA configurations;
- frozen-layer experiments;
- module-ablation experiments;
- full-model or broader parameter-efficient fine-tuning.
Training Domain
The model was adapted using medical reasoning data derived from:
FreedomIntelligence/medical-o1-reasoning-SFT
The dataset contains medical reasoning examples intended for supervised fine-tuning and research into medical reasoning capabilities.
The dataset authors describe it as a medical reasoning SFT dataset built using medical verifiable problems and reasoning traces.
Dataset
Hugging Face:
FreedomIntelligence/medical-o1-reasoning-SFT
The dataset supports medical question-answering and text-generation research and contains English and Chinese material.
Dataset license: Apache-2.0
Dataset Credit
All credit for creation and release of the underlying medical reasoning dataset belongs to the FreedomIntelligence authors and contributors.
This model repository does not claim ownership of the original dataset.
Researchers using this model should independently review the dataset card, licensing conditions, provenance, limitations, and applicable terms before redistribution or downstream use.
Dataset Citation
If this model or its training data contributes to academic or research work, please cite the original dataset/research work associated with the Medical-O1 project.
@misc{chen2024huatuogpto1medicalcomplexreasoning,
title={HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs},
author={Junying Chen and Zhenyang Cai and Ke Ji and Xidong Wang and Wanlong Liu and Rongsheng Wang and Jianye Hou and Benyou Wang},
year={2024},
eprint={2412.18925},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2412.18925},
}
Please verify the most current citation information from the original dataset or associated paper before publication.
arXiv: 2412.18925
Dataset: FreedomIntelligence/medical-o1-reasoning-SFT
Intended Use
This model is intended for research and experimental purposes.
Potential appropriate uses include:
- studying medical-domain language-model adaptation;
- comparing layer-specific fine-tuning approaches;
- studying representations learned by early transformer MLP layers;
- benchmarking medical reasoning datasets;
- evaluating parameter-efficient adaptation approaches;
- studying catastrophic forgetting or specialization;
- comparing first-layer and last-layer adaptation;
- investigating inference behavior after targeted fine-tuning;
- educational demonstrations of LLM fine-tuning;
- reproducibility studies.
Out-of-Scope Uses
This model is not intended for autonomous clinical use.
It should not be used as the sole or authoritative source for:
- medical diagnosis;
- treatment recommendations;
- medication selection;
- medication dosage;
- emergency medical decisions;
- triage;
- interpretation of laboratory results;
- interpretation of diagnostic imaging;
- surgical decisions;
- patient-specific clinical management;
- replacing physicians, pharmacists, nurses, or other qualified healthcare professionals.
It should also not be deployed in a healthcare workflow without appropriate validation, governance, human oversight, regulatory assessment, and safety testing.
Medical Disclaimer
This model is not a medical device and is not a substitute for professional medical advice, diagnosis, or treatment.
Outputs generated by this model may be:
- factually incorrect;
- incomplete;
- misleading;
- outdated;
- internally inconsistent;
- medically unsafe;
- overconfident;
- unsupported by clinical evidence.
Language models can generate plausible-sounding medical statements even when those statements are incorrect.
No output from this model should be interpreted as establishing a diagnosis, recommending treatment, prescribing medication, determining medication dosage, or providing individualized medical advice.
Any medical information produced by this model should be independently reviewed by an appropriately qualified healthcare professional before being relied upon.
For medical emergencies, users should contact the appropriate emergency medical service or qualified healthcare provider rather than relying on this model.
Usage Disclaimer
This repository is provided for research, experimentation, benchmarking, and educational purposes.
The repository maintainer makes no representation or warranty that the model is:
- clinically accurate;
- medically validated;
- fit for a particular purpose;
- safe for production healthcare use;
- compliant with healthcare regulations;
- suitable for decision-making involving patients.
Users are responsible for determining whether their intended use complies with applicable laws, regulations, licenses, institutional policies, safety requirements, and ethical standards.
Users assume responsibility for evaluating model outputs and any consequences resulting from downstream use.
Research Status
This is an experimental model.
The presence of medical-domain fine-tuning does not imply clinical competence.
Performance on a training or evaluation dataset does not establish:
- clinical validity;
- generalization to real patients;
- robustness across medical specialties;
- robustness across demographic populations;
- factual reliability;
- regulatory approval;
- safety in real-world healthcare environments.
Limitations
Important limitations may include:
Hallucination
The model can generate information that appears credible but is factually incorrect.
Dataset Bias
The model can inherit biases, inaccuracies, omissions, formatting patterns, or other limitations present in its training data.
Domain Generalization
Performance on medical reasoning examples does not guarantee equivalent performance on real-world patient cases.
Knowledge Limitations
The model's underlying knowledge may not contain recent clinical guidelines, recently approved medications, updated contraindications, new research findings, or changes to medical standards of care.
Small Model Capacity
The approximately 0.8B-parameter model size provides computational advantages but may also limit reasoning depth, factual retention, robustness, and generalization relative to larger models.
Layer-Restricted Adaptation
This experiment intentionally concentrates adaptation on a restricted subset of the model architecture.
Consequently, its behavior should not be assumed to match:
- full fine-tuning;
- full-model LoRA;
- adaptation of all MLP layers;
- adaptation of attention modules;
- adaptation of later transformer layers.
This architectural restriction is part of the experiment itself.
Recommended Evaluation
Before drawing conclusions from this model, researchers should consider evaluating at least:
- validation loss;
- test loss;
- perplexity;
- token accuracy;
- reasoning-token accuracy;
- answer-token accuracy;
- factual medical QA accuracy;
- hallucination rate;
- calibration;
- robustness to prompt variations;
- base-model versus adapted-model performance;
- first-5-layer versus last-5-layer performance;
- VRAM usage;
- inference memory requirements;
- training compute;
- trainable parameter count.
For medical-domain research, automated metrics should ideally be supplemented by expert review where feasible.
Comparison Guidance
Because this repository represents a targeted architectural experiment, its results are most meaningful when compared against controlled baselines using the same:
- base model;
- dataset;
- train/validation/test split;
- random seed;
- sequence length;
- optimizer;
- learning-rate schedule;
- evaluation implementation;
- decoding strategy;
- hardware class where relevant.
Changing several experimental variables simultaneously can make attribution difficult.
Example Loading
The exact loading procedure depends on whether the repository contains a complete model or parameter-efficient adapter weights.
For a standard Transformers-compatible full model, usage may resemble:
from transformers import AutoTokenizer, AutoModelForCausalLM
repo_id = "adityag-india/qwen3p5-0p8b-medical-mlp-first5-layers"
tokenizer = AutoTokenizer.from_pretrained(
repo_id,
trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
device_map="auto",
torch_dtype="auto",
trust_remote_code=True,
)
If the repository contains PEFT/LoRA adapter weights rather than a complete standalone model, load the original compatible base model first and then attach the adapter using PEFT.
Example:
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model_id = "Qwen/Qwen3.5-0.8B"
adapter_id = "adityag-india/qwen3p5-0p8b-medical-mlp-first5-layers"
tokenizer = AutoTokenizer.from_pretrained(
base_model_id,
trust_remote_code=True,
)
base_model = AutoModelForCausalLM.from_pretrained(
base_model_id,
device_map="auto",
torch_dtype="auto",
trust_remote_code=True,
)
model = PeftModel.from_pretrained(
base_model,
adapter_id,
)
Users should inspect the repository configuration files before choosing the appropriate loading approach.
Responsible Use
Researchers and developers using this model should:
- Clearly communicate that generated medical information comes from an experimental AI system.
- Avoid presenting model output as authoritative clinical advice.
- Use qualified human review for consequential healthcare applications.
- Evaluate hallucinations and factual accuracy independently.
- Protect patient privacy and avoid submitting identifiable patient information without appropriate safeguards.
- Follow applicable healthcare, privacy, research, and AI regulations.
- Document model version, configuration, evaluation procedure, and limitations when reporting results.
Privacy Considerations
Users should avoid supplying protected, confidential, or personally identifiable medical information unless they have appropriate authorization and a compliant processing environment.
Nothing about this repository by itself establishes compliance with healthcare privacy frameworks or regulations.
Reproducibility
For reproducible research, downstream users are encouraged to record:
- repository revision/commit;
- base-model revision;
- dataset revision;
- dataset split;
- random seed;
- training configuration;
- LoRA/PEFT configuration, if applicable;
- optimizer;
- learning rate;
- scheduler;
- batch size;
- gradient accumulation;
- maximum sequence length;
- number of epochs;
- evaluation methodology;
- software-library versions;
- GPU hardware.
Licensing
Users should separately review and comply with:
- the license governing the original Qwen base model;
- the license governing the FreedomIntelligence dataset;
- licenses covering any additional software or dependencies used with this repository.
The source medical reasoning dataset is identified by Hugging Face as Apache-2.0 licensed.
Redistribution of adapted model weights does not automatically supersede conditions associated with the original base model or source datasets.
Acknowledgements
This experiment builds upon work from:
Qwen / Alibaba Cloud — for development and release of the underlying Qwen model family.
FreedomIntelligence — for creating and publicly releasing the medical-o1-reasoning-SFT dataset and associated medical reasoning research.
Hugging Face — for providing the model and dataset hosting ecosystem and open-source machine-learning tooling used throughout the broader experimentation workflow.
Citation and Attribution
When using this repository in research, please provide appropriate attribution to:
- the original Qwen base model and its authors;
- the FreedomIntelligence Medical-O1 dataset and associated research;
- this adapted model repository where relevant.
Repository:
adityag-india/qwen3p5-0p8b-medical-mlp-first5-layers
Dataset:
FreedomIntelligence/medical-o1-reasoning-SFT
Associated dataset/research reference:
arXiv:2412.18925
Final Notice
This repository is an experimental medical-language-model research artifact. It has not been clinically validated, should not be considered a medical device, and must not be relied upon for patient diagnosis, treatment, medication, emergency decisions, or other clinical decision-making.