Llama-4-Scout-1.7B-0.4B-Instruct
This is a tiny version of meta-llama/Llama-4-Scout-17B-16E-Instruct created for testing and development.
Model Details
- Base Model: meta-llama/Llama-4-Scout-17B-16E-Instruct
- Architecture: llama4 (multimodal vision-language with MoE)
- Total Parameters: 1.72B
- Activated Parameters: ~0.43B (1 expert activated per token out of 4)
Configuration Changes
The following parameters were reduced from the original model:
| Parameter | Original | Tiny |
|---|---|---|
| Text Model | ||
| num_hidden_layers | 48 | 8 |
| num_local_experts | 16 | 4 |
| num_experts_per_tok | 1 | 1 |
| hidden_size | 5120 | 2048 |
| intermediate_size | 8192 | 3072 |
| intermediate_size_mlp | 16384 | 6144 |
| num_attention_heads | 40 | 16 |
| num_key_value_heads | 8 | 4 |
| layer_types | 48 layers (chunked/full pattern) | 8 layers (maintains 3:1 pattern) |
| Vision Model | ||
| num_hidden_layers | 34 | 6 |
| hidden_size | 1408 | 768 |
| intermediate_size | 5632 | 3072 |
| num_attention_heads | 16 | 12 |
Architecture Preservation
The tiny model maintains the original Llama-4-Scout architecture patterns:
- MoE Structure: Retained mixture-of-experts with shared expert
- Attention Pattern: Maintains the chunked_attention/full_attention pattern (every 4th layer is full_attention)
- No-RoPE Layers: Preserved the pattern where 3 out of every 4 layers use alternative position encoding
Checkpoint Structure
The model is saved as a single safetensors file following the original checkpoint structure:
language_model.model.layers.{X}.feed_forward.experts.*language_model.model.layers.{X}.feed_forward.shared_expert.*vision_model.model.layers.{X}.*
This structure is compatible with transformers' Llama4ForConditionalGeneration.
Usage
from transformers import Llama4ForConditionalGeneration, AutoProcessor
model = Llama4ForConditionalGeneration.from_pretrained(
"inference-optimization/Llama-4-Scout-1.7B-0.4B-Instruct",
device_map="auto"
)
processor = AutoProcessor.from_pretrained("inference-optimization/Llama-4-Scout-1.7B-0.4B-Instruct")
# Text-only input
text = "Hello, world!"
inputs = processor.tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=20)
print(processor.tokenizer.decode(outputs[0]))
Creation Process
This model was created using the llm-compressor create-tiny-model skill:
- Config Modification: Reduced layers, experts, and hidden dimensions while preserving architectural patterns
- Weight Initialization: Randomly initialized weights using the model's init_weights() method
- Fine-tuning Attempt: Attempted text-only fine-tuning on a small corpus (note: the multimodal architecture made standard text-only fine-tuning ineffective, but the model structure is valid)
- Validation: Verified model loads correctly and can perform inference
Notes
Important: This is a tiny model with randomly initialized weights intended for testing and development purposes only. It is not trained and will not produce meaningful outputs. The vision tower is completely untrained.
Use Cases
- Testing model loading and inference pipelines
- Validating quantization and compression workflows
- Debugging multimodal model handling
- CI/CD pipeline testing with realistic model sizes
- Memory profiling and optimization experiments
Limitations
- Randomly initialized weights (not trained)
- Will generate nonsensical outputs
- Vision capabilities are non-functional
- Not suitable for any production use or evaluation benchmarks
Technical Warnings
When loading this model, you may see the warning:
[transformers] `rope_parameters`'s high_freq_factor field must be greater than low_freq_factor
This is a known issue with the Llama-4 config and can be safely ignored.