LLama3 SatCom 8B
LLama3 SatCom 8B is a fine-tuned open Large Language Model (LLM) developed under the ESA ARTES programme as part of the SatcomLLM / SCEVA (SatCom Expert Virtual Assistant) project.
It is designed to support satellite communications (SatCom) experts, engineers, and mission planners through domain-specialised reasoning, question answering, and document-based assistance.
Model Description
- Base model:
meta-llama/Llama-3.1-8B-Instruct - Fine-tuning type: Instruction fine-tuning (IFT)
- Training data: Domain-specific question–answer datasets (manual, synthetic, and multiple-choice)
- Architecture: Decoder-only transformer, 8 billion parameters
- Languages: English
- License: LLama-3.1 Communiy License Agreement
The model has been fine-tuned on curated SatCom-related corpora to enhance its understanding of technical language, protocols, and reasoning processes common to satellite communications, including 5G/6G non-terrestrial networks, link budget evaluation, and mission engineering tasks.
Training Datasets
| Dataset | Description |
|---|---|
| esa-sceva/satcom-synth-qa | Synthetic QA data generated via agentic pipelines using large teacher models |
| esa-sceva/satcom-synth-qa-cot | Chain-of-thought annotated QA used to improve reasoning depth and factual traceability |
Intended Use
Primary use cases:
- Technical Q&A and reasoning on SatCom systems
- Support for link budget and RF engineering questions
- Guidance for 5G/6G NTN (Non-Terrestrial Network) operations
- Mission design, planning, and anomaly detection support
- Educational and research use within the SatCom sector
Intended users:
- ESA engineers and project officers
- SatCom and aerospace researchers
- SMEs and technical operators in satellite communication
- Academic and educational users
Limitations
- The model does not access real-time mission data or proprietary ESA documents.
- Answers are based on training data and may require expert validation for operational use.
- It should not be relied upon for flight-critical or safety-critical decisions.
- Limited context window (base 8B configuration) may constrain long-document reasoning.
Technical Details
| Parameter | Value |
|---|---|
| Base Model | Llama 3.1 8B Instruct |
| Parameters | 8 billion |
| Context length | 8k tokens |
| Precision | bfloat16 / fp16 |
| Framework | Lit-GPT (Lightning AI) |
| Training infra | EuroHPC MareNostrum5 + AWS EC2 |
| Optimisation | LoRA fine-tuning, cosine LR schedule |
Evaluation
Evaluation Datasets
The model was benchmarked on both general-purpose and domain-specific QA tasks. Regarding Satcom-specific datasets:
| Dataset | Subset | Description |
|---|---|---|
| esa-sceva/satcom-qa | Open SatCom QA | Conceptual and reasoning-based questions on SatCom workflows, regulations, and mission/system design |
| Math SatCom QA | Quantitative and formula-based questions derived from system design and orbital mechanics topics | |
| esa-sceva/satcom-mcqa | Open MCQA | Conceptual multiple-choice questions on RF systems, communication protocols, and architecture |
| Math MCQA | Numerical and link-budget-focused multiple-choice questions testing applied calculations |
Results
Table: Evaluation results (%).
- Norm. denotes the normalized score obtained by averaging 1–5 ratings from a panel of LLM judges (Qwen3, gpt-4.1-mini, Mistral-Large-2512, and DeepSeek-V3.2) and scaling to [0,1].
- Bin. is the binary accuracy computed from correctness judgments.
- WR (Adjusted Win Rate) is defined as (wins + 0.5 × ties) / total, based on pairwise comparisons with randomized answer order.
Multiple-choice performance is measured using standard accuracy.
All results are averaged over 3–5 runs; standard deviation ≤ 0.25 pp for open-ended QA and ≤ 0.10 pp for MCQA.