⚛️ Photon-1.0-1M

Welcome to Photon-1.0-1M, an ultra-compact, hyper-dense Small Language Model (SLM) developed by AtomixLabs. Built to explore performance efficiency under tight resource constraints, Photon-1.0-1M packs just 1 million parameters and is designed to deliver steady reasoning and retrieval capabilities within a highly compact footprint.

Instead of relying on sprawling parameter counts to capture general knowledge, AtomixLabs designed Photon to be streamlined. By allocating its parameter budget directly into dense, high-utility transformer layers rather than larger vocabulary tables, this model functions as a sharp, fast, and capable logic engine for its size.

🛠️ Architectural Configuration

While many micro-models in this parameter range rely on highly customized, non-standard scripts that layer on experimental attention mechanisms, exotic gating structures, or non-native dependencies, AtomixLabs opted to stick to a clean, standard LLaMA architecture. The team believes that if a core architecture requires heavy modifications to function effectively, the challenge often lies in the training and optimization process rather than the standard transformer block itself.

By utilizing native, standard LLaMA layers, Photon remains compatible, stable, and highly dense:

# Photon-1.0-1M Structural Specification
vocab_size: int = 1536              # Optimized vocabulary allocation
hidden_size: int = 128              # High-density representations
intermediate_size: int = 384        # Symmetric 3.0x SwiGLU ratio
num_hidden_layers: int = 4          # The standard "Markov Floor" limit
num_attention_heads: int = 4        # Head dimension = 32
num_key_value_heads: int = 4        # 4:4 Multi-Head Attention (MHA)
max_position_embeddings: int = 512  # Native context length
rope_theta: float = 1110.0          # Tuned angular frequency for context scaling

By keeping the vocabulary tightly constrained and utilizing a full 4:4 Multi-Head Attention (MHA) scheme, the model retains reliable contextual memory and precise information-retrieval characteristics without needing external, non-native code dependencies.

📚 Dataset & Training Mix

To encourage collaboration, open science, and reproducibility in the micro-SLM space, AtomixLabs is sharing the training recipe.

Photon-1.0-1M was trained on approximately 2.56 billion tokens in total. To ensure licensing compliance and proper attribution, the training datasets listed below are custom, curated subsets extracted directly from the main openbmb/UltraX-Preview dataset. While the overarching UltraX repository is distributed under an Apache-2.0 license, users must abide by the specific underlying licenses of the original source datasets:

The Math Boost

To round out the 2.56-billion-token run, AtomixLabs augmented the subsets above with approximately 160,000,000 tokens of highly curated, synthetically generated arithmetic.

While the team is keeping the exact generation recipe and structural constraints of this synthetic data proprietary, this targeted math scaffolding was helpful in teaching the 1M-parameter model how to process logical relationships and numerical consistency.

📊 Benchmark Evaluation Results

All evaluations were executed in a strict zero-shot (num_fewshot=0) environment to ensure genuine model capability and prevent few-shot prompt bias.

We evaluate the overall score using a standard 4-way average formula that aggregates the two ARC benchmarks first: $$\text{ARC Combined} = \frac{\text{ARC-Easy} + \text{ARC-Challenge}}{2}$$ $$\text{Overall Average} = \frac{\text{Hellaswag} + \text{ARC Combined} + \text{PIQA} + \text{ArithMark 2.0}}{4}$$

Zero-Shot Results

🖥️ Reproducibility & Environment Context

To ensure you can replicate these exact scores in your own environment, we have extracted a snippet of our hardware and library diagnostic report.

OS Platform              : Windows (11)
CPU Architecture         : AMD64
Target GPU Name          : NVIDIA GeForce RTX 5070 (11.94 GB VRAM)
PyTorch Version          : 2.13.0+cu132
transformers Library     : 5.12.1
lm-eval-harness Library  : 0.4.13.dev0

1. Architectural Limitations and Inherent Risks

Photon-1.0-1M is an experimental, micro-scale Small Language Model (SLM) consisting of approximately 1 million parameters. Due to this extreme parameter constraint, the model lacks the capacity to retain broad general knowledge, synthesize complex reasoning paths, or maintain stable safety boundaries.

Users and developers must acknowledge the following technical behaviors inherent to micro-scale models:

• Severe Hallucination Rate: The model's capacity for factual retention is extremely limited. It will generate plausible-sounding but entirely fabricated facts, dates, citations, and mathematical assertions.
• Logical Inversion: The model may occasionally invert logical operators (e.g., substituting "safe" for "unsafe" or "do" for "do not"), which can drastically alter the meaning of generated instructions.
• High Sensitivity to Prompt Bias: Lacking robust system-prompt enforcement mechanisms, the model is highly susceptible to trivial jailbreaks, prompt injections, and leading questions, which can easily bypass intended behavioral constraints.
• Semantic Drift: During longer context generation, the model is prone to rapid degradation of coherence, repetitive looping, and semantic drift, which may result in unexpected or nonsensical outputs.

2. Categorical Safety Hazards

Photon-1.0-1M has not undergone Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), or any proprietary safety alignment post-training. Consequently, it may generate outputs containing:

• Bias and Discrimination: The training data subsets, despite filtering efforts, may contain historical and systemic biases. The model may reproduce, amplify, or generate harmful stereotypes regarding race, gender, religion, sexual orientation, disability, or nationality.
• Toxicity and Offensive Content: The model can be induced to generate profane, crude, harassing, or otherwise objectionable language with minimal adversarial prompting.
• Inaccurate Technical Advice: Any instructions generated regarding software development, configuration, or hardware assembly must be treated as untrustworthy and potentially hazardous to computing environments.

3. Out-of-Scope & Non-Recommended Uses

To prevent harm and reduce operational risk, the deployment of Photon-1.0-1M is strongly discouraged and considered out-of-scope in the following domains:

• High-Risk Decision-Making: Any system involving automated scoring, legal profiling, credit evaluation, employment screening, or resource allocation affecting civil liberties.
• Clinical and Medical Applications: Providing medical advice, diagnostic assistance, pharmaceutical dosage recommendations, or psychological support.
• Financial and Legal Advice: Generating contractual text, investment strategies, regulatory compliance evaluations, or tax planning guidance.
• Critical Infrastructure: Integration into control loops for energy grids, water systems, public transportation, or automated industrial machinery.
• Military and Defense: Target identification, strategic analysis, weapon systems design, or any defense-related deployment.

4. Suggested Usage Guidelines

Given the experimental nature and 1M parameter size of Photon-1.0-1M, developers should observe basic precautions:

• Expect Factual Errors: The model is highly prone to hallucination. Do not rely on its output for factual queries without independent verification.
• Monitor Coherence: Micro-models can drift into repetitive loops or gibberish. Implement basic output checks (such as length limits or repetition penalties) to manage user experience.

5. Legal Disclaimer and Limitation of Liability

This model is provided "as-is" and "as-available" without warranties of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, non-infringement, or course of performance.

AtomixLabs, its contributors, and affiliates do not warrant that:

1. The model will meet your specific technical or operational requirements.
2. The model's outputs will be accurate, reliable, secure, or free from error.
3. The model will be compatible with all downstream software packages or hardware architectures.

In no event shall AtomixLabs, its developers, or its contributors be liable for any direct, indirect, incidental, special, exemplary, or consequential damages (including, but not limited to, procurement of substitute goods or services; loss of use, data, or profits; or business interruption) however caused and on any theory of liability, whether in contract, strict liability, or tort (including negligence or otherwise) arising in any way out of the use, modification, distribution, or deployment of this model, even if advised of the possibility of such damage.