⚡ ReverX-1-100M: Reversible Transformer Causal LM (RevNet Architecture)
🔬 FOR RESEARCH PURPOSES ONLY
Developed and Trained by the Cortiqa Team.
This artifact is an experimental exploration into Reversible Residual Transformer architectures to investigate zero-activation memory dynamics during causal autoregressive training.
🏛️ Academic Attribution & Foundational Citations
The architectural principles implemented in ReverX are directly built upon foundational academic research. We gratefully cite and attribute the original inventors:
-
The Reversible Residual Network (RevNet):
- Aidan N. Gomez, Mengye Ren, Raquel Urtasun, Roger B. Grosse (2017).
- Paper: "The Reversible Residual Network: Backpropagation Without Storing Activations", Advances in Neural Information Processing Systems (NeurIPS 2017).
- Contribution: Introduced bijective coupling equations allowing exact analytical input reconstruction during backward passes without caching forward activations.
-
The Reformer (Google Research):
- Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya (2020).
- Paper: "Reformer: The Efficient Transformer", International Conference on Learning Representations (ICLR 2020).
- Contribution: Demonstrated the adaptation of RevNet reversible coupling to Multi-Head Attention and Feed-Forward transformer layers.
📐 Architecture Specifications (Cortiqa Team)
| Specification | Details |
|---|---|
| Model Name | ReverX-1-100M |
| Organization | Cortiqa (Cortiqa) |
| Development Team | Cortiqa Team |
| Research Status | Experimental Research Prototype |
| Total Parameters | 100,457,472 (~100.45 Million) |
| Coupling Layers | 20 Bijective Reversible Blocks |
| Hidden Size | 768 (Split into two 384-dim channels) |
| Attention Heads | 12 Heads (Head Dim: 32 per channel) |
| FFN Intermediate | 3,200 |
| Tokenizer | GPT-2 BPE (vocab_size = 50,257) |
| Training Steps | 130,391 optimization steps |
| Training Hardware | 1x NVIDIA Tesla T4 GPU (16GB) on Kaggle |
| Training Dataset | tatsu-lab/alpaca (52,002 instruction pairs) |
| Final Loss | 0.1356 (Cross-Entropy) |
🧪 Benchmark Observations & Emergent Capabilities
Trained from scratch (random initialization, zero pre-trained weights), ReverX-1-100M demonstrated the following properties during benchmark evaluations:
1. Factual Knowledge Retrieval
- Prompt: "What is the capital of France?"
- Output:
The capital of France is Paris.(100% exact retrieval)
2. Scientific Concept Recall
- Prompt: "What is photosynthesis in simple terms?"
- Output:
Photosynthesis is the process by which plants, algae and oxygen are created and light energy. This absorbs molecules of chlorophyll... convert carbon dioxide
3. Syntactically Valid Code Generation
- Prompt: "Write a short python function to add two numbers."
- Output:
def add(a, b):
return a + b
⚠️ Research Purpose & Usage Limitations
- Experimental Prototype: This model was created purely for academic inquiry into reversible transformers. It is not aligned for commercial or open-ended production dialogue.
- Channel Coupling Bias: Splitting the hidden dimension into [x1, x2] reduces full-rank cross-channel interaction in compact (100M) networks, leading to token drift on open-ended creative tasks.
- Inference Guidance: To mitigate repetition, generation should be run with
repetition_penalty >= 1.2and nucleus samplingtop_p = 0.85.
📜 Citation
@misc{reverx100m_cortiqa2026,
title={ReverX-1-100M: Reversible Transformer Causal LM (RevNet Architecture)},
author={Cortiqa Team and Collaborators},
year={2026},
publisher={Hugging Face},
howpublished={\url{https://huggingface.co/Cortiqa/reverx-1-100m}}
}