Model Card for Cosmos-v2
Model Details
Model Description
Cosmos-v2 is a 2.1 billion parameter causal language model trained from scratch. It is based on a Qwen-like architecture and was developed over a period of 3 years through extensive experimentation (more than 116 training runs, of which 113 failed).
The model was trained on 16 billion carefully curated and manually filtered high-quality tokens. It currently shows decent understanding and reasoning capabilities but remains significantly under-trained due to the limited data volume.
Important: This model is still in an experimental stage and is currently undergoing Direct Preference Optimization (DPO) for safety alignment. At the moment, it is largely uncensored and does not reliably refuse harmful requests.
- Developed by: hsanyyasyn97
- Model type: Causal Language Model (Decoder-only)
- Language(s) (NLP): Primarily multilingual with focus on Arabic and English
- License: [Specify your license, e.g. Apache 2.0 or other]
- Finetuned from model: None (trained from scratch)
Model Sources
Uses
Direct Use
This model can be used for text generation and basic conversational tasks. It is intended for research and experimentation purposes only.
Downstream Use
Possible future use after proper alignment: further fine-tuning, instruction tuning, or domain adaptation.
Out-of-Scope Use
- Any production or high-stakes application
- Generating harmful, illegal, or dangerous content
- Relying on the model for factual accuracy without verification
- Use in safety-critical systems
Bias, Risks, and Limitations
- The model is currently uncensored and may generate harmful, dangerous, or unethical content without refusal.
- Trained on only 16B tokens, so it is under-trained and has limited knowledge coverage.
- May produce biased, incorrect, or nonsensical outputs.
- Performance on complex reasoning and long-context tasks is still limited.
- Arabic and English performance is better than other languages.
Recommendations
Users should be fully aware that this is an experimental, partially aligned model.
Do not use it for sensitive or production purposes until safety alignment (DPO) is completed.
Always review and filter the model's outputs.
How to Get Started with the Model
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "hsanyyasyn97gmail/cosmos-v2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
inputs = tokenizer("Your prompt here", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Training Details
Training Data
Total tokens: 16 billion
Data was heavily filtered, cleaned, and manually curated over a long period (high quality, not random web scrapes)
Mix of Arabic and English data + some additional sources
Training Procedure
Trained from scratch (not continued pretraining)
Architecture: Qwen-like
Framework: JAX + XLA
Hardware: Cloud TPU v5 (8 chips)
Current stage: Undergoing DPO (Direct Preference Optimization) using open-source safety preference data (including Anthropic-sourced data) + synthetic preference pairs
Training Hyperparameters
Precision: Mixed precision (details to be updated)
Optimizer and scheduler: Standard for JAX TPU training
Evaluation
No formal public evaluation results available yet.
Internal testing shows reasonable language understanding and basic reasoning, but clear limitations due to limited training tokens.
Environmental Impact
Hardware Type: Google TPU v5
Cloud Provider: Cloud TPU (exact provider not specified)
Hours used: Significant (multiple long training runs over 3 years)
Carbon Emitted: Not calculated
Technical Specifications
Model Architecture and Objective
Architecture: Qwen-like decoder-only Transformer
Parameters: 2.1 Billion
Objective: Causal language modeling (next-token prediction)
Current additional objective: DPO for preference alignment
Compute Infrastructure
Hardware
8× TPU v5
80 vCPU
330 GB RAM
Software
JAX (raw) + XLA
Transformers (for inference and model card)
Model Card Authors
hsanyyasyn97
Model Card Contact
Contact via Hugging Face profile: hsanyyasyn97gmail