Alpha-Orionis-v0.1
Disclaimer
Note: All models and LoRAs from the Orion series were created with the sole purpose of research. The usage of this model and/or its related LoRA implies agreement with the following terms:
- The user is responsible for what they might do with it, including how the output of the model is interpreted and used;
- The user should not use the model and its outputs for any illegal purposes;
- The user is the only one resposible for any misuse or negative consequences from using this model and/or its related LoRA.
I do not endorse any particular perspectives presented in the training data.
Orion Series
This series aims to develop highly uncensored Large Language Models (LLMs) with the following focuses:
- Science, Technology, Engineering, and Mathematics (STEM)
- Computer Science (including programming)
- Social Sciences
And several key cognitive skills, including but not limited to:
- Reasoning and logical deduction
- Critical thinking
- Analysis
While maintaining strong overall knowledge and expertise, the models will undergo refinement through:
- Fine-tuning processes
- Model merging techniques including Mixture of Experts (MoE)
Please note that these models are experimental and may demonstrate varied levels of effectiveness. Your feedback, critique, or queries are most welcome for improvement purposes.
Base
This model and its related LoRA was fine-tuned on https://huggingface.co/fearlessdots/WizardLM-2-7B-abliterated.
LoRA
The LoRA merged with the base model is available at https://huggingface.co/fearlessdots/Alpha-Orionis-v0.1-LoRA.
GGUF
I provide some GGUF files here: https://huggingface.co/fearlessdots/Alpha-Orionis-v0.1-GGUF.
Datasets
Fine Tuning
- Quantization Configuration
- load_in_4bit=True
- bnb_4bit_quant_type="fp4"
- bnb_4bit_compute_dtype=compute_dtype
- bnb_4bit_use_double_quant=False
- PEFT Parameters
- lora_alpha=64
- lora_dropout=0.05
- r=128
- bias="none"
- Training Arguments
- num_train_epochs=1
- per_device_train_batch_size=1
- gradient_accumulation_steps=4
- optim="adamw_bnb_8bit"
- save_steps=25
- logging_steps=25
- learning_rate=2e-4
- weight_decay=0.001
- fp16=False
- bf16=False
- max_grad_norm=0.3
- max_steps=-1
- warmup_ratio=0.03
- group_by_length=True
- lr_scheduler_type="constant"
Credits
- The Wizard team for creating the incredible base model;
- HuggingFace: for hosting this model and for creating the fine-tuning tools used;
- failspy (https://huggingface.co/failspy): for the orthogonalization implementation;
- NobodyExistsOnTheInternet (https://huggingface.co/NobodyExistsOnTheInternet): for the incredible dataset;
- Undi95 (https://huggingface.co/Undi95) and Sao10k (https://huggingface.co/Sao10K): my main inspirations for doing these models =]
A huge thank you to all of them ☺️
About Alpha Orionis
Alpha Orionis, commonly known as Betelgeuse, is a red supergiant star located in the constellation Orion. With an apparent magnitude ranging from +0.0 to +1.6, it is the second-brightest star in the constellation and the tenth-brightest in the night sky. It appears distinctly reddish and is classified as a semi-regular variable star due to its wide range in brightness. At near-infrared wavelengths, it becomes the brightest star in the night sky.
Alpha Orionis has a radius approximately 760 times larger than the sun, meaning it would extend far past the orbit of Mars if placed at the center of our solar system. Estimates suggest it has a mass between 10 and 20 times that of the sun. Despite being relatively close to us—its distance ranges from around 400 to 600 light-years away, according to recent measurements—there remains significant uncertainty regarding its exact position.
This young stellar giant—less than 10 million years old—has already exhausted much of its nuclear fuel and will eventually explode in a spectacular supernova, potentially within the next 100,000 years. Such an event could cause it to outshine even the Moon for several months, though it poses no threat to life on Earth. As a result of its high velocity relative to other celestial objects—approximately 30 kilometers per second—it creates a massive bow shock in space, extending up to four light-years across.
In addition to these remarkable features, Alpha Orionis holds the distinction of having had its photospheric angular size calculated before any other extrasolar star, back in 1920. Modern observations reveal an average angular diameter of 0.048 arcseconds, making it one of the largest visible objects in the night sky. Moreover, it boasts a vast, irregular envelope surrounding the star, encompassing nearly 250 times its diameter, resulting from substantial mass loss throughout its lifetime. These combined characteristics place Alpha Orionis among the most fascinating and intriguing celestial bodies observable from Earth.
Source: retrived from https://en.wikipedia.org/wiki/Betelgeuse and processed with https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1.