AstraQuasar-4B is our first pre-trained Large Language Model (LLM) for text generation. It is a model with 4B parameters, whithout embeddings. AstraQuasar-4B-v.0.1 is built upon the foundation of the Phi-2 architecture, with significant enhancements including an increased number of layers and the innovative introduction of a novel technique known as the duplicate trick.
AstraQuasar-4B-v.0.1 at the moment is an under trained model. Serving as a demonstration of the potential of the duplication trick and its implications for future advancements in language modeling. Despite its nascent status, our model has already demonstrated superior performance compared to both the base Phi-2 model and earlier iterations of AstraQuasar-4B that do not utilize the duplication trick.
One of the key milestones achieved by AstraQuasar-4B is its successful application of backpropagation on the duplication trick, setting a precedent for future research and development in this area.
The use of the duplicate trick had shown to instantly decrease the loss by ~21% with no added instability
Our model's architecture is fully compatible with leading training frameworks such as Axolotl and LLaMA Factory, ensuring seamless integration into existing workflows leveraging the standard Hugging Face Transformers library.
Example:
AstraQuasar-4B can be easily instantiated using the Hugging Face Transformers library:
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model = AutoModelForCausalLM.from_pretrained("AstraMindAI/AstraQuasar-4B", torch_dtype=torch.float16, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("AstraMindAI/AstraQuasar-4B")
# you can optionally disable the duplicate trick
# model.model.duplicate_trick = False
# you can also disable the duplicate gradient calculation during training
# model.model.duplicate_grad = False
# You can specify the layer ranges for the duplicate trick
# model.model.layer_ranges = [(0, 16),(8, 24),(17, 32),(25, 40),(33, 49),(40, 56)]
prompt = "I love my dog because "
inputs = tokenizer(prompt, return_tensors="pt")
# Generate
generate_ids = model.generate(inputs.input_ids, max_length=30)
tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
Pre-training and fine-tuning can be performed using accelerate or deepspeed.
Notice
It's important to note that AstraQuasar-4B is a pre-trained base model and does not incorporate any moderation mechanisms.
NEWS
Stay tuned for exciting developments! A new architecture, AstraPulsar, is on the horizon, promising further advancements in language modeling.
Credits:
- Undi95 for helping us figuring out the process of self-calling layers.