Instella-Long✨: Fully Open Language Model with Long-context Capability

AMD is excited to announce Instella-Long, a long-context language model continually trained from Instella-3B-Instruct on AMD Instinct™ MI300X GPUs. To our knowledge, Instella-Long makes Instella series the first fully open language model trained from scratch that supports long-context. Instella-Long can support 128K context length and achieve competitive performance outperforming open-weights models such as Phi-3.5-mini, Gemma-3-4B, and Qwen2.5-3B on the long-context benchmark.

By training Instella with long context extension on Instinct MI300X GPUs, we highlight our hardware’s capability and scalability in handling demanding AI training workloads, offering a viable alternative in the AI hardware landscape. In line with the AMD commitment to open source, we are sharing all the model weights, detailed training configurations, datasets, and code, enabling the AI community to collaborate, replicate, and innovate, thereby accelerating progress.

Takeaways

• Announcing Instella-Long, a 3B long-context language model with 128K context length support developed by AMD, trained on 64 Instinct MI300X GPUs.

• To our knowledge, Instella-Long makes Instella series the first fully open language model trained from scratch that supports long-context. Hugging face model, training data, and training code are fully open-sourced.

• Supported by the AMD ROCm software stack, Instella-Long employs efficient training techniques such as Sequence Parallelism, FlashAttention-2, Torch Compile, and FSDP to distribute model training over 8 MI300 nodes each with 8 GPUs.

Instella-Long

Instella-Long is based on the Instella model released in March. Specifically, Instella-Long is continually trained from Instella-3B-Instruct and follows the same model architecture. The training of Instella-Long comprises three stages: 1. Continued Pre-Training, 2. Supervised Finetuning (SFT), 3. Direct Preference Optimization (DPO).

Continued Pre-Training

Training: We employ a two-phase pre-training starting from Instella-3B-Instruct (4K context length). Phase 1: We extend the context length from 4,096 to 65,536 tokens and train the model using 20B tokens. We follow the RoPE scaling law to increase the base frequency of RoPE from 10,000 to 514,640. Phase 2: As indicated by Prolong, it is beneficial to train the model with the data whose context length is longer than the target context length. In this phase, we train the model on 20B tokens with a maximum context length of 262,144 - 2× the target context length of 128K. Following the RoPE scaling law, we increase the RoPE base frequency to 3,691,950.

Data: Our continued pre-training data originates from the data mix created by Prolong. We use the text data curated by Prolong and tokenize the data with our tokenizer. In each phase of the continued pre-training, we train on a mix of long and short context data. Specific details are outlined as follows:

Supervised Finetuning (SFT)

Training: After continued training on the long-context pre-training data, we perform supervised finetuning on long-context instruction data. We train the model using a 1B-token mixture of short- and long-context instruction data.

Data: Similar to the continued pre-training stage, we train the model on a mixture of short- and long-context instruction data with a ratio of 4 to 6. For short-context instruction data, we use Ultrachat 200K, OpenMathinstruct-2, Tülu-3 Instruction Following, and MMLU auxiliary train set. For long-context instruction data, we construct a synthetic long-context instruction dataset due to the lack of long-context SFT data. Specifically, we make use of the long documents from Books, which is part of our continued pre-training data corpus. We select documents with a minimum length of 8K tokens and truncate those exceeding 128K tokens to a maximum length of 128K. Then, we use Qwen2.5-14B-Instruct-1M as a teacher model to synthetically generate question-answer pairs for the documents. To speed up this process, we randomly choose a subpart of the document for the QA generation instead of using the whole document. The length of the subpart is randomly set to be between 2K and 8K tokens. We use NLTK sentence tokenizer to divide documents into sentences to make sure that the selected subpart has complete sentences. The generated question and answer are appended to the end of the long document, serving as a complete single-round instruction-following data sample. In addition, we also generate long-context instruction data using short documents, in order to increase the dataset diversity with more data sources. We use ArXiv from our continued pre-training corpus and the DCLM subset from Dolmino-Mix-1124. We first generate QA for each short document following the same pipeline aforementioned. Then, we iteratively concatenate different short documents until it reaches 128K tokens. The concatenated document can exceed 128K as we do not truncate the last document. Lastly, we randomly choose one QA corresponding to one of the short documents and append it to the end of the concatenated document. The final data mixture for the SFT stage is shown as follows:

Direct Preference Optimization (DPO)

Training: At the last training stage, we perform human preference alignment training using Direct Preference Optimization. We employ the same DPO training as Instella-3B-Instruct using the same data. Different from previous training stages, in the DPO stage, we train on short data only whose maximum context length is 2K. Consistent with the findings of other open-weights models, we observe that performing DPO on short data only continues to improve the model performance on long-context tasks.

Data: We use the OLMo-2-1124-7B-Preference-Mix dataset as our DPO data which contains 0.76B tokens.

Sequence Parallelism

To enable training with extremely long inputs, we implement sequence parallelism based on Deepspeed Ulysses. The sequence parallelism distributes the attention heads across GPUs during the attention computation. It is more efficient than Ring-Attention in GPU communications. We use four GPUs as a sequence parallelism group for the Phase 2 continued pre-training and SFT due to the long inputs.

Example Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "amd/Instella-3B-Long-Instruct"

tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto", trust_remote_code=True)

prompt = [{"role": "user", "content": "What are the benefits of open-source AI research?"}]
inputs = tokenizer.apply_chat_template(
    prompt,
    add_generation_prompt=True,
    return_tensors='pt'
)

tokens = model.generate(
    inputs.to(model.device),
    max_new_tokens=1024,
    temperature=0.8,
    do_sample=True
)

print(tokenizer.decode(tokens[0], skip_special_tokens=False))

Results

• We evaluate the long-context performance on Helmet, a recent and comprehensive long-context evaluation benchmark encompassing diverse categories. Helmet demonstrates better consistency of human perception than the previous long-context benchmarks.

• Instella-3B-Long-Instruct outperforms open weights models including Phi-3.5-mini-instruct, Gemma-3-4B-it, Qwen2.5-3B-Instruct, and MiniCPM-2B-128k on most tasks of the Helmet benchmark.

• We made a side-by-side comparison at 8K, 16K, and 32K context lengths with Qwen2.5-3B-Instruct as its context length is 32K. Instella-3B-Long-Instruct outperforms Qwen2.5-3B-Instruct by 2.75% on average.

Evaluation Metric: We use substring exact match (SubEM) for the RAG tasks including Natural Questions, TriviaQA, and HotpotQA. We use recall for NIAH and exact match for InfiniteBench MC. For InfiniteBench QA and NarrativeQA, where the answers are open-ended, we use gpt-4o-mini to evaluate the answers against the ground truth using the prompt and metric provided by the Helmet.

Short-context results: We observe performance drops on some short-context benchmarks compared to Instella-3B-Instruct. Interestingly, TruthfulQA remains stable, while Crows-Pair shows a slight improvement, indicating potential gains in certain responsible AI metrics. The reduction in Toxigen (57.02 → 42.34, lower is better) suggests improved toxicity avoidance in the long-context variant. We hypothesize that these results reflect a trade-off between optimizing for longer context lengths and retaining short-context performance, which may be more pronounced at the 3B parameter scale compared to larger models.

Training Data

[!NOTE] Further information concerning the training datasets, including applicable licensing terms and use restrictions, may be located at the linked source location.

Conclusion

The release of the Instella-Long model represents a significant stride in advancing open-source AI and demonstrates the capabilities of AMD hardware in language model training. To our knowledge, Instella-Long makes Instella series the first fully open language model trained from scratch that supports long-context, while achieving competitive performance compared to open-weights models.

By fully open sourcing the Instella-Long model, including weights, training configurations, datasets, and code, we aim to foster innovation and collaboration within the AI community. We believe that transparency, reproducibility and accessibility are key drivers of progress in AI research and development. We invite developers, researchers, and AI enthusiasts to explore Instella-long, contribute to its ongoing improvement, and join us in pushing the boundaries of what is possible with language models.

Additional Resources

Code

• Github: https://github.com/AMD-AIG-AIMA/Instella/tree/instella-long

Please refer to the following blogs to get started with using these techniques on AMD GPUs:

• Introducing Instella: New State-of-the-art Fully Open 3B Language Models
• PyTorch Fully Sharded Data Parallel (FSDP) on AMD GPUs with ROCm™
• Accelerating Large Language Models with Flash Attention on AMD GPUs
• Accelerate PyTorch Models using torch.compile on AMD GPUs with ROCm™
• Introducing the First AMD 1B Language Models: AMD OLMo

Bias, Risks, and Limitations

• The models are being released for research purposes only and are not intended for use cases that require high levels of factuality, safety critical situations, health, or medical applications, generating false information, facilitating toxic conversations.

• Model checkpoints are made accessible without any safety promises. It is crucial for users to conduct comprehensive evaluations and implement safety filtering mechanisms as per their respective use cases.

• It may be possible to prompt the model to generate content that may be factually inaccurate, harmful, violent, toxic, biased, or otherwise objectionable. Such content may also get generated by prompts that did not intend to produce output as such. Users are thus requested to be aware of this and exercise caution and responsible thinking when using the model.

• Multi-lingual abilities of the models have not been tested and thus may misunderstand and generate erroneous responses across different languages.

License

• The Instella-3B-Long-Instruct model is licensed for academic and research purposes under a ResearchRAIL license. Refer to the LICENSE and NOTICES files for more information.
• The amd/Instella-Long is a collection of pre-training and instruction following data that is used to train Instella-3B-Long-Instruct, and is licensed for academic and research purposes under a ResearchRAIL license. Refer to the LICENSE in the amd/Instella-Long dataset card for more information.

Citations

Feel free to cite our Instella-3B models:

@article{instella,
  title={Instella: Fully Open Language Models with Stellar Performance},
  author={Liu, Jiang and Wu, Jialian and Yu, Xiaodong and Su, Yusheng and Mishra, Prakamya and Ramesh, Gowtham and Ranjan, Sudhanshu and Manem, Chaitanya and Sun, Ximeng and Wang, Ze and Brahma, Pratik Prabhanjan and Liu, Zicheng and Barsoum, Emad},
  journal={arXiv preprint arXiv:2511.10628},
  year={2025}
}