SOAR-ARC Models: Self-Improving Language Models for Program Synthesis
This repository contains one of the models fine-tuned using the SOAR (Self-improving Operators for Automated program Refinements) framework, as presented in the paper:
Self-Improving Language Models for Evolutionary Program Synthesis: A Case Study on ARC-AGI
Julien Pourcel, Cédric Colas, Pierre-Yves Oudeyer. Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025.
These models are specialized in solving tasks from the challenging Abstraction and Reasoning Corpus (ARC) by synthesizing Python programs.
SOAR
Large Language Models (LLMs) have become incredibly powerful, but they often hit a wall when faced with truly complex reasoning tasks that require discovering a solution from scratch. Simply throwing more computing power or using a bigger model often yields diminishing returns. But what if a model could learn from its own experience, getting smarter with every attempt?
We introduce a framework called SOAR (Self-improving Operators for Automated program Refinements) that does just that. By creating a "virtuous cycle" of evolutionary search and learning, SOAR enables AI models to bootstrap their own capabilities and solve problems previously beyond their reach. we tested SOAR on the Abstraction and Reasoning Corpus (ARC-AGI-1), a notoriously difficult benchmark designed to challenge an AI's core reasoning abilities. We show that using SOAR with only open weight LLM, we can significantly outperforming much larger closed source LLMs.
We have released a dataset containing 5 million ARC solutions. For solutions that successfully solve an original ARC task, we deduplicate entries by their code to ensure uniqueness. For solutions that correspond to new synthetic tasks generated via hindsight relabeling, we deduplicate based on their output results. This approach ensures a diverse and high-quality dataset for further research and development.
We have also released all five of our SOAR models on Hugging Face:
SOAR framework
-
Evolutionary Search (Sample & Refine): SOAR uses an LLM to generate an initial pool of thousands of candidate programs (the "sampling" step). It then tests these programs and uses the LLM again to intelligently modify or "refine" the most promising ones based on their performance.
-
Learning from Hindsight: SOAR takes all the programs generated during the search phase—including both successes and failures—and uses them as training data. The key insight is that any failed program is simply a correct program for a different task. By "relabeling" these failed attempts as correct solutions for the synthetic tasks they inadvertently solve, SOAR creates a diverse dataset to learn from.
This process creates a powerful feedback loop: the fine-tuned model becomes better at sampling and refining, which leads to a more effective search in the next iteration, which in turn generates even better training data. And unlike previous approaches that rely on human-engineered domain-specific languages or human-generated solutions, SOAR learns to synthesize programs in Python solely from its own synthesis attempts, encompassing both successes and failures.
How to Use the Model
The primary use of this model is to generate a Python function that solves an ARC task. The input to the model should be a formatted prompt containing the training and test examples of the ARC task.
Here's a quick example to get started:
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "julien31/Soar-qwen-7b" # or any other Soar-qwen model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16, # Use torch.float16 for GPUs that don't support bfloat16
device_map="auto",
)
prompt = "def solve_arc_task(input_grid, output_grid):\
\\\"\\\"\\\"Given an ARC-AGI task, transform the input grid to the output grid by applying a series of operations.\
\\\"\\\"\\\""
messages = [
{"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
model_inputs = tokenizer(text, return_tensors="pt").to(model.device)
generated_ids = model.generate(
model_inputs.input_ids,
max_new_tokens=256,
do_sample=True,
temperature=0.7,
top_p=0.8,
repetition_penalty=1.1,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id, # This is often the same as eos_token_id for Qwen models
)
# Decode only the newly generated text
decoded_output = tokenizer.decode(generated_ids[0, model_inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(decoded_output)
For a complete, end-to-end example of how to format the prompt, run inference, execute the generated code, and visualize the results, please refer to the official repository and notebook:
- Official SOAR GitHub Repository: https://github.com/flowersteam/SOAR
- Inference & Visualization Notebook: https://github.com/flowersteam/SOAR/blob/main/notebook/inference_visualisation.ipynb
Installation
Conda inference environment
pip install --upgrade pip
git clone https://github.com/flowersteam/SOAR
cd SOAR
conda create --name sglang47 \
python=3.11 \
-y
conda activate sglang47
pip install "sglang[all]>=0.4.7"
pip install -e .
pip install -r requirements
Conda training environment
conda create --name unsloth_env \
python=3.11 \
pytorch-cuda=12.1 \
pytorch cudatoolkit xformers -c pytorch -c nvidia -c xformers \
-y
conda activate unsloth_env
pip install unsloth
cd SOAR
pip install -e .
pip install -r requirements.txt
Run SOAR
To run SOAR, please refer to execution instructions located in the experience folder.
For simple instructions on running sampling and refinement with SOAR, as well as exploring the dataset, please see the Jupyter notebooks provided in the notebook folder. These notebooks walk through the basic SOAR step, including how to generate candidate solutions, perform refinement, and analyze results. This hands-on guide will help you get started quickly and understand each step of the SOAR process.