Bioinspired-Base-Hermes-3-Llama-3.1-8B

This model was constructed from the Llama-3.1-8B base model using a combination of Continued Pre-training (CPT), Supervised fine-tuning (SFT), ORPO, and then a SLERP merge with the Hermes-3-Llama-3.1-8B model.

Only the Llama-3.1-8B base model was used in the construction of this model, including in the CPT-SFT-ORPO phases and the development of NousResearch/Hermes-3-Llama-3.1-8B (see their technical report).

The model was trained on a mix of publically available datasets and a corpus of around 8,000 scientific papers in the bio-inspired materials field. During the CPT phase, the raw text of all papers is used. During SFT and ORPO, the model is shown question-answer pairs and question-answer-rejected triples, respectively, along with other datasets to train the model for instructions and chat interactions.

Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

def generate_response(model, tokenizer, text_input="Biology offers amazing materials. Tell me more!", system_prompt='You are a materials scientist.',
                      num_return_sequences=1, temperature=0.3, max_new_tokens=256, do_sample=True,
                      num_beams=1, eos_token_id=[128001, 128008, 128009, 128040],
                      device='cuda', top_k=50, top_p=0.9, repetition_penalty=1.1, messages=None, ):

    if messages is None:
        if system_prompt:
            messages = [{"role": "user", "content": system_prompt + text_input}]
        else:
            messages = [{"role": "user", "content": text_input}]
    else:
        messages.append({"role": "user", "content": text_input})

    text_input = tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
    )

    inputs = tokenizer([text_input], add_special_tokens=False, return_tensors='pt').to(device)

    with torch.no_grad():
        outputs = model.generate(
            **inputs, 
            max_new_tokens=max_new_tokens,
            temperature=temperature,
            num_beams=num_beams,
            top_k=top_k,
            do_sample=do_sample,
            top_p=top_p,
            eos_token_id=eos_token_id,
            num_return_sequences=num_return_sequences, 
            repetition_penalty=repetition_penalty,
        )

    outputs = outputs[:, inputs["input_ids"].shape[1]:]

    return tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True), messages            

def load_model(model_name, chat_template=None, compile_mode=None,
               attn_implementation="flash_attention_2", quant=False):
    
    if quant:
        bnb_config4bit = BitsAndBytesConfig(
            load_in_4bit=True,
            bnb_4bit_quant_type="nf4",
            bnb_4bit_compute_dtype=torch.bfloat16,
            bnb_4bit_use_double_quant=True,
        )
        model = AutoModelForCausalLM.from_pretrained(
            model_name,
            trust_remote_code=True,
            use_cache=False,
            quantization_config=bnb_config4bit,
            device_map="auto",
            torch_dtype=torch.bfloat16,
            attn_implementation=attn_implementation,
        )
    else:
        model = AutoModelForCausalLM.from_pretrained(
            model_name,
            trust_remote_code=True,
            use_cache=False,
            device_map="auto",
            torch_dtype=torch.bfloat16,
            attn_implementation=attn_implementation,
        )

    if compile_mode:
        model.generation_config.cache_implementation = "static"
        model.forward = torch.compile(model.forward, mode=compile_mode, fullgraph=True)

    tokenizer = AutoTokenizer.from_pretrained(
        model_name, 
        trust_remote_code=True,
        add_bos_token=False
    )

    if chat_template:
        tokenizer.chat_template = chat_template

    return model, tokenizer

Chat template

The model uses the ChatML template, as so:

messages = [
    {"role": "system", "content": "You are a biomaterials scientist."},
    {"role": "user", "content": "What is the role of collagen in bone?"},
    {"role": "assistant", "content": "Collagen is a key structural protein."},
]
prompt = tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
)
print (prompt)

The output:

<|begin_of_text|><|im_start|>system
You are a biomaterials scientist.<|im_end|>
<|im_start|>user
What is the role of collagen in bone?<|im_end|>
<|im_start|>assistant
Collagen is a key structural protein.<|im_end|>
<|im_start|>assistant

Example use cases

Load the model:

model, tokenizer,=load_model (model_name='lamm-mit/Bioinspired-Base-Hermes-3-Llama-3.1-8B')

Bio-inspired material role play as a worm discussing leaves and collagen

messages=[]
query = "Collagen and leaves, discuss their relationship."
result,messages=generate_response (model, tokenizer, text_input = , num_return_sequences=1, 
                                   system_prompt="You are a worm. Everything you say is from that perspective. ",
                                   temperature=0.1, max_new_tokens=1024 , messages=[], )

for res in result:
    print(res, "\n")

messages.append (  {"role": "assistant", "content":result[0]})

result,messages=generate_response (model, tokenizer, text_input="Summarize this nicely in a JSON object.",num_return_sequences=1, 
                                   temperature=0.1,max_new_tokens=1024 , messages=messages, )

for res in result:
    print(res, "\n")

messages.append (  {"role": "assistant", "content":result[0]})  

{
  "collagen": {
    "description": "A protein that provides structural support to tissues in the body, including skin, tendons, and ligaments.",
    "function": "Maintains the integrity and function of tissues in animals."
  },
  "leaves": {
    "description": "The green, flat structures that grow on plants, primarily composed of cellulose.",
    "function": "Capture sunlight and produce energy through photosynthesis."
  }
}

Fish and Glass Bead Game

messages=[]
result,messages=generate_response (model, tokenizer, text_input="Collagen and the Glass Bead Game, discuss their relationship. Work out a detailed roleplay and hypotheses.",
                                   num_return_sequences=1,  temperature=0.3,max_new_tokens=1024 , messages=[]
                                   system_prompt="You are a fish. Everything you say is from that perspective. ",)

for res in result:
    print(res, "\n")

messages.append (  {"role": "assistant", "content":result[0]})  

You can ask a follow-up question like so:

result,messages=generate_response (model, tokenizer, text_input="Summarize the key points in JSON format, covering analogies, differences, and definitions.",num_return_sequences=1, 
                                   temperature=0.3,max_new_tokens=1024 , messages=messages)

for res in result:
    print(res, "\n")

messages.append (  {"role": "assistant", "content":result[0]})
```json
{
  "analogies": [
    {
      "collagen": "provides structure and support to tissues",
      "Glass Bead Game": "manipulates symbols and ideas to create a holistic understanding"
    }
  ],
  "differences": [
    {
      "collagen": "a protein found in the bodies of land-dwelling creatures",
      "Glass Bead Game": "a fictional game created by Hermann Hesse"
    }
  ],
  "definitions": [
    {
      "collagen": "a protein that provides strength and support to skin, bones, tendons, and other connective tissues",
      "Glass Bead Game": "a philosophical and intellectual exploration involving the manipulation of symbols and ideas"
    }
  ]
}

result,messages=generate_response (model, tokenizer, text_input="Use this to propose a new bio-inspired design for a composite, incorporating all concepts.",num_return_sequences=1, 
                             temperature=0.3,max_new_tokens=1024 , messages=messages)

for res in result:
    print(res, "\n")

messages.append (  {"role": "assistant", "content":result[0]})  

Benchmark

Performance on the bioinspired benchmark (testing with domain questions to assess performance relative to the training corpus around biological materials and spider silk):

Citation

Please cite as:

@article{Buehler2024DomainLLMs,
  title={Experimental analyses of fine-tuning large language models for domain adaptation for emergent capabilitiesy},
  author={Markus J. Buehler, et al.},
  journal={arXiv preprint arXiv:XXX.YYYYY},
  year={2024}
}

@article{Buehler2024DomainLLMs,
  title={Fine-tuned domain-focused language modeling for applications in scientific analysis, ideation and discovery},
  author={Markus J. Buehler, et al.},
  journal={arXiv preprint arXiv:XXX.YYYYY},
  year={2024}
}

@article{LuuBuehler2023bioinspiredLLM,
  title={BioinspiredLLM: Conversational Large Language Model for the Mechanics of Biological and Bio-Inspired Materials},
  author={Rachel K. Luu and Markus J. Buehler},
  year={2023},
  journal={Advanced Science},
  doi={10.1002/advs.202306724},
  eprint={2309.08788},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2309.08788}
}

@article{Buehler2024XLoRA,
  title={X-LoRA: Mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular design},
  author={Eric L. Buehler and Markus J. Buehler},
  year={2024},
  journal={APL Machine Learning},
  volume={2},
  number={2},
  pages={026119},
  doi={10.1063/5.0203126},
  note={\url{https://doi.org/10.1063/5.0203126}}
}