IL-7B (Immuno-LLM 7 Billion) is a 7 billion parameter LLM trained and merged from Mistral-7B for the domain of clinical rheumatology and immunology. It is a merge of 2 models trained with the same recipe and data, initialized from 2 different weights: the original Mistral-7B weights and the BioMistral-7B weights. Merging was done using mergekit.
Note: IL-7B is an AI tool developed for research and general interest in rheumatology and autoimmune diseases. It has not been validated in and should not be used for direct clinical decision making.
Additional note: an improved version is on its way...
Intended Use
IL-7B uses the same prompt format as Zephyr from HF.
import torch
from transformers import pipeline
pipe = pipeline("text-generation", model="cmcmaster/il_7b", torch_dtype=torch.bfloat16, device_map="auto")
messages = [
{"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipe(prompt, max_new_tokens=1024, do_sample=True, temperature=0.7)
print(outputs[0]["generated_text"])
# <|user|>
# A patient with longstanding psoriasis presents with pain in the hands, particularly first thing in the morning, associated with stiffness. You notice swelling of several metacarpophalangeal joints and both wrists. ESR is 38, CRP is 63 and the rheumatoid factor is weakly positive (31). What is the most likely diagnosis and why?</s>
# <|assistant|>
# The most likely diagnosis is psoriatic arthritis (PsA). The patient has a longstanding history of psoriasis, which is a skin condition characterized by red, scaly patches. The symptoms of pain, stiffness, swelling of the metacarpophalangeal joints, and both wrists are common in psoriatic arthritis. The elevated ESR and CRP levels indicate inflammation, which is also consistent with psoriatic arthritis. The weakly positive rheumatoid factor could be due to the psoriatic arthritis, as it may sometimes occur in patients with this condition.
Merge Details
Merge Method
This model was merged using the SLERP merge method.
Models Merged
The merge was made from two unreleased models:
- rheumistral-sft was trained from the original mistral checkpoint in two stages: 1) "continued pretraining" on a large, curated dataset of rheumatology and immunology texts; 2) supervised finetuning on a combination of synthetic and human generated QA pairs and chat logs
- biorheumistral-sft was trained the same way as rheumistral-sft, only it started from the BioMistral-7B checkpoint.