BioMistral-7B-Natural-Products-RE-Diversity-1000-synt-v1.2
Natural products represent a large pool of bioactive compounds of high interest in drug-discovery. However, these relationships are sparsely distributed across organisms and a growing part of the literature remains unannotated. This volume necessitates the development of a machine assistant to boost the completion of existing resources. Framing the task as an end-to-end Relation Extraction (RE), we propose a new BioMistral model fined-tuned on synthetic data. See details about this procedure in the join article.
The model is a derived from BioMistral/BioMistral-7B and was trained on a synthetic dataset generated with Mixtral-8x7B-Instruct-v0.1.
The dataset is available on zenodo.
You can use the model as:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
# model and quantization config
device = torch.device("cuda")
BNB_CONFIG = BitsAndBytesConfig(load_in_8bit=True)
model_hf = "mdelmas/BioMistral-7B-Natural-Products-RE-Diversity-1000-synt-v1.2"
# Load model
model = AutoModelForCausalLM.from_pretrained(model_hf, quantization_config=BNB_CONFIG)
tokenizer = AutoTokenizer.from_pretrained(model_hf)
# Example from PubMed article 24048364
title_text = "Producers and important dietary sources of ochratoxin A and citrinin."
abstract_text = "Ochratoxin A (OTA) is a very important mycotoxin, and its research is focused right now on the new findings of OTA, like being a complete carcinogen, information about OTA producers and new exposure sources of OTA. Citrinin (CIT) is another important mycotoxin, too, and its research turns towards nephrotoxicity. Both additive and synergistic effects have been described in combination with OTA. OTA is produced in foodstuffs by Aspergillus Section Circumdati (Aspergillus ochraceus, A. westerdijkiae, A. steynii) and Aspergillus Section Nigri (Aspergillus carbonarius, A. foetidus, A. lacticoffeatus, A. niger, A. sclerotioniger, A. tubingensis), mostly in subtropical and tropical areas. OTA is produced in foodstuffs by Penicillium verrucosum and P. nordicum, notably in temperate and colder zones. CIT is produced in foodstuffs by Monascus species (Monascus purpureus, M. ruber) and Penicillium species (Penicillium citrinum, P. expansum, P. radicicola, P. verrucosum). OTA was frequently found in foodstuffs of both plant origin (e.g., cereal products, coffee, vegetable, liquorice, raisins, wine) and animal origin (e.g., pork/poultry). CIT was also found in foodstuffs of vegetable origin (e.g., cereals, pomaceous fruits, black olive, roasted nuts, spices), food supplements based on rice fermented with red microfungi Monascus purpureus and in foodstuffs of animal origin (e.g., cheese)."
text = title_text + " " + abstract_text
# Tokenisation
input_text = text + tokenizer.eos_token + tokenizer.bos_token
input_tokens = tokenizer(input_text, return_tensors='pt')
input_tokens.to(device)
# Decoding parameters
EVAL_GENERATION_ARGS = {"do_sample": False,
"num_beams": 3,
"length_penalty": 3,
"max_length": 2048,
"temperature": 1,
"forced_eos_token_id": tokenizer.eos_token_id,
"pad_token_id": tokenizer.pad_token_id}
# Generate
with torch.no_grad():
beam_output = model.generate(**input_tokens, **EVAL_GENERATION_ARGS)
output = tokenizer.decode(beam_output[0][len(input_tokens["input_ids"][0]):], skip_special_tokens=True)
# Parse and print
rels = output.strip().split("; ")
for rel in rels:
print("- " + rel)
torch.cuda.empty_cache()