ToothXpert Model

ToothXpert is a multimodal AI model for comprehensive dental X-ray (OPG) analysis, combining vision and language understanding for automatic diagnosis and condition detection.

Quick Start

Installation

pip install torch torchvision transformers
pip install opencv-python einops peft medpy
pip install "numpy<2.0"  # Important for compatibility

Download Model

from huggingface_hub import snapshot_download

model_path = snapshot_download(
    repo_id='jeffrey423/ToothXpert',
    local_dir='./ToothXpert_pretrained'
)

Simple Inference

import cv2
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, CLIPImageProcessor
from model.ToothXpert_MOE import ToothXpertForCausalLMMOE
from model.llava import conversation as conversation_lib
from model.llava.mm_utils import tokenizer_image_token
from model.segment_anything.utils.transforms import ResizeLongestSide
from utils.utils import (DEFAULT_IM_END_TOKEN, DEFAULT_IM_START_TOKEN,
                         DEFAULT_IMAGE_TOKEN, IMAGE_TOKEN_INDEX)

# Preprocessing function
def preprocess(x, pixel_mean=torch.Tensor([123.675, 116.28, 103.53]).view(-1, 1, 1),
               pixel_std=torch.Tensor([58.395, 57.12, 57.375]).view(-1, 1, 1), img_size=1024):
    x = (x - pixel_mean) / pixel_std
    h, w = x.shape[-2:]
    padh = img_size - h
    padw = img_size - w
    x = F.pad(x, (0, padw, 0, padh))
    return x

# Load model
model_path = "./ToothXpert_pretrained"
device = "cuda:0"

tokenizer = AutoTokenizer.from_pretrained(
    model_path,
    model_max_length=512,
    padding_side="right",
    use_fast=False,
)
tokenizer.pad_token = tokenizer.unk_token
tokenizer.add_tokens("[SEG]")
seg_token_idx = tokenizer("[SEG]", add_special_tokens=False).input_ids[0]
tokenizer.add_tokens([DEFAULT_IM_START_TOKEN, DEFAULT_IM_END_TOKEN], special_tokens=True)

moe_lora_args = {
    "lora_r": 8,
    "lora_alpha": 16,
    "lora_dropout": 0.05,
    "lora_target_modules": "q_proj,v_proj",
    "moe_lora": False,
    "expert_num": 3,
    "guide": True,
    "guide_mode": "smmulsm",
    "vocab_size": len(tokenizer),
}

model = ToothXpertForCausalLMMOE.from_pretrained(
    model_path,
    low_cpu_mem_usage=True,
    vision_tower="openai/clip-vit-large-patch14",
    seg_token_idx=seg_token_idx,
    torch_dtype=torch.bfloat16,
    train_mask_decoder=True,
    out_dim=256,
    moe_lora_args=moe_lora_args,
)

model.config.eos_token_id = tokenizer.eos_token_id
model.config.bos_token_id = tokenizer.bos_token_id
model.config.pad_token_id = tokenizer.pad_token_id

model.get_model().initialize_vision_modules(model.get_model().config)
vision_tower = model.get_model().get_vision_tower()
vision_tower.to(dtype=torch.bfloat16, device=device)

model = model.bfloat16().to(device)
model.eval()

# Load and process image
image_path = "your_dental_xray.png"
image_np = cv2.imread(image_path)
image_np = cv2.cvtColor(image_np, cv2.COLOR_BGR2RGB)
original_size_list = [image_np.shape[:2]]

clip_image_processor = CLIPImageProcessor.from_pretrained("openai/clip-vit-large-patch14")
transform = ResizeLongestSide(1024)

image_clip = (
    clip_image_processor.preprocess(image_np, return_tensors="pt")["pixel_values"][0]
    .unsqueeze(0).to(device).bfloat16()
)

image = transform.apply_image(image_np)
resize_list = [image.shape[:2]]
image = (
    preprocess(torch.from_numpy(image).permute(2, 0, 1).contiguous())
    .unsqueeze(0).to(device).bfloat16()
)

# Prepare prompt
question = "Can you describe the image for me?"
conv = conversation_lib.conv_templates["llava_v1"].copy()
conv.messages = []
prompt = DEFAULT_IMAGE_TOKEN + "\n" + question
prompt = prompt.replace(DEFAULT_IMAGE_TOKEN,
                       DEFAULT_IM_START_TOKEN + DEFAULT_IMAGE_TOKEN + DEFAULT_IM_END_TOKEN)

conv.append_message(conv.roles[0], prompt)
conv.append_message(conv.roles[1], "")
prompt = conv.get_prompt()

input_ids = tokenizer_image_token(prompt, tokenizer, return_tensors="pt")
input_ids = input_ids.unsqueeze(0).to(device)

# Run inference
with torch.no_grad():
    output_ids, pred_masks = model.evaluate(
        image_clip,
        image,
        input_ids,
        resize_list,
        original_size_list,
        max_new_tokens=512,
        tokenizer=tokenizer,
    )

output_ids = output_ids[0][output_ids[0] != IMAGE_TOKEN_INDEX]
text_output = tokenizer.decode(output_ids, skip_special_tokens=False)
text_output = text_output.split('ASSISTANT:')[-1].replace('</s>', '').strip()

print(f"Question: {question}")
print(f"Answer: {text_output}")

Example Questions

General Description:

• "Can you describe the image for me?"

Specific Conditions:

• "Is there any amalgam restorations in the image?"
• "Any R/L suggestive of caries present?"
• "Is there any dental implant present?"
• "Is there any root canal treated teeth?"

Supported Conditions

ToothXpert can detect 11 dental conditions:

1. Amalgam restorations
2. Caries (R/L)
3. Crestal bone loss (mandible)
4. Crestal bone loss (maxillary)
5. Implant-supported bridge
6. Dental implant
7. Metallic/non-metallic post
8. Non-metallic restorations
9. Periapical radiolucency
10. Root canal treated teeth
11. Tooth-supported bridge

Requirements

• GPU: NVIDIA GPU with at least 16GB VRAM
• Python: 3.11 (recommended)
• CUDA: 12.1 or compatible

Model Details

• Base Model: LLaVA-1.5-7B
• Vision Encoder: CLIP ViT-L/14
• Segmentation: SAM (Segment Anything Model) ViT-H
• Adaptation: Guided Mixture of LoRA Experts (G-MoLE)
• Model Size: ~15GB

Citation

If you use ToothXpert in your research, please cite:

@article{liu2026toothxpert,
  title={Developing and Evaluating Multimodal Large Language Model for Orthopantomography Analysis to Support Clinical Dentistry},
  author={Liu, Xinyu and Hung, Kuo Feng and Yu, Weihao and Ng, Ray Anthony W T and Li, Wuyang and Niu, Tianye and Chen, Hui and Yuan, Yixuan},
  journal={Cell Reports Medicine},
  year={2026}
}

Links

• GitHub Repository: CUHK-AIM-Group/ToothXpert
• Dataset: jeffrey423/ToothXpert.MM-OPG-Annotations

License

Apache License 2.0