llava-med-v1.5-mistral-7b-hf
This repository contains a drop-in, Hugging Face–compatible checkpoint converted from
https://huggingface.co/microsoft/llava-med-v1.5-mistral-7b.
You can load it with the exact same code you use for the original model—no extra conversion steps required.
✅ Clarification on Vocab Size Expansion & Weight Integrity
You can refer to the original conversion code we provided to access the relevant implementation details and code snippets. You may wonder if the vocab size mismatch (e.g., 32000 vs. 32064) breaks the original weights by causing dimension mismatches between embed_tokens/lm_head and the original weights.
The answer is no. Below are key clarifications:
1. Does expanding embed_tokens affect the attention mechanism?
No. The embed_tokens layer is simply an embedding lookup table (nn.Embedding) that maps token IDs to vectors.
The attention mechanism (e.g., nn.MultiheadAttention or LlamaAttention) does not operate on this lookup table directly. It only cares if the dimension of input hidden states is consistent.
Thus, even if you add more tokens, as long as the hidden_size remains unchanged, the weight shape of the attention layer is not affected at all.
2. Which layers actually change after expansion?
| Module | Weight Shape Change | Affects Attention Calculation? |
|---|---|---|
| embed_tokens | [vocab_size+2, hidden_size] | ❌ No |
| lm_head | [vocab_size+2, hidden_size] | ❌ No |
| All attention layers | No shape change | ✅ Completely unchanged |
Analogy for easier understanding
Think of the model as a dictionary:
- embed_tokens is the "new word list": You add two new words, but the length of each word’s explanation (hidden_size) stays the same.
- Attention is the "reading rule": It only focuses on how the vectors of each word in a sentence interact with each other, not how many words are in the dictionary.
The only "change" lies in input distribution
- The original model never encountered embeddings for tokens like
<image>or<pad>. - Now these tokens are initialized, and their new vectors appear during the first forward pass.
- This is a data-level change, not damage to the model’s parameters.
Summary
Expanding the vocab only changes the "dictionary size", not the "reading rules". The weight shape and calculation logic of the attention mechanism remain completely unchanged. The output layer (lm_head) changes in shape but not in functional logic.
3. What exactly changes in the output layer (lm_head)?
The line below modifies the weight matrix of lm_head (the language modeling head):
model.resize_token_embeddings(config.text_config.vocab_size + 2, pad_shape)
Quick Start
from transformers import LlavaForConditionalGeneration, AutoProcessor
import torch
model_path = "chaoyinshe/llava-med-v1.5-mistral-7b-hf"
model = LlavaForConditionalGeneration.from_pretrained(
model_path,
torch_dtype=torch.bfloat16,
attn_implementation="flash_attention_2", # requires FA2
device_map="auto" # multi-GPU ready
)
processor = AutoProcessor.from_pretrained(model_path)
# Example inference
messages = [
{
"role": "user",
"content": [
{"type": "image"},
{"type": "text", "text": "What is the main finding in this chest X-ray?"}
]
}
]
prompt = processor.tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
inputs = processor(
images=[image], text=prompt, return_tensors="pt"
).to(model.device, torch.bfloat16)
with torch.inference_mode():
out = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(out[0], skip_special_tokens=True))
✅ Training Screenshot
(Note: The image below is for illustrative purposes only — actual training metrics may vary.) 🤗
