adityarajsahu/LaTeX-OCR
A compact image-to-LaTeX model built by merging:
- Vision encoder:
openai/clip-vit-base-patch32 - Language decoder:
EleutherAI/pythia-70m - Projector: a small MLP mapping CLIP patch embeddings into Pythia's embedding space
Fine-tuned on lukbl/LaTeX-OCR-dataset
to convert images of rendered math formulas into their LaTeX source.
Usage
This model uses custom (trust_remote_code-free) classes registered in this
repo; load it via the project's LaTeXOCRModel, or copy src/model.py
into your own code:
from transformers import AutoTokenizer, CLIPImageProcessor
from src.model import LaTeXOCRModel
from PIL import Image
repo_id = "adityarajsahu/LaTeX-OCR"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
image_processor = CLIPImageProcessor.from_pretrained(repo_id)
model = LaTeXOCRModel.from_pretrained(repo_id)
image = Image.open("formula.png").convert("RGB")
pixel_values = image_processor(images=image, return_tensors="pt")["pixel_values"]
generated_ids = model.generate(pixel_values=pixel_values, tokenizer=tokenizer, max_new_tokens=192)
print(tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0])
Evaluation
See eval_report_test.json in this repo for BLEU-4, exact-match rate, and
normalized edit-distance on a held-out split, plus sample predictions.
Training details
Trained with src/train.py in the accompanying project repository — see
configs/config.yaml there for the exact hyperparameters used.