Mistral-7B Academic Metadata Extractor
A fine-tuned Mistral-7B-Instruct-v0.3 model for structured metadata extraction from academic documents.
The model extracts metadata from academic text and produces structured JSON output.
Extracted Metadata
The model extracts the following fields:
- Title
- Keywords
- Authors
- Affiliations
- Publisher
- Journal
- Publication Year
Performance
The fine-tuned model achieved an overall field-level accuracy of 91.8% on the test set.
| Metadata Field | Accuracy |
|---|---|
| Title | 98% |
| Keywords | 85% |
| Authors | 96% |
| Affiliations | 93% |
| Publisher | 90% |
| Journal | 88% |
| Year | 87% |
| Overall | 91.8% |
Training Dataset
A dataset of 10,000 academic PDF documents was prepared for the project.
The final dataset was divided into:
- Training: 9,000 documents
- Validation: 900 documents
- Testing: 100 documents
Approximately 3,000 documents were manually reviewed to complete or correct missing metadata fields.
Training Configuration
| Parameter | Value |
|---|---|
| Base Model | Mistral-7B-Instruct-v0.3 |
| Fine-Tuning Method | LoRA |
| Epochs | 3 (early stopping at 1.97) |
| Batch Size | 2 |
| Learning Rate | 2e-5 |
| Maximum Sequence Length | 1024 tokens |
| GPU | NVIDIA A100 80 GB |
| CPUs per Task | 6 |
| Framework | Unsloth + Hugging Face |
Output Format
The model is designed to return metadata in JSON format:
{
"title": "",
"keywords": [],
"authors": [
{
"name": "",
"affiliations": [""]
}
],
"publisher": "",
"year": "",
"journal": ""
}
Research
This model was developed as part of the research:
Developing AI Agents That Extract Metadata Using The RAG Method With Fine-Tuned Large Language Models
Authors: Tamam Alhasan and Savaş Takan
Affiliation: Artificial Intelligence and Data Engineering Department, Ankara University
The research compared RAG-based metadata extraction with supervised fine-tuning. Mistral-7B-Instruct-v0.3 achieved 74.43% overall accuracy in the RAG setup, while the fine-tuned model improved the overall accuracy to 91.8%.
Intended Use
This model is intended for extracting structured bibliographic metadata from academic and scientific documents.
Potential applications include:
- Academic repositories
- Digital libraries
- Research databases
- Automated document indexing
- Scientific knowledge management
Limitations
The model was trained and evaluated primarily on English-language academic documents. Performance may vary for documents with unusual layouts, incomplete metadata, OCR errors, or domains significantly different from the training data.
Authors
Tamam Alhasan
Artificial Intelligence and Data Engineering
Ankara University, Türkiye