DeBERTa-v3-Small CEFR Level Predictor (LoRA Merged)

This model is a fine-tuned version of microsoft/deberta-v3-small designed to predict English language proficiency levels according to the Common European Framework of Reference for Languages (CEFR). It classifies input text into one of six levels: A1, A2, B1, B2, C1, and C2.

The model was trained using LoRA (Low-Rank Adaptation) and subsequently merged into the base model to ensure easy deployment and optimized inference performance.

Dataset Information

The training process utilized a diverse collection of high-quality CEFR-labeled datasets to ensure robustness across various writing styles and complexities:

• UniversalCEFR Collection:
  • UniversalCEFR/cefr_sp_en: Sentence-level CEFR annotations.
  • UniversalCEFR/cefr_asag_en: Automated Short Answer Grading data mapped to CEFR levels.
  • UniversalCEFR/elg_cefr_en: Data from the European Language Grid.
• Kaggle Sources:
  • Curated benchmarks from CEFR Classification on Kaggle.

Quick Usage (Inference)

You can easily integrate this model using the transformers library:

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
import torch.nn.functional as F

tokenizer = AutoTokenizer.from_pretrained("oguzhandurmaz/deberta-v3-small-cefr-lora-merged")
model = AutoModelForSequenceClassification.from_pretrained("oguzhandurmaz/deberta-v3-small-cefr-lora-merged")

text = "The economic infrastructure is undergoing a significant transformation."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    logits = model(**inputs).logits
    probs = F.softmax(logits, dim=-1)
    predicted_class = torch.argmax(probs, dim=-1).item()

labels = ['A1', 'A2', 'B1', 'B2', 'C1', 'C2']
print(f"Predicted CEFR Level: {labels[predicted_class]}")

Training Hyperparameters

• Architecture: DeBERTa-v3-Small
• Method: LoRA (PEFT)
• Rank (r): 16
• Alpha: 32
• Target Modules: query_proj, value_proj
• Optimizer: AdamW
• Scheduler: Cosine Learning Rate Scheduler
• Max Sequence Length: 512 tokens

Known Limitations

• Short Inputs: While the model is highly accurate for sentences and paragraphs, performance on single words or very short fragments may vary.
• Contextual Bias: The model evaluates complexity based on grammar and vocabulary typical of academic and formal datasets.

ONNX Deployment (Quantized INT8)

This model has been converted to the ONNX Runtime format and quantized to INT8 precision. This optimization allows for significantly faster inference speeds and reduced memory footprint, making it ideal for production environments and CPU-only servers.

Repository Structure

The optimized files are located in the onnx/ directory:

• model_quantized.onnx: The core model weights in INT8 format.
• tokenizer.json & tokenizer_config.json: Vocabulary and tokenizer settings.
• config.json: Model architecture details and CEFR label mapping.
• spm.model: SentencePiece model for DeBERTa-based tokenization.

Usage (Python)

To run the optimized version, you need the onnxruntime and transformers libraries:

import onnxruntime as ort
from transformers import AutoTokenizer
import numpy as np

# Load Model and Tokenizer
model_id = "oguzhandurmaz/deberta-v3-small-cefr-lora-merged"
onnx_path = "onnx/model_quantized.onnx"

tokenizer = AutoTokenizer.from_pretrained(model_id)
# Initialize ONNX Runtime Session
session = ort.InferenceSession(onnx_path, providers=['CPUExecutionProvider'])

def predict(text):
    # Tokenize input
    inputs = tokenizer(text, return_tensors="np")
    # Convert inputs to int64 (required by ONNX)
    onnx_inputs = {k: v.astype(np.int64) for k, v in inputs.items()}
    
    # Run Inference
    logits = session.run(None, onnx_inputs)[0]
    
    # Softmax to get probabilities
    probs = np.exp(logits) / np.sum(np.exp(logits), axis=-1)
    
    # CEFR Label Map
    labels = ['A1', 'A2', 'B1', 'B2', 'C1', 'C2']
    return {
        "label": labels[np.argmax(probs)],
        "confidence": float(np.max(probs))
    }

# Example Usage
result = predict("The economic ramifications of the policy were far-reaching and complex.")
print(f"Predicted Level: {result['label']} (Confidence: {result['confidence']:.2f})")