EmbeddingGemma 300M Norwegian Health Domain Adapted

This is a domain-adapted version of Google's EmbeddingGemma-300M model, fine-tuned specifically for Norwegian health and medical information retrieval tasks.

Model Description

Base Model: google/embeddinggemma-300m
Architecture: Gemma-based encoder with pooling and normalization layers
Embedding Dimension: 768
Max Sequence Length: 2048 tokens
Language: Norwegian (with English support)
Domain: Health, medical, and general Norwegian text

Training Data

The model was fine-tuned on a curated dataset of Norwegian question-answer pairs focusing on health and medical information:

• ETI Medical Data: 77,310 training pairs from Norwegian health documents
• NorQuAD: Norwegian reading comprehension questions
• ScandiQA: Scandinavian question-answering dataset
• Supervised Domain Adaptation: Curated Norwegian QA pairs

Total Training Samples: ~80,000 (anchor, positive) pairs
Validation: 788 pairs
Test: 790 pairs

Performance

The model demonstrates excellent retrieval performance on Norwegian health information:

Training Configuration

• Learning Rate: 5e-6 (conservative for domain adaptation)
• Epochs: 1.0
• Batch Size: 32
• Loss Function: CachedMultipleNegativesRankingLoss
• Mini Batch Size: 16 (for gradient caching)
• Training Time: ~3.6 hours on H100

Usage

Basic Usage

from sentence_transformers import SentenceTransformer

# Load the model
model = SentenceTransformer('thivy/embeddinggemma-300m-norwegian-health')

# Encode text
texts = [
    "Hva er symptomene på diabetes?",
    "Diabetes kan gi tørste, hyppig vannlating og tretthet."
]
embeddings = model.encode(texts)

# Calculate similarity
similarity = model.similarity(embeddings[0], embeddings[1])
print(f"Similarity: {similarity:.4f}")

Information Retrieval

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('thivy/embeddinggemma-300m-norwegian-health')

# Example: Medical FAQ retrieval
queries = ["Hva er symptomene på influensa?"]
documents = [
    "Influensa gir feber, hodepine og muskelsmerter.",
    "Diabetes kan føre til økt tørste og hyppig vannlating.",
    "Høyt blodtrykk kan være årsak til hodepine."
]

# Encode
query_embeddings = model.encode(queries)
doc_embeddings = model.encode(documents)

# Find most relevant document
similarities = np.dot(query_embeddings, doc_embeddings.T)
best_match_idx = similarities.argmax()
print(f"Best match: {documents[best_match_idx]}")

RAG Pipeline Integration

from sentence_transformers import SentenceTransformer
import faiss
import numpy as np

model = SentenceTransformer('thivy/embeddinggemma-300m-norwegian-health')

# Create vector database
documents = [...]  # Your Norwegian health documents
doc_embeddings = model.encode(documents)

# Create FAISS index
index = faiss.IndexFlatIP(768)  # Inner product for cosine similarity
index.add(doc_embeddings.astype('float32'))

# Query
query = "Hva er behandling for høyt blodtrykk?"
query_embedding = model.encode([query])

# Search
k = 5  # Top 5 results
scores, indices = index.search(query_embedding.astype('float32'), k)

# Get relevant documents
relevant_docs = [documents[i] for i in indices[0]]

Technical Details

Architecture Components:

1. Transformer Encoder: Gemma-based language model
2. Pooling Layer: Mean pooling of token representations
3. Dense Layers: Two linear projection layers
4. Normalization: L2 normalization for cosine similarity

Training Approach:

• Domain adaptation rather than full fine-tuning
• Conservative learning rate to prevent catastrophic forgetting
• CachedMultipleNegativesRankingLoss for efficient contrastive learning
• InformationRetrievalEvaluator for retrieval-specific metrics

Intended Use

This model is specifically designed for:

• Norwegian health information retrieval
• Medical question-answering systems
• RAG pipelines for healthcare applications
• Semantic search in Norwegian medical documents
• Clinical decision support tools

Limitations

• Optimized primarily for Norwegian text (limited multilingual capability)
• Domain-specific to health/medical content
• May not perform optimally on other domains without further adaptation
• Requires sufficient context for optimal performance (not suitable for very short texts)

Citation

If you use this model, please cite:

@misc{embeddinggemma-300m-norwegian-health,
    title={EmbeddingGemma 300M Norwegian Health Domain Adapted},
    author={Thivyesh},
    year={2026},
    url={https://huggingface.co/thivy/embeddinggemma-300m-norwegian-health}
}

License

This model inherits the license from the base EmbeddingGemma model. Please refer to Google's licensing terms for EmbeddingGemma.