KielEmbed-Code
A specialized dense embedding model fine-tuned from microsoft/codebert-base for code and text representation.
This is a sentence-transformers model fine-tuned from microsoft/codebert-base. It maps sentences and code blocks into a 768-dimensional dense vector space optimized for semantic textual similarity, semantic search, and clustering tasks.
Model Details
Model Description
- Model Type: Sentence Transformer / Dense Embedding Backbone
- Base Model: microsoft/codebert-base
- Maximum Sequence Length: 512 tokens
- Output Dimensionality: 768 dimensions
- Similarity Function: Cosine Similarity
- Supported Modality: Text & Code
Model Sources
- Documentation: Sentence Transformers Documentation
- Repository: Sentence Transformers on GitHub
- Hugging Face Hub: kiel/KielEmbed-Code
Full Model Architecture
SentenceTransformer(
(0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'RobertaModel'})
(1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)
UsageDirect Usage (Sentence Transformers)First, install the Sentence Transformers library:Bashpip install -U sentence-transformers
Then load your model and run inference:Pythonfrom sentence_transformers import SentenceTransformer
# Load your custom fine-tuned CodeBERT model from the Hugging Face Hub
model = SentenceTransformer("kiel/KielEmbed-Code")
# Run inference
sentences = [
'" Any decision on Charleroi will have huge implications for regional airports in France , " he said .',
'" A bad decision on Charleroi would have huge implications for state-owned regional airports in France .',
"He said the ferry 's crew will be interviewed and tested for drugs and alcohol .",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
Training DetailsTraining DatasetKiel Code & Semantic CorpusSize: 10,000 training samplesColumns: text1, text2, and labelApproximate Token Statistics (First 100 samples):Text 1: Min: 12 tokens | Mean: 27.7 tokens | Max: 41 tokensText 2: Min: 15 tokens | Mean: 27.64 tokens | Max: 46 tokensLabel Distribution: Class 0 (~34.62%), Class 1 (~65.38%)Loss Function: CosineSimilarityLoss with parameters:JSON{
"loss_fct": "torch.nn.modules.loss.MSELoss",
"cos_score_transformation": "torch.nn.modules.linear.Identity"
}
Training HyperparametersPer Device Train Batch Size: 8Gradient Accumulation Steps: 4 (Effective batch size = 32)Learning Rate: 2e-05Number of Epochs: 1Warmup Steps: 0.1Mixed Precision: FP16 EnabledOptimizer: adamw_torch_fusedTraining LogsEpochStepTraining Loss0.165028.07520.3210024.76680.4815021.00550.6420026.55380.825026.06000.9630026.3027Total Training Time: 5.1 minutesFramework VersionsPython: 3.13.15Sentence Transformers: 5.7.0Transformers: 5.16.1PyTorch: 2.11.0+cu128Accelerate: 1.14.0Datasets: 4.8.5Tokenizers: 0.23.1Additional ResourcesTraining and Finetuning Embedding Models with Sentence Transformers: End-to-end guide for fine-tuning Sentence Transformer models.CitationBibTeXCode snippet@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "[https://arxiv.org/abs/1908.10084](https://arxiv.org/abs/1908.10084)",
}