SentenceTransformer based on answerdotai/ModernBERT-base

This is a sentence-transformers model finetuned from answerdotai/ModernBERT-base on the code-retrieval-combined-v2 dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: answerdotai/ModernBERT-base
• Maximum Sequence Length: 1024 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • code-retrieval-combined-v2

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False, 'architecture': 'OptimizedModule'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("modernbert-code-v2")
# Run inference
queries = [
    "If MultiTenantMiddleware is used, filter queryset by request.site_id",
]
documents = [
    "def get_queryset(self):\n        '''\n        If MultiTenantMiddleware is used, filter queryset by request.site_id\n        '''\n        queryset = super(PageList, self).get_queryset()\n        if hasattr(self.request, 'site_id'):\n            queryset = queryset.filter(site_id=self.request.site_id)\n        return queryset",
    'def reduce_ticks(ax, which, maxticks=3):\n    """Given a pyplot axis, resamples its `which`-axis ticks such that are at most\n    `maxticks` left.\n\n    Parameters\n    ----------\n    ax : axis\n        The axis to adjust.\n    which : {\'x\' | \'y\'}\n        Which axis to adjust.\n    maxticks : {3, int}\n        Maximum number of ticks to use.\n\n    Returns\n    -------\n    array\n        An array of the selected ticks.\n    """\n    ticks = getattr(ax, \'get_{}ticks\'.format(which))()\n    if len(ticks) > maxticks:\n        # make sure the left/right value is not at the edge\n        minax, maxax = getattr(ax, \'get_{}lim\'.format(which))()\n        dw = abs(maxax-minax)/10.\n        start_idx, end_idx = 0, len(ticks)\n        if ticks[0] < minax + dw:\n            start_idx += 1\n        if ticks[-1] > maxax - dw:\n            end_idx -= 1\n        # get reduction factor\n        fac = int(len(ticks) / maxticks)\n        ticks = ticks[start_idx:end_idx:fac]\n    return ticks',
    'function (isPublic, name, data, ttl, published_at, coreid) {\n        var rawFn = function (msg) {\n            try {\n                msg.setMaxAge(parseInt((ttl && (ttl >= 0)) ? ttl : 60));\n                if (published_at) {\n                    msg.setTimestamp(moment(published_at).toDate());\n                }\n            }\n            catch (ex) {\n                logger.error("onCoreHeard - " + ex);\n            }\n            return msg;\n        };\n\n        var msgName = (isPublic) ? "PublicEvent" : "PrivateEvent";\n        var userID = (this.userID || "").toLowerCase() + "/";\n        name = (name) ? name.toString() : name;\n        if (name && name.indexOf && (name.indexOf(userID) == 0)) {\n            name = name.substring(userID.length);\n        }\n\n        data = (data) ? data.toString() : data;\n        this.sendNONTypeMessage(msgName, { event_name: name, _raw: rawFn }, data);\n    }',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.9183, -0.0231, -0.0561]])

Evaluation

Metrics

Information Retrieval

• Dataset: eval
• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

code-retrieval-combined-v2

• Dataset: code-retrieval-combined-v2 at 2b971a6
• Size: 283,621 training samples
• Columns: query and positive
• Approximate statistics based on the first 1000 samples:

  	query	positive
  type	string	string
  details	min: 5 tokensmean: 44.94 tokensmax: 856 tokens	min: 30 tokensmean: 181.2 tokensmax: 1024 tokens

• Samples:

  query	positive
  Start the asyncio event loop and runs the application.	def main(): """Start the asyncio event loop and runs the application.""" # Helper method so that the coroutine exits cleanly if an exception # happens (which would leave resources dangling) async def _run_application(loop): try: return await cli_handler(loop) except KeyboardInterrupt: pass # User pressed Ctrl+C, just ignore it except SystemExit: pass # sys.exit() was used - do nothing except: # pylint: disable=bare-except # noqa import traceback traceback.print_exc(file=sys.stderr) sys.stderr.writelines( '\n>>> An error occurred, full stack trace above\n') return 1 try: loop = asyncio.get_event_loop() return loop.run_until_complete(_run_application(loop)) except KeyboardInterrupt: pass return 1
  Initialize the pool manager with the number of pools, the entry sizes for eachpool, and the maximum depth of the free pool.@param bufferEntrySizes the memory sizes of each entry in the pools@param bufferEntryDepths the maximum number of entries in the free pool	public void initialize(int[] bufferEntrySizes, int[] bufferEntryDepths) { if (TraceComponent.isAnyTracingEnabled() && tc.isEntryEnabled()) { Tr.entry(tc, "initialize"); } // order both lists from smallest to largest, based only on Entry Sizes int len = bufferEntrySizes.length; int[] bSizes = new int[len]; int[] bDepths = new int[len]; int sizeCompare; int depth; int sizeSort; int j; for (int i = 0; i < len; i++) { sizeCompare = bufferEntrySizes[i]; depth = bufferEntryDepths[i]; // go backwards, for speed, since first Array List is // probably already ordered small to large for (j = i - 1; j >= 0; j--) { sizeSort = bSizes[j]; if (sizeCompare > sizeSort) { // add the bigger one after the smaller one bSizes[j + 1] = sizeCompare; bDepths[j ...
  // List lists all of the documents in an index. The documents are returned in// increasing ID order.	func (x *Index) List(c context.Context, opts *ListOptions) *Iterator { t := &Iterator{ c: c, index: x, count: -1, listInclusive: true, more: moreList, limit: -1, } if opts != nil { t.listStartID = opts.StartID if opts.Limit > 0 { t.limit = opts.Limit } t.idsOnly = opts.IDsOnly } return t}

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 128,
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

code-retrieval-combined-v2

• Dataset: code-retrieval-combined-v2 at 2b971a6
• Size: 31,516 evaluation samples
• Columns: query and positive
• Approximate statistics based on the first 1000 samples:

  	query	positive
  type	string	string
  details	min: 5 tokensmean: 42.73 tokensmax: 834 tokens	min: 30 tokensmean: 180.42 tokensmax: 1024 tokens

• Samples:

  query	positive
  This gets the version of OpenALPR :return: Version information	def get_version(self): """ This gets the version of OpenALPR :return: Version information """ ptr = self._get_version_func(self.alpr_pointer) version_number = ctypes.cast(ptr, ctypes.c_char_p).value version_number = _convert_from_charp(version_number) self._free_json_mem_func(ctypes.c_void_p(ptr)) return version_number
  Remove all unnecessary comments from a lexer or parser file	public String stripUnnecessaryComments(String javaContent, AntlrOptions options) { if (!options.isOptimizeCodeQuality()) { return javaContent; } javaContent = stripMachineDependentPaths(javaContent); if (options.isStripAllComments()) { javaContent = stripAllComments(javaContent); } return javaContent; }
  Serialize reply to array or JSON.@param {Object} packet@param {String} packet.method "get", "search", "post", "put", "delete", "sub", "unsub".@param {String} packet.resource@param {String} packet.id@param {*} packet.body@param {Number} [packet.status]@param {Number|String} [packet.date]@param {Object} [packet.headers]@param {Boolean} [json] true to generate JSON instead of array.@returns {Array|String|null}	function reply(packet, json) { return _create(packet, packet.status || 500, (METHODS[packet.method] || '') + packet.resource, json);}

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 128,
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 1024
• per_device_eval_batch_size: 1024
• learning_rate: 8e-05
• num_train_epochs: 1
• warmup_steps: 0.05
• bf16: True
• dataloader_num_workers: 4
• load_best_model_at_end: True
• push_to_hub: True
• hub_model_id: modernbert-code-v2
• batch_sampler: no_duplicates

All Hyperparameters

• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 1024
• per_device_eval_batch_size: 1024
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 8e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.05
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 4
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: True
• resume_from_checkpoint: None
• hub_model_id: modernbert-code-v2
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.3.0
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}