SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2 (Physics)

This is a sentence-transformers model finetuned from
sentence-transformers/all-MiniLM-L6-v2.

It maps sentences & paragraphs to a 384-dimensional dense vector space and is optimised for:

• semantic textual similarity in Physics
• semantic search over NCERT Physics-style content
• paraphrase mining, clustering, and downstream classification for physics questions and explanations.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Domain: CBSE / NCERT Class 11–12 Physics text (questions, explanations, summaries)
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False,
                'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False,
                'pooling_mode_mean_sqrt_len_tokens': False,
                'pooling_mode_weightedmean_tokens': False,
                'pooling_mode_lasttoken': False,
                'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

Install the library:

pip install -U sentence-transformers

Then load the model and run inference:

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")

sentences = [
    "Interference of light in Young's double-slit experiment leads to a pattern of bright and dark fringes.",
    "The superposition of coherent light waves from two slits produces constructive and destructive interference on a screen.",
    "Centripetal force is required to keep an object in uniform circular motion and acts towards the centre of the circle.",
]

embeddings = model.encode(sentences)
print(embeddings.shape)  # [3, 384]

# Pairwise similarity
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.7669, -0.0300],
#         [ 0.7669,  1.0000,  0.0155],
#         [-0.0300,  0.0155,  1.0000]])

Training Details

The model was trained in two phases, with a focus on Physics-specific text.

Phase 1 – Contrastive Fine-Tuning (Physics Positives)

Training Dataset

Unnamed internal dataset of Physics pairs:

• Size: 33,038 training samples

• Columns: <sentence_0> and <sentence_1> (semantically similar Physics sentences)

• Approximate token statistics (first 1000 samples):

• Example training pairs (Physics):

• Loss: MultipleNegativesRankingLoss

{
  "scale": 20.0,
  "similarity_fct": "cos_sim",
  "gather_across_devices": false
}

Phase 2 – Hard-Negative Training (NCE on NCERT Physics)

This phase focuses on making the embedding model more discriminative for Retrieval-Augmented Generation (RAG) over NCERT Physics by introducing hard-negative mining and Noise-Contrastive Estimation (NCE) training.

1. Hard-Negative Mining

To expose the model to challenging non-answer passages, we performed systematic hard-negative mining.

Data Preparation

• 810 NCERT Physics chunks embedded and indexed using FAISS.
• 1,859 Physics queries collected as phase-2 seed questions.

Retrieval & Re-Ranking

1. For each query, top-k nearest neighbors were retrieved using FAISS.
2. Retrieved candidates were re-ranked using a cross-encoder:

2. Noise-Contrastive Estimation (NCE) Training

Initialization

• Training started from the Phase-1 encoder checkpoint.

Objective

The model is optimized to:

• Maximize similarity between the query and the gold NCERT chunk.
• Minimize similarity between the query and the hard negative in the embedding space.

This effectively teaches the model to distinguish very similar but incorrect Physics passages from the correct one.

3. Training Performance

Training Metrics

• Train Loss:
  ≈ 2.18 → 0.13 / 0.03

• Train Acc@1:
  ≈ 0.40 → 0.98–1.00

Validation Metrics

• Validation Acc@1 (3 epochs): ≈ 0.81
• Validation Acc@1 (5 epochs): ≈ 0.76

Checkpoint Selection

• The best checkpoint was selected using lowest validation loss
  (early-stopping style criterion).

4. Outcome

This phase significantly sharpens the embedding space so that for NCERT Physics questions:

• The correct Physics chunk is ranked above:
• Semantically similar
• Conceptually misleading
• Structurally related but incorrect passages

This directly improves RAG retrieval precision for downstream QA and tutoring systems.

5. Summary

✅ This phase enables high-precision Physics retrieval under domain-specific confusion, which is critical for exam-grade question answering systems. Evaluation is reported using top-1 classification accuracy on triplets:

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• num_train_epochs: 10
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 100
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.1.2
• Transformers: 4.57.2
• PyTorch: 2.9.0+cu126
• Accelerate: 1.12.0
• Datasets: 4.0.0
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}