AtmicQuoterv3

Fine-tuned from BAAI/bge-small-en-v1.5 (MultipleNegativesRankingLoss, 3 epochs, batch size 16, lr 2e-5) on the combined AtmicQuoterv1 + AtmicQuoterv3 training data, for exact-citation retrieval over Sri Ramana Maharshi's teachings. Queries are encoded with the BGE query instruction (Represent this sentence for searching relevant passages: ); passages are not.

Benchmark: base vs. AtmicQuoterv3 (alone vs. hybrid BM25)

Validation set: 1,006 queries (48 from the root validation.jsonl + 958 from AtmicQuoterv1's validation split). Corpus: 8,742 unique passages (AtmicQuoterv1's corpus and the full citation/RAG_citation.json corpus, after the text normalization described below).

"alone" = dense retrieval only, ranking the full corpus. "hybrid BM25" = AtmicQuoterv3 blended with BM25 (0.7 model / 0.3 BM25, min-max normalized per query), no reranker.

Training data

Sources (5,655 training rows): AtmicQuoterv1's training set (3,830 rows) + AtmicQuoterv3's training set (1,825 rows: layer2 rows whose positive contains a citation/RAG_citation.json chunk verbatim, plus root book/page-cited rows; negatives are hard negatives mined from the corpus).

Text normalization applied to every chunk (train positives/negatives, validation positives, corpus): the leading [source tag] block is removed and the trailing | Topics: ... metadata is removed, e.g. Verse 30: All ego gone, ... the Self. | Topics: ego-extinction, self-abidance becomes Verse 30: All ego gone, ... the Self.. Whitespace is normalized and exact duplicates are dropped.

Format: JSON Lines, one training example per line, with three negatives per row. At training time query becomes the anchor (BGE query instruction prepended), pos[0] the positive and neg[0..2] the negatives.

Training record from AtmicQuoterv1:

{"query": "What is Jnana according to Ramana Maharshi?", "pos": ["This knowledge that there is nothing but God or the Self... is Jnana"], "neg": ["Verse 30: All ego gone, Living as That alone Is penance good for growth, Sings Ramana, the Self.", "Bhagavan Sri Ramana Maharshi, born Venkataraman Iyer, was the second of four children born to a devout family in Tiruchuzhi, Tamil Nadu, India.", "Everything happens according to each individual’s karma"], "register": null}

Training record from AtmicQuoterv3 (register marks the v3 rows; v1 rows have null):

{"query": "What is Bhakti & Jnana? Are they two different or Interrelated?", "pos": ["Since God is verily the Self, love of the Self is love of God; and that is bhakti"], "neg": ["Jnana and bhakti are thus one and the same", "Bhakti is jnana mata", "In that state bhakti is no other than jnana, jnana nothing else but bhakti"], "register": "v3_train"}

Validation record (query + gold passage pos) and corpus record (id is a hash of the normalized text):

{"query": "What should I focus on whenever I have free time during the day?", "pos": ["The Self is the real book, you can glance anywhere in that book; nobody can take it away from you. Whenever you are free turn toward the Self. Thereafter you may read whatever you like."]}
{"id": "p_...", "text": "Everything happens according to each individual’s karma"}

SentenceTransformer based on BAAI/bge-small-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-small-en-v1.5. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-small-en-v1.5
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    'Represent this sentence for searching relevant passages: unstable soul notion how to rest in silence Ramana',
]
documents = [
    'The notion ‘I am an unstable soul’ has arisen by letting go one’s immovable Nature. The sadhaka should cast off this notion and rest in the Supreme Silence',
    'There is no audio recording of Bhagavan Sri Ramana Maharshi. When devotees suggested broadcasting his voice, he replied that his “voice is silence” and could not be captured on a medium.',
    'Everything appears to be the same for the Realized Soul',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 384] [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.6570, 0.2615, 0.2363]])

Evaluation

See "Benchmark: base vs. AtmicQuoterv3" above.

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

Training Details

Training Dataset

• Size: 5,655 training samples (see "Training data" above for the format and example records)
• Columns: anchor, positive, negative_1, negative_2, and negative_3
• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• learning_rate: 2e-05
• warmup_steps: 0.1
• weight_decay: 0.01
• load_best_model_at_end: True

All Hyperparameters

• per_device_train_batch_size: 16
• num_train_epochs: 3.0
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Time

• Training: 18.1 minutes
• Evaluation: 1.1 minutes
• Total: 19.2 minutes

Framework Versions

• Python: 3.11.9
• Sentence Transformers: 5.6.0
• Transformers: 5.12.1
• PyTorch: 2.12.1
• Accelerate: 1.14.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}