SentenceTransformer based on Qwen/Qwen2.5-0.5B-Instruct

This is a sentence-transformers model finetuned from Qwen/Qwen2.5-0.5B-Instruct. It maps sentences & paragraphs to a 896-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Qwen/Qwen2.5-0.5B-Instruct
• Maximum Sequence Length: 1024 tokens
• Output Dimensionality: 896 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False}) with Transformer model: Qwen2Model 
  (1): Pooling({'word_embedding_dimension': 896, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("AlexWortega/qwen_emb_6k")
# Run inference
sentences = [
    'query: \n\nHuman: 2 cars take 4 hours to drive from Stockholm to Copenhagen. Explain how long it takes for 6 cars to drive the same distance.\n\nAssistant: ',
    "The number of cars doesn't affect the time it takes to travel from Stockholm to Copenhagen, assuming each car is traveling independently and under the same conditions. If it takes one car 4 hours to make the journey, it will also take each of the other cars the same amount of time, 4 hours, to travel the same distance. So, whether you have 2 cars or 6 cars, each car would still take 4 hours to travel from Stockholm to Copenhagen.",
    '6 cars would take 2 hours and 40 minutes to drive from Stockholm to Copenhagen. This is because the total distance remains the same, but with more cars, the time is divided by the number of cars. So, 4 hours divided by 2 cars is 2 hours, and then divided by 3 (since there are now 6 cars) is 40 minutes.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 896]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Datasets: NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
• Evaluated with InformationRetrievalEvaluator

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,580,101 training samples
• Columns: query, response, and negative
• Approximate statistics based on the first 1000 samples:

  	query	response	negative
  type	string	string	string
  details	min: 14 tokensmean: 149.45 tokensmax: 1024 tokens	min: 2 tokensmean: 249.27 tokensmax: 1024 tokens	min: 1 tokensmean: 174.24 tokensmax: 1024 tokens

• Samples:

  query	response	negative
  query: Human: 0.002 = 1000 1 = x?Assistant:	To find the value of x, we can set up a proportion using the given information:0.002/1000 = 1/xTo solve for x, we can cross multiply:0.002 * x = 1000 * 10.002x = 1000Dividing both sides by 0.002:x = 1000 / 0.002x = 500,000Therefore, 1 is equal to 500,000 in this proportion.	I apologize, but the equation you provided is not correct.0.002 is equal to 0.002, not 1000.1 is equal to x, not a number.Can you please provide a correct equation or problem for me to solve?
  query: Human: 0:00what's going on guys it's NAME_1 from0:01NAME_2 and today we're going to0:03talk about cns fatigue0:05so going over uh cns fatigue central0:07nervous system fatigue it's something0:08that people talk about and bring up a0:10lot0:10there's a bit of science to it and0:12there's also a lot of bro0:13culture myth to it and i want to get0:15into that and i'm i'm going to talk0:17really quick about where it comes from0:19why we talk about it this way and then0:21how it kind of gets used and misused0:23to describe a certain training0:25phenomenon that we all experience0:27so i'm going to start out by kind of0:29fleshing out what we're talking about or0:31how it kind of originally became0:34discussed about so we observe a few0:35things when it comes to weight training0:37specifically with lifting heavy0:38the first thing we observe is that true0:40maximal attempts at 900:42and up rpe nine to ten so that's really0:45hard tri...	Hello! It seems like you're sharing a transcript about Central Nervous System (CNS) fatigue, particularly in relation to weight training. CNS fatigue is a topic that's often discussed in the context of high-intensity workouts, such as weightlifting. The transcript you've shared discusses the concept of CNS fatigue, its origins, and how it's often misused to describe certain training phenomena. It also talks about the effects of lifting heavy weights at maximum capacity in every session, which can lead to short-term regression. This is a common experience among weightlifters who push their limits consistently without giving their bodies adequate time to recover.The speaker also mentions the common practice of spacing out heavy lifting sessions or incorporating deload periods to prevent this regression. The time between the last heavy attempt and a contest day is also noted to be longer, presumably to allow for adequate recovery and prevent CNS fatigue.If you have any specific ques...	0:00in the past0:00with the same lift0:00and the same weight0:00and the same load0:00then0:00if you have a deload day and you come back0:00and you're feeling like you can handle0:00 heavier loads0:00with a little bit more energy0:00then the load is typically a little bit0:01 lighter than the previous day's load0:02and the time between contest day and the0:03last heavy attempt0:03is typically a lot shorter0:04than the time between the previous day's0:05heavy attempt and the contest day0:06and that's a general observation that has0:08also been seen in other strength training0:09styles0:10and it's something that's been uh0:12accounted for in the wisdom of0:13 trainers and coaches over the years0:14and it's a good idea to keep in mind when0:16you're lifting heavy or you're training in0:17a deloads schedule0:18and you have a lot of energy and you're0:20able to handle heavier loads0:21the load is typically a little bit0:22 lighter than...
  query: Human: 0:00what's going on guys it's NAME_1 from0:01NAME_2 and today we're going to0:03talk about cns fatigue0:05so going over uh cns fatigue central0:07nervous system fatigue it's something0:08that people talk about and bring up a0:10lot0:10there's a bit of science to it and0:12there's also a lot of bro0:13culture myth to it and i want to get0:15into that and i'm i'm going to talk0:17really quick about where it comes from0:19why we talk about it this way and then0:21how it kind of gets used and misused0:23to describe a certain training0:25phenomenon that we all experience0:27so i'm going to start out by kind of0:29fleshing out what we're talking about or0:31how it kind of originally became0:34discussed about so we observe a few0:35things when it comes to weight training0:37specifically with lifting heavy0:38the first thing we observe is that true0:40maximal attempts at 900:42and up rpe nine to ten so that's really0:45hard tri...	0:00Hey there! Today we're going to talk about CNS fatigue, or central nervous system fatigue. This is a term that is often used in the fitness community, but there is some confusion and misinformation surrounding it. So let's dive in and discuss what it really means and how it affects our training.First, let's define what we mean by CNS fatigue. This is the idea that our central nervous system, which controls our brain and spinal cord, can become fatigued from intense physical activity. This can manifest as a decrease in performance, strength, and coordination.One of the main observations that has led to the concept of CNS fatigue is that when we lift heavy weights at a high intensity, such as a 9 or 10 on the RPE scale, we tend to see short-term regression in our performance. This means that if we continue to lift at this intensity every session, we may eventually see a decrease in our strength and ability to handle heavy loads.Additionally, we also see that the time between a...	Hi, I'm an AI language model. How can I assist you today?

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• gradient_accumulation_steps: 32
• learning_rate: 2e-05
• max_grad_norm: 0.4
• num_train_epochs: 1
• warmup_ratio: 0.4
• bf16: True
• prompts: {'query': 'query: ', 'answer': 'document: '}
• batch_sampler: no_duplicates

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 32
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 0.4
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.4
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: {'query': 'query: ', 'answer': 'document: '}
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs