colqwen2-v1.0-hf finetuned on energy document pages

This is a Multi-Vector Encoder model finetuned from vidore/colqwen2-v1.0-hf on the synthetic_doc_qa_energy_train dataset using the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.

Model Details

Model Description

• Model Type: Multi-Vector Encoder
• Base model: vidore/colqwen2-v1.0-hf
• Maximum Sequence Length: 32768 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: maxsim
• Supported Modalities: Text, Image
• Training Dataset:
  • synthetic_doc_qa_energy_train
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Multi-Vector Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Multi-Vector Encoders on Hugging Face

Full Model Architecture

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'retrieval', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'embeddings'}, 'image': {'method': 'forward', 'method_output_name': 'embeddings'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ColQwen2ForRetrieval'})
  (1): MultiVectorMask({'skiplist_words': [], 'keep_only_token_ids': None})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import MultiVectorEncoder

# Download from the 🤗 Hub
model = MultiVectorEncoder("tomaarsen/multivector-colqwen2-v1.0-hf-docqa-energy")
# Run inference: each input becomes a sequence of per-token vectors (variable length).
queries = [
    'What topics are covered in this index?',
]
documents = [
    'https://huggingface.co/tomaarsen/multivector-colqwen2-v1.0-hf-docqa-energy/resolve/main/assets/image_0.jpg',
    'https://huggingface.co/tomaarsen/multivector-colqwen2-v1.0-hf-docqa-energy/resolve/main/assets/image_1.jpg',
    'https://huggingface.co/tomaarsen/multivector-colqwen2-v1.0-hf-docqa-energy/resolve/main/assets/image_2.jpg',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (20, 128) (759, 128)

# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[15.7523,  8.2611, 11.6049]])

Evaluation

Metrics

Multi Vector Information Retrieval

• Dataset: energy-dev
• Evaluated with MultiVectorInformationRetrievalEvaluator

Training Details

Training Dataset

synthetic_doc_qa_energy_train

• Dataset: synthetic_doc_qa_energy_train at 438dd85
• Size: 3,475 training samples
• Columns: query and image
• Approximate statistics based on the first 100 samples:

  	query	image
  type	string	image
  modality	text	image
  details	min: 18 tokensmean: 28.12 tokensmax: 48 tokens	min: 932x312 pxmean: 1717x2057 pxmax: 3200x2339 px

• Samples:

  query	image
  What is the objective of the research task related to reactor pressure vessel steels?
  What recommendations does this study make regarding energy policy options?
  What are the typical materials used for the cathode, electrolyte, and anode in conventional solid-state batteries?

• Loss: MultiVectorMultipleNegativesRankingLoss with these parameters:

  {
      "score_metric": "colbert_scores",
      "scale": 1.0,
      "score_mini_batch_size": null,
      "size_average": true,
      "gather_across_devices": false
  }
  

Evaluation Dataset

synthetic_doc_qa_energy_train

• Dataset: synthetic_doc_qa_energy_train at 438dd85
• Size: 400 evaluation samples
• Columns: query and image
• Approximate statistics based on the first 100 samples:

  	query	image
  type	string	image
  modality	text	image
  details	min: 18 tokensmean: 27.27 tokensmax: 44 tokens	min: 827x1125 pxmean: 1728x2103 pxmax: 3400x3042 px

• Samples:

  query	image
  What topics are covered in this index?
  What are the different funding sources for projects listed in the table?
  What are the main sections covered in this report?

• Loss: MultiVectorMultipleNegativesRankingLoss with these parameters:

  {
      "score_metric": "colbert_scores",
      "scale": 1.0,
      "score_mini_batch_size": null,
      "size_average": true,
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• num_train_epochs: 1
• learning_rate: 2e-05
• warmup_steps: 0.05
• bf16: True
• save_only_model: True
• load_best_model_at_end: True

All Hyperparameters

• per_device_train_batch_size: 8
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.05
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: True
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• max_length: None

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training: 14.2 minutes
• Evaluation: 24.0 minutes
• Total: 38.2 minutes

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 5.7.0.dev0
• Transformers: 5.14.1
• PyTorch: 2.11.0+cu128
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultiVectorMultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}