span-marker-bert-base-multilingual-cased-multinerd

This model is a fine-tuned version of bert-base-multilingual-cased on an Babelscape/multinerd dataset.

Is your data not (always) capitalized correctly? Then consider using the uncased variant of this model instead for better performance: lxyuan/span-marker-bert-base-multilingual-uncased-multinerd.

This model achieves the following results on the evaluation set:

• Loss: 0.0049
• Overall Precision: 0.9242
• Overall Recall: 0.9281
• Overall F1: 0.9261
• Overall Accuracy: 0.9852

Test set results:

• test_loss: 0.005226554349064827,
• test_overall_accuracy: 0.9851129807294873,
• test_overall_f1: 0.9270450073152169,
• test_overall_precision: 0.9281906912835416,
• test_overall_recall: 0.9259021481405626,
• test_runtime: 2690.9722,
• test_samples_per_second: 150.748,
• test_steps_per_second: 4.711

This is a replication of Tom's work. Everything remains unchanged, except that we extended the number of training epochs to 3 for a slightly longer training duration and set the gradient_accumulation_steps to 2. Please refer to the official model page to review their results and training script

Results:

• Special thanks to Tom for creating the evaluation script and generating the results.

Label set

Inference Example

# install span_marker
(env)$ pip install span_marker


from span_marker import SpanMarkerModel

model = SpanMarkerModel.from_pretrained("lxyuan/span-marker-bert-base-multilingual-cased-multinerd")

description = "Singapore is renowned for its hawker centers offering dishes \
like Hainanese chicken rice and laksa, while Malaysia boasts dishes such as \
nasi lemak and rendang, reflecting its rich culinary heritage."

entities = model.predict(description)

entities
>>>
[
  {'span': 'Singapore', 'label': 'LOC', 'score': 0.999988317489624, 'char_start_index': 0, 'char_end_index': 9},
  {'span': 'Hainanese chicken rice', 'label': 'FOOD', 'score': 0.9894770383834839, 'char_start_index': 66, 'char_end_index': 88},
  {'span': 'laksa', 'label': 'FOOD', 'score': 0.9224908947944641, 'char_start_index': 93, 'char_end_index': 98},
  {'span': 'Malaysia', 'label': 'LOC', 'score': 0.9999839067459106, 'char_start_index': 106, 'char_end_index': 114}]

# missed: nasi lemak as FOOD
# missed: rendang as FOOD
# :(

Quick test on Chinese

from span_marker import SpanMarkerModel

model = SpanMarkerModel.from_pretrained("lxyuan/span-marker-bert-base-multilingual-cased-multinerd")

# translate to chinese
description = "Singapore is renowned for its hawker centers offering dishes \
like Hainanese chicken rice and laksa, while Malaysia boasts dishes such as \
nasi lemak and rendang, reflecting its rich culinary heritage."

zh_description = "新加坡因其小贩中心提供海南鸡饭和叻沙等菜肴而闻名, 而马来西亚则拥有椰浆饭和仁当等菜肴，反映了其丰富的烹饪传统."

entities = model.predict(zh_description)

entities
>>>
[
  {'span': '新加坡', 'label': 'LOC', 'score': 0.9282007813453674, 'char_start_index': 0, 'char_end_index': 3},
  {'span': '马来西亚', 'label': 'LOC', 'score': 0.7439665794372559, 'char_start_index': 27, 'char_end_index': 31}]

# It only managed to capture two countries: Singapore and Malaysia.
# All other entities were missed out.

Training procedure

One can reproduce the result running this script

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3

Training results

Framework versions

• Transformers 4.30.2
• Pytorch 2.0.1+cu117
• Datasets 2.14.3
• Tokenizers 0.13.3