Hindi turn detector

The idea is to have a detect and label correctly whether a speech taken completely in hindi has paused or will continue.

This dataset used for this is the hindi subset from pipecat smart turn v3.2. The data was streamed and only rows pertaining to hindi were picked up.

Adding Pauses

After the hindi data points were separated, I added a script that would directly inject pauses within the audio itself, ranging from 250ms to 1500ms mimicking real human pauses. Although its done at a very basic level and can be improved further.

Model description

Whisper tiny has been used here with a classifier head and frozen layers. Only keeping the attention layers open during the training process

Training and evaluation data

The data was augmented with a ratio of 0.5, which meant that out of 10K training data points 5K would be randomly injected with pauses. Eval data was kept in two chunks where we had one set without the injected pauses and another having injected pauses.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 0.1
• num_epochs: 40
• mixed_precision_training: Native AMP

Training results

Result on Clean Eval Set

Result on Augmented Eval Set

Framework versions

• Transformers 5.16.1
• Pytorch 2.11.0+cu128
• Datasets 4.0.0
• Tokenizers 0.23.1

Code

You can find the training and data-prep code here.