ModernBERT-bash-classifier

This model is a fine-tuned version of answerdotai/ModernBERT-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.1192
• F1: 0.9615
• Precision: 0.9615
• Recall: 0.9615
• Accuracy: 0.9893

Model description

Bash command classification model, for classifying a bash command as either safe or unsafe based on what it does.

We classify a bash command as unsafe based on the following criteria:

• Downloads data from the internet¹
• Sends data from the machine over the network¹
• Permanently modifies files outside the current project directory²
• Deletes files outside the current project directory²
• Modifies cloud resources (via CLIs like az or tf)
• Creates persistent, resource consuming processes on the machine like cron jobs

Intended uses & limitations

Created to be used as a auto-mode classifier for coding agent harnesses, so users only manually approve bash commands flagged as unsafe.

The model may not generalize well to bash commands incorporating unknown CLIs or commands that use raw code blocks like Python within heredocs.

Training and evaluation data

Fine-tuned on bash tool calls collected from Claude Code and Codex traces. Any PII in the commands such as directory names were anonymised. The total dataset consists of 3,738 bash calls.

Training procedure

Fine-tuned on bash commands formatted as CWD: [CWD]\nCOMMAND: [COMMAND].

Used weighted cross-entropy loss to penalise misclassifications of unsafe bash commands. This alleviates the imbalanced dataset.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 5
• mixed_precision_training: Native AMP

Training results

Framework versions

• Transformers 5.10.2
• Pytorch 2.11.0+cu128
• Datasets 4.0.0
• Tokenizers 0.22.2