ASL Unified Webcam Gesture Detector (YOLO11n, MediaPipe boxes)

This model detects American Sign Language (ASL) alphabet gestures (A–Z) and digits (0–9) in webcam video and images. It is fine-tuned with real webcam data and background frames to work robustly in real-world human–robot interaction scenarios, such as commanding a mobile robot swarm.

Model details

• Architecture: YOLO11n (Ultralytics)
• Input size: 512×512
• Classes (36):
  • Digits: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
  • Letters: A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V, W, X, Y, Z
• Training framework: Ultralytics YOLO
• Box generation: MediaPipe HandLandmarker-based bounding boxes
• Special focus: Webcam domain, with explicit background (no-gesture) images to reduce false positives.

Dataset

The model is trained on a unified YOLO-format dataset combining:

1. Public ASL datasets from Kaggle:
   • ASL Alphabet Dataset (A–Y)
   • ASL Dataset (0–9 + A–Z)
2. Custom webcam data:
   • Background frames (no hands, typical room/robot setup).
   • Gesture frames for digits 0–9 and letters A, B, F, L, R, U, D, E, C, T, S, P.

All images were converted to YOLO detection format with MediaPipe-based bounding boxes and a unified 36-class mapping. Background images are included without labels to teach the model “no gesture” regions.

See dataset_info.md for details on the structure and conversion.

Metrics

Validation metrics (val split, 2,620 instances, 36 classes):

• Precision: 0.951
• Recall: 0.958
• mAP50: 0.982
• mAP50–95: 0.932

Key command classes

These classes are particularly important for robot-swarm control:

Full per-class metrics are available in the training logs.

Usage

With Ultralytics (PyTorch)

from ultralytics import YOLO
import cv2

model = YOLO("asl_unified_webcam_yolo11n_mp_best.pt")

cap = cv2.VideoCapture(0)
while True:
    ret, frame = cap.read()
    if not ret:
        break
    results = model(frame, conf=0.75, iou=0.45, verbose=False)
    boxes = results.boxes
    if len(boxes) > 0:
        cls = int(boxes.cls)
        conf = float(boxes.conf)
        symbol = model.names[cls]
        print(f"Detected: {symbol} (conf={conf:.2f})")
    cv2.imshow("frame", frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cap.release()
cv2.destroyAllWindows()

With the gesture_swarm package

from gesture_swarm.detector import GestureDetector

detector = GestureDetector(
    "asl_unified_webcam_yolo11n_mp_best.pt",
    conf_thresh=0.75,
)

# frame: OpenCV BGR image from webcam
symbol = detector.detect_from_frame(frame)
if symbol is not None:
    print("Detected symbol:", symbol)

Symbols can be accumulated into sequences (e.g. S 1 9 2 P 1 6 8 P 1 P 4 2) and parsed into structured swarm commands (subscribe, task, converge, motion).

Intended use

• Real-time gesture recognition from a standard webcam for human–robot interaction.
• Mapping ASL letters and digits to commands for a mobile robot swarm.
• Prototyping gesture-based interfaces in research, education, and robotics projects.

Limitations

• Optimized for webcam-like viewpoints and backgrounds; performance may differ on very different cameras or environments.
• Single-hand detection; multi-hand scenarios are not explicitly modeled.
• Some classes (e.g., U) have lower recall; consider adding more varied samples if “up” is critical in your setup.
• Not evaluated for safety-critical applications.

How to cite

If you use this model in a paper, project report, or demo, please cite it as:

@misc{jdanoff2026aslunifiedwebcamyolo11nmp,
  author       = {Alexander Jdanoff},
  title        = {{ASL Unified Webcam Gesture Detector (YOLO11n, MediaPipe boxes)}},
  year         = {2026},
  howpublished = {\url{[https://huggingface.co/ajdanoff/asl-unified-webcam-yolo11n-mp](https://huggingface.co/ajdanoff/asl-unified-webcam-yolo11n-mp)}},
  note         = {YOLO11n model for ASL alphabet (A–Z) and digits (0–9) gesture detection, fine-tuned with real webcam data and background frames.}
}

In text, you can refer to it as:

“We use the ASL Unified Webcam Gesture Detector (YOLO11n, MediaPipe boxes) [Jdanoff, 2026], available at https://huggingface.co/ajdanoff/asl-unified-webcam-yolo11n-mp.”

License & acknowledgements

Model weights and code: MIT License (see LICENSE).
Datasets: see the original Kaggle dataset licenses and attribution requirements.

This project uses:

• Ultralytics YOLO
• MediaPipe