🐦 EmbeddingGemma 2 (740M) — Korean 620 Bird Species Trimodal Bird Finder (Main Model)

English | 한국어

🌍 English Overview

This repository hosts the fine-tuned EmbeddingGemma 2 (google/embeddinggemma-2, 740M) trimodal model and multi-vector reference index covering 620 species of wild birds. It aligns Bird Audio (16kHz vocalizations), Bird Photos (RGB field images), and Bilingual Ecological Descriptions (Korean + English taxonomy & acoustic traits) into a single unified 768-dimensional L2-normalized embedding space.

📱 Looking for the lightweight On-Device Mobile (LiteRT .tflite) & Browser (WebGPU .onnx) models? Visit the companion on-device repository: jaichang/embeddinggemma2-korean-birds-620-litert

✨ Key Highlights

• Unified 768-D Trimodal Space: Query by bird call (.wav/.mp3), field photo (.jpg/.png), or natural-language description (English/Korean) and retrieve the Top-$K$ matching bird species via simple cosine similarity (dot product).
• Fine-Tuned on Real Wildlife Observations: Trained with Multiple Negatives Ranking Loss (InfoNCE) across audio, visual, and encyclopedic text pairs sourced from iNaturalist Research-Grade observations and Wikipedia.
• Teacher Model for On-Device Knowledge Distillation: Serves as the 768-d teacher space for our ultra-compact mobile/WebGPU student encoders (9.2 MB Vision .onnx & 14.7 MB Audio .onnx in jaichang/embeddinggemma2-korean-birds-620-litert).

📦 Files in This Repository

🚀 Quick Start (Python)

import json
import numpy as np
from huggingface_hub import hf_hub_download
from sentence_transformers import SentenceTransformer

repo_id = "jaichang/embeddinggemma2-korean-birds-620"
model = SentenceTransformer(repo_id, trust_remote_code=True)

# Load precomputed 768-d reference index & catalog
npz_path = hf_hub_download(repo_id=repo_id, filename="korean_620_birds_multivector_index.npz")
cat_path = hf_hub_download(repo_id=repo_id, filename="korean_620_birds_catalog.json")

index = np.load(npz_path)
catalog = json.load(open(cat_path, encoding="utf-8"))["species"]

# Text-to-Bird search example (works in both English and Korean)
query = "black and white bird with long tail and iridescent blue-green wings, chak-chak call"
q_emb = model.encode(query, normalize_embeddings=True)

sims = index["photo_hybrid_embs"] @ q_emb
top1 = int(np.argmax(sims))
print(f"Top-1: {catalog[top1]['en_name']} ({catalog[top1]['kr_name']}, {catalog[top1]['sci_name']}) — cos={sims[top1]:.4f}")

📚 Dataset Provenance & Attribution

🇰🇷 한국어 안내 (Korean Overview)

야생조류 620종 전체의 새소리(Audio), 새 사진(Vision), 생태 도감(Text)을 단일 768차원 임베딩 공간에서 검색하는 본 모델(PyTorch / SentenceTransformers) 레포지토리입니다.

📱 Flutter 모바일 앱(LiteRT .tflite) 및 웹 브라우저(WebGPU .onnx) 온디바이스 추론 전용 레포지토리는 아래 링크에 분리되어 있습니다: 👉 jaichang/embeddinggemma2-korean-birds-620-litert

✨ 주요 특징

• 단일 768차원 트라이모달(Trimodal) 정렬: 새소리 녹음, 야생 사진, 자연어 특징 설명(한국어/영어)을 모두 동일한 768차원 초구면(L2-normalized) 공간에 매핑하여 코사인 유사도 내적만으로 즉시 종을 판별합니다.
• 온디바이스 지식증류(Knowledge Distillation) 교사 모델: 본 모델(740M, 1.2GB)이 생성한 768차원 임베딩 공간을 정답지(Teacher)로 삼아, 모바일/웹에서 1초 미만에 동작하는 초경량 학생 인코더(9.2MB 비전 / 14.7MB 오디오)를 지식증류하는 기반 모델입니다.

📚 데이터셋 출처 및 라이선스