Model Card

Narada is a Supervised Fine-Tuned (SFT) variant of the Llama 3.2 3B Instruct model, specialized for classifying detected secrets in code snippets. It takes as input a code snippet with prepended line numbers for each line, along with the specific line number indicating the location of the potential secret, and outputs a classification label of "true positive" (actual sensitive secret) or "false positive" (benign or non-sensitive string).

Model Details

Model Description

This is a specialized(fine-tuned) variant of Meta's Llama 3.2 3B Instruct model, refined through Supervised Fine-Tuning (SFT) for binary classification of potential secrets/credentials in code snippets. As a lightweight transformer-based LLM with ~3 billion parameters, it's optimized for efficient inference on standard hardware. The SFT process adapts the base model to distinguish genuine sensitive information (e.g., API keys, passwords, private keys) from benign strings, addressing the common issue of false positives in automated secret scanners.

Core Functionality

The model evaluates "secrets" flagged in code, classifying them as "True Positive" (TP: actual risk if exposed) or "False Positive" (FP: non-sensitive, like placeholders or test values). It uses a structured evaluation prompt (detailed in the attached system prompt) that emphasizes context analysis, secret categorization (e.g., authentication credentials, cryptographic material, third-party tokens), false positive detection (e.g., low-entropy strings, dev artifacts), and entropy/format checks. Outputs include the classification label, exact secret value (preserving original structure for multiline cases), and a concise reasoning explanation.

Input and Output

Input: A code snippet with prepended line numbers (e.g., "1: import os\n2: API_KEY = 'sk-12345'"), plus the target line number for the suspected secret. Formatted as an instruct prompt:

user_prompt = f"<input>\n# CODE:\n{code}\n\n# LINE NUMBER: {line_number}\n</input>"

Where code is the code snippet you want to analyze, and line_number is the location of the secret in the snippet.

Output: Valid JSON only, example:

{"line_number": 2, "label": "True Positive", "secret_value": "sk-12345", "reason": "High-entropy string matching OpenAI API key format, assigned to sensitive variable in production-like context."}

This ensures easy integration into tools.

How to Get Started with the Model

Use the code below to get started with the model:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load the model and tokenizer
model_name = "deepsource/Narada-3.2-3B-v1" 
device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16).to(device)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Prepare input: Format the code snippet (with prepended line numbers) and target line number as shown
code = """1: import requests
2: import sys
3: 
4: class GitHubClient:
5:     def __init__(self):
6:         # Hardcoded API token (bad practice, for demo only)
7:         self.token = "ghp_9aB3XyZkLmN8QrT5UvWj7xY2ZaQ4b9CdEfGh"
8:         self.base_url = "https://api.github.com"
9: 
10:     def get_user(self, username):
11:         headers = {
12:             "Authorization": f"token {self.token}",
13:             "Accept": "application/vnd.github.v3+json"
14:         }
15:         url = f"{self.base_url}/users/{username}"
16:         response = requests.get(url, headers=headers)
17:         if response.status_code == 200:
18:             return response.json()
19:         else:
20:             return {"error": response.status_code, "message": response.text}
"""
line_number = "7"
input_text = f"<input>\n# CODE:\n```{code}```\n\n# LINE NUMBER: {line_number}\n</input>"

system_prompt = "<replace_with_system_prompt>" # Replace with the system prompt present at https://huggingface.co/deepsource/Narada-3.2-3B-v1/blob/main/system_prompt.md
messages = [
    {"role": "system", "content": system_prompt}
    {"role": "user", "content": input_text}
]
input_ids = tokenizer.apply_chat_template(messages, tokenize=True, return_tensors="pt").to(device)

# Generate response
outputs = model.generate(input_ids, max_new_tokens=1024)
generated_tokens = outputs[0][input_ids.shape[1]:]
response = tokenizer.decode(generated_tokens, skip_special_tokens=True).strip()

# Parse the JSON output (model responds with <json>{...}</json>)
import re
import json
match = re.search(r'<json>\s*(.*?)\s*</json>', response, re.DOTALL)
if match:
    json_str = match.group(1)
    data = json.loads(json_str)
    print(data)  # Example output: {'line_number': 7, 'label': 'True Positive', 'secret_value': 'ghp_9aB3XyZkLmN8QrT5UvWj7xY2ZaQ4b9CdEfGh', 'reason': '...'}
else:
    raise ValueError("No valid JSON found in response")

Bias, Risks, and Limitations

• The model is in the early stages of development, so errors and misclassifications are expected, particularly with obfuscated secrets (e.g., encoded, split across lines, or dynamically generated), emerging or custom secret types not well-represented in the training data, or code in niche languages.
• It relies on structured inputs (code snippets with line numbers and a specified line), and deviations may lead to unreliable outputs.