DLM-NL2JSON-4B

A 4B-parameter service-specific LLM that outperforms GPT-4o (+14%p) and Qwen3.5-35B (+22%p) on structured JSON extraction from Korean natural language queries.

DLM (Domain-specific Language Model) is a series of task-specialized models by Data Science Lab., Ltd.. This model is a LoRA-merged Qwen3-4B fine-tuned for structured JSON extraction in the Busan Metropolitan City public data analytics service.

Key Results

Evaluated on 2,041 test samples across 10 task categories (field-level exact match, summary excluded):

*adj: 64 CSM samples with known gold label noise excluded (see Evaluation section)

Per-Category Breakdown

DLM-NL2JSON-4B wins 8 out of 10 categories, with the largest gains on ALP (+43%p vs GPT-4o) and CREDIT-Income (+41%p).

Important: This is a Service-Specific Model

This model is NOT a general-purpose NL-to-JSON converter. It is trained exclusively for a fixed set of predefined schemas used in a specific production service. It will not generalize to arbitrary JSON schemas or different prompt formats.

To use this model correctly, you must:

1. Use the exact system prompts it was trained on (one per task category — see Usage section)
2. Include the corresponding special token (<TASK_CSM>, <TASK_CREDIT>, <TASK_GIS>, <TASK_ALP>, <TASK_CPI>) in the input
3. Expect output conforming only to the predefined schemas listed below

Why publish a service-specific model? This model serves as a reference implementation demonstrating that task-specific LoRA fine-tuning on a 4B model can dramatically outperform GPT-4o and larger open-source models on constrained structured output tasks. We believe the DLM (Domain-specific Language Model) approach — training small, cheap-to-serve models for specific service endpoints — is an underexplored but highly practical paradigm.

Intended Use

This model converts Korean natural language queries about public/economic data into structured JSON conforming to its predefined schemas. It is designed for and deployed in the Busan Metropolitan City Big Data Wave analytics dashboard.

Input: Free-form Korean query + task-specific system prompt

Output: Single-line JSON with exact schema compliance:

{"summary":"##2025년 5월 부산광역시 해운대구 유통/의료 소비분석##","base_ym":202505,"region_nm":"부산광역시 해운대구","industry_select":{"3":[],"8":[]},"sex_cd":[1],"age_cd":[30],"category":2}

Task Categories

Training Details

Training data consists of synthetically generated Korean natural language queries paired with structured JSON outputs, covering the Busan public data analytics domain.

Evaluation Methodology

• Metric: Field-level exact match — each JSON key's value is compared against the gold label. The summary field is excluded from comparison.
• Test set: 2,041 samples, stratified by category
• Gold label noise: 64/700 CSM samples have age_cd capped at [10..60] instead of [10..70] for "all ages" queries, conflicting with the prompt specification. These affect all models equally and are excluded in the adjusted metric.
• Train/Test overlap: 16/2,041 input strings (0.78%) appear in both sets — retained for consistency.
• All models received identical system prompts per category.

Hardware

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "dataslab/DLM-NL2JSON-4B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

# System prompt (example: CSM consumer spending schema — abbreviated for readability)
# Full prompts per category are available in the repository's eval/prompts.py
system_prompt = """너는 반드시 **JSON 한 줄**만 출력한다. 설명/텍스트/코멘트/마크다운/코드블록/이모지/공백 줄 금지.
출력은 항상 { 로 시작하고 } 로 끝난다.

[스키마: TASK_CSM] (키/타입/순서 엄수)
{"summary":string,"base_ym":int,"region_nm":string,"industry_select":object,"sex_cd":[int],"age_cd":[int],"category":2}

[기본값]
- base_ym: 0, region_nm: "부산광역시"
- industry_select: 업종 미지정 시 전 대분류 키를 []로 설정
- sex_cd: [0,1], age_cd: [10,20,30,40,50,60,70]
- category: 항상 2

[대분류 코드표] 1:여행/숙박 2:여가/문화 3:유통 4:음식/주점 5:음식료품
6:의류/잡화 7:미용 8:의료 9:교육 10:생활 11:자동차"""

# Note: special token <TASK_CSM> must be included in the user message
user_query = "<TASK_CSM>\n2024년 1월 해운대구 중동 의류/잡화랑 뷰티 쪽 남성 20~40대 위주로 알려줘"

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": user_query}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.0, do_sample=False)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True))
# {"summary":"##2024년 1월 부산광역시 해운대구 중동 의류/잡화/미용 소비분석##","base_ym":202401,"region_nm":"부산광역시 해운대구 중동","industry_select":{"6":[],"7":[]},"sex_cd":[0],"age_cd":[20,30,40],"category":2}
# Note: "뷰티" → mapped to 미용(code 7), "해운대구 중동" → normalized to "부산광역시 해운대구 중동"

vLLM / OpenAI-compatible serving

from openai import OpenAI

client = OpenAI(base_url="http://your-server:8006/v1", api_key="token")
resp = client.chat.completions.create(
    model="DLM-NL2JSON-4B",
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": "<TASK_CSM>\n2024년 1월 해운대구 중동 의류/잡화랑 뷰티 쪽 남성 20~40대 위주로 알려줘"}
    ],
    max_tokens=512,
    temperature=0.0,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}}  # disable thinking mode
)
print(resp.choices[0].message.content)

Important: When serving with vLLM/TensorRT-LLM, pass chat_template_kwargs: {"enable_thinking": false} to disable the Qwen3 thinking mode. Otherwise, reasoning tokens will consume the output budget and truncate the JSON.

Known Limitations

1. CPI category (86.3%) is the weakest — complex industry classification codes (A~U with sub-codes) are harder to extract.
2. CSM training data noise: ~8% of CSM training samples have age_cd capped at 60 instead of 70 for "all ages" queries, introducing inconsistency.
3. Domain-specific only: This model is trained exclusively for the Busan public data schema extraction task. It has no general-purpose capabilities and should not be used as a general chatbot.
4. Korean only: All training data and prompts are in Korean.

Citation

If you use this model, please cite:

@misc{dsl-dlm-nl2json-4b,
  title={DLM-NL2JSON-4B: A Domain-Specific Language Model for Korean Public Data Schema Extraction},
  author={Data Science Lab., Ltd.},
  year={2026},
  url={https://huggingface.co/dataslab/DLM-NL2JSON-4B}
}

Contact

• Organization: Data Science Lab., Ltd.
• Project: Busan Metropolitan City Big Data Wave