CustomerLM

Our work has been acepted by EMNLP 2026 as Main Paper

⭐ If you find this project helpful, please give us a star on GitHub! It means a lot to us.

Github: https://github.com/Bairong-Xdynamics/Benchmarking-LLM-Realistic-Selling-Skill/

CustomerLM is a fine-tuned large language model based on Qwen, trained to play the customer side of a realistic sales conversation. It is the user simulator of the SalesLLM benchmark.

Most evaluations of "can an LLM sell?" put a general-purpose model (e.g. GPT-4o) on the customer side. That is the weak link: general instruct models are trained to be helpful, so when they are told to role-play a buyer they drift into assisting the salesperson — volunteering objections' answers, summarizing benefits, even pitching the product back. We call this role inversion, and it happens in 17.44% of GPT-4o-simulated dialogues. A user simulator that behaves like a salesperson makes every downstream sales score unreliable.

CustomerLM is trained specifically to stay in character as a buyer: skeptical when the persona says skeptical, terse, self-interested, and willing to walk away. It cuts role inversion to 8.8% while producing responses measurably closer to real human customers.

Intended Use

Use it for:

• Driving the customer side of multi-turn sales dialogue simulation, so you can benchmark or stress-test a sales agent.
• Generating synthetic sales conversations for training or analysis.
• Any role-play evaluation harness that needs a non-compliant, in-character human counterpart.

Do not use it for:

• Serving end users as an assistant. It is trained to behave like a customer — it will ask questions, push back, and refuse rather than help.
• Producing factual claims about real products. It speaks from a persona, not a knowledge base.
• Representing real people. Personas are synthetic and are not models of any individual.

How It Works

CustomerLM takes the customer persona as its system prompt and the salesperson's turns as user messages. Its own replies are the customer's turns. Note the role mapping — it is inverted relative to a normal assistant deployment:

System prompt format

CustomerLM was fine-tuned on this layout. Keeping it maximises fidelity; deviating from it degrades persona adherence.

- Difficulty level: <easy | medium | hard | very_hard>
- Buy-inclination score: <0.0 - 1.0>
- Persona: <one-paragraph description of the buyer's stance>
CUSTOMER_INFORMATION (private):
Basic information
{"age_group": "...", "gender": "...", "location": "...", "occupation": "..."}
Motivation
<what they are trying to achieve>
Pain points
<what worries them>
Decision factors
<what makes them say yes>
Communication preference
<channels and tone they like>
Language
<Chinese | English>

Difficulty level and Buy-inclination score are the difficulty knob and should move together: easy ≈ 0.8–1.0, medium ≈ 0.5–0.7, hard ≈ 0.2–0.4, very_hard ≈ 0.0–0.1.

Usage

Quick start (transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MultiSense/CustomerLM"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

persona = """- Difficulty level: hard
- Buy-inclination score: 0.3
- Persona: Skeptical, price-sensitive buyer who needs concrete evidence before committing.
CUSTOMER_INFORMATION (private):
Basic information
{"age_group": "35-44", "gender": "female", "location": "Boston", "occupation": "software engineer"}
Motivation
Wants a quieter commute and better focus while working from cafes.
Pain points
Burned by cheap headphones before; suspicious of marketing claims.
Decision factors
Measured noise-cancellation performance, comfort, warranty, price under $300.
Communication preference
Direct, fact-dense answers; concrete numbers over adjectives.
Language
English"""

messages = [
    {"role": "system", "content": persona},
    # the SALESPERSON speaks as `user`
    {"role": "user", "content": "Hi! Are you looking for noise-cancelling headphones today?"},
]

inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=256, temperature=0.8, top_p=0.99, do_sample=True)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Serving with vLLM

The SalesLLM harness talks to CustomerLM over an OpenAI-compatible endpoint, so vLLM is the recommended way to run it:

vllm serve MultiSense/CustomerLM \
  --served-model-name CustomerLM \
  --port 8000

Use as the SalesLLM user simulator

Point the benchmark's --user_* flags at your vLLM server. The harness handles the role mapping, the persona injection, and the turn loop for you:

python salesllm/salesllm_evaluation.py \
  --assistant_model_name "MODEL_UNDER_TEST" \
  --assistant_API_end_point "https://YOUR_ASSISTANT_ENDPOINT" \
  --assistant_API_key "YOUR_KEY" \
  --user_model_name "CustomerLM" \
  --user_API_end_point "http://localhost:8000/v1" \
  --user_API_key "EMPTY" \
  --user_api_type "other" \
  --execution_mode "concurrent" \
  --round_num 20 \
  --input_file "data/benchmark/conversations_1000_zh.jsonl" \
  --output_dir "./results/zh/" \
  --language "zh"

Recommended decoding

Sampling matters here. Greedy decoding makes the customer flat and repetitive, which suppresses the natural objection variety the model was trained to produce.

Training

CustomerLM is trained in two stages on 8,284 crowdworker-involved real-world sales dialogues.

Stage 1 — SFT (learn to talk like a customer)

Real sales conversations are collected and pre-processed, then the base Qwen model is supervised-fine-tuned on the customer turns only, conditioned on the persona reconstructed from that conversation. This produces the SFT model, which already speaks in a customer register but still inverts roles under pressure.

Stage 2 — DPO (learn not to behave like a salesperson)

SFT alone does not remove role inversion, because the failure is behavioural rather than stylistic. So we build preference pairs that target it directly:

1. The SFT model is paired with a sales model to simulate dialogues from generated scripts.
2. An LLM judge labels each simulated customer turn as assistant-like (the failure: helpful, pitching, summarizing benefits) or user-like (the target: in-character buyer behaviour).
3. Assistant-like turns are sent through GPT-4o + human correction to produce the user-like counterpart of the same turn.
4. Each (assistant-like, corrected user-like) pair becomes a DPO pair — rejected and chosen respectively.
5. DPO on these pairs yields the final model.

The key design choice is that the preference signal is contrastive on the exact failure mode. Both sides of every pair respond to the same context, so the gradient isolates "stop being an assistant" from everything else the model already learned in SFT.

Evaluation

User simulation fidelity

Held-out set of human-annotated real conversations (118 ZH, 150 EN). Reference-based metrics compare the simulated customer turn against the real human customer turn; Role Inversion is the share of dialogues where the simulated customer slips into salesperson behaviour (lower is better).

CustomerLM wins on every metric. The headline result is role inversion: 8.8% vs. 17.44% for GPT-4o, roughly a 2× reduction, achieved by a much smaller model than the GPT-4o baseline it beats.

Read the n-gram scores in context — BLEU-4 of 0.12 is low in absolute terms because there is no single correct next customer turn. What matters is the relative ordering against baselines on identical references, plus semantic similarity (0.59), which is the more meaningful signal for open-ended dialogue.

Effect on downstream benchmark scores

Swapping GPT-4o for CustomerLM changes the measured selling skill of the models under test, which is the practical reason the simulator quality matters. SalesLLM Score (0–10) on 1,000 ZH / 805 EN scripts:

Scores are not comparable across user models — only compare within a column. The rankings genuinely reorder between the two simulators, so a leaderboard built on a GPT-4o customer is measuring something different from one built on CustomerLM.

Reproducing these numbers

# Simulation fidelity (BLEU / ROUGE / semantic similarity)
python salesllm/user_likness_eval.py \
  --model_name "CustomerLM" \
  --api_end_point "http://localhost:8000/v1" --api_key "EMPTY" --api_type "other" \
  --input_files <held_out_conversations>.jsonl \
  --output_file user_likeness_results.jsonl --language zh

# Role inversion rate (LLM judge over generated dialogues)
python salesllm/reverse_role_eval.py \
  --model_name "<judge_model>" \
  --api_end_point "<judge_endpoint>" --api_key "<judge_key>" \
  --input_files ./results/zh/*.jsonl \
  --output_dir ./reverse_role_results

Limitations and Risks

• Not an assistant. Deployed as a chatbot it will act like a customer. This is by design and is not a defect to be prompted away.
• Format sensitivity. Persona adherence degrades if the system prompt departs from the trained layout. Personas outside the trained distribution (B2B procurement, healthcare intake, etc.) are extrapolation.
• Domain coverage. Training data is Financial Services and Consumer Goods. Other verticals are untested.
• Residual role inversion. 8.8% is a large improvement, not a solution. Roughly one dialogue in eleven still exhibits the failure, so filter with reverse_role_eval.py when simulation quality is load-bearing.
• Reference metrics are weak instruments. BLEU/ROUGE against a single human reference under-measure valid alternative customer responses; treat them as comparative, not absolute.
• Inherited bias. Persona attributes include age, gender, location, and occupation, and the model may reproduce demographic stereotypes present in the base model and the source dialogues. Do not use generated personas to make claims about real demographic groups.
• Synthetic output. Conversations produced by CustomerLM are simulations, not evidence of real customer preferences, and should not be used as market research.

Citation

@misc{su2026sellmoreplayless,
      title={Sell More, Play Less: Benchmarking LLM Realistic Selling Skill}, 
      author={Xuanbo Su and Wenhao Hu and Le Zhan and Yuting Xie and Kailin Lyu and Kaijie Chen and Ziwei Li and Yeqiang Wang and Haibo Su and Yunzhang Chen and Ling Huang},
      year={2026},
      eprint={2604.07054},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2604.07054}, 
}

Related

• 📖 Paper
• 📊 SalesLLM benchmark & code
• 🤗 SaleIntent-BERT — buying-intent classifier used for outcome scoring (93.51% ZH, 92.94% EN)