Benchmark · Following written rules

Can a model follow a rulebook?

Each model plays a customer support desk. It gets a one-page rulebook and a customer's ticket, and makes three decisions. The rulebook settles every answer, so each has exactly one right answer, and we check it automatically.

suite v1 · 60 tickets · 12 models · $0.0341 total cost · updated 20 Oct 2026

1
Read the rules and the ticket
The rulebook covers which team handles what, when a refund is allowed, and how urgent a ticket is.
2
Make three decisions
Which team? Approve the refund, yes or no? How urgent, 0 to 4? Each answer comes with the model's confidence.
3
Check against the rules
Every answer is compared with what the rules require. We also check whether the model's confidence matched how often it was right.

Leaderboard

Triage leaderboard
	Model
1	d1 UltraLiquid	100.0%	100.0%	100.0%	100.0%	0.005	0.0%	$0.0290
2	Decision-1Microsoft	100.0%	100.0%	100.0%	100.0%	0.076	0.0%	$0.0260
3	Mercury DecideInception	98.3%	100.0%	100.0%	95.0%	0.014	66.7%	$0.0159
4	Decider v1.2Perplexity	98.3%	100.0%	100.0%	95.0%	0.015	100.0%	$0.0141
5	Decider Flash v1.2Perplexity	97.2%	100.0%	100.0%	91.7%	0.016	60.0%	$0.0168
6	Jev 1.14TypeSafe	96.1%	100.0%	100.0%	88.3%	0.042	0.0%	$0.0409
7	d1Liquid	93.9%	100.0%	100.0%	81.7%	0.038	0.0%	$0.0138
8	ClefCloudflare	87.2%	100.0%	96.7%	65.0%	0.057	4.3%	$0.0451
9	GPT-6 Luna DecisionsOpenAI	85.0%	100.0%	98.3%	56.7%	0.074	3.7%	$0.0916
10	Clef FlashCloudflare	81.7%	100.0%	90.0%	55.0%	0.116	0.0%	$0.0242
11	Clef OmniCloudflare	76.1%	100.0%	73.3%	55.0%	0.140	9.3%	$0.1296
12	Solar Decide FlashUpstage	70.0%	100.0%	85.0%	25.0%	0.086	0.0%	$0.1220

Hover a column for what it measures. Click a header to sort, # for the official ranking.

One ticket, worked through

This is ticket T-1002, exactly as every model saw it, and the answers the rules require.

The ticket

“I changed my mind about order #468974 and want to return it.”

Customer: enterprise plan
Product: physical
Order value: $488
Bought: 30 days ago
Refunds in last 90 days: 3
Open chargeback: no
Blocks their work: no
Waiting for a reply: 6 days

1. Which team should handle it?

pick one of 5

Right answer: returns

The customer is writing about wanting to send back or undo a purchase, which the rules route to returns.

2. Approve the refund?

yes or no · all four rules must pass

Right answer: no, deny

✓It's a billing, shipping or returns problem: yes
✓Bought within 30 days (physical): 30 days ago
✕Fewer than 2 refunds in 90 days: 3 so far
✓No open chargeback: none

3. How urgent is it?

0 to 4 · add up points

Right answer: 2

+2Enterprise customer
+0Order of $488 (+1 at $500 or more)
+0Doesn't block their work
+0Waiting 6 days (+1 at 7 or more)
=2total

Read the full rulebook every model gets

SUPPORT TRIAGE POLICY. Apply these rules exactly, using only the ticket's fields and message.

ROUTING (pick exactly one team):
1. If chargeback_open is true, route to fraud, whatever the message says.
2. Otherwise route by the problem the customer reports:
   - billing: wrong, duplicate or unexpected charges, invoices, payment status.
   - shipping: a physical package that is late, lost, or delivered to the wrong place.
   - returns: the customer wants to send back or undo a purchase (wrong item, wrong size, changed mind).
   - technical: the product or app is broken, crashing, or the customer cannot log in.
   - fraud: unauthorised activity on the account (unknown logins, changed details, orders they did not place).

REFUNDS (approve only if ALL are true):
a. The problem is a billing, shipping or returns problem (technical and fraud problems are never refunded at triage).
b. days_since_purchase is at most 30 for physical products, at most 14 for digital and subscription products.
c. refunds_last_90_days is less than 2.
d. chargeback_open is false.

URGENCY (add up points, cap at 4):
+2 if customer_tier is enterprise, +1 if pro, +0 if free.
+1 if order_value_usd is 500 or more.
+1 if blocks_work is true.
+1 if days_waiting is 7 or more.

The team options are described to the model as: Billing: charges, invoices and payment status · Shipping: late, lost or misdelivered packages · Returns: sending back or undoing a purchase · Technical: broken product, crashes, login problems · Fraud & security: unauthorised account activity, or any ticket with an open chargeback.

What the scores mean

Accuracy: The average of three scores: right team, right refund call, right urgency, over all tickets. This is the number used for ranking.
Confidence gap: A model says how sure it is of each answer. If it says 80%, it should be right about 80% of the time. The gap measures how far off it is: 0 is perfectly honest.
Sure & wrong: Of the model's mistakes, the share it made while at least 90% sure. These are the dangerous ones: you can't catch them by double-checking only the unsure answers.
Tickets: 60 made-up tickets, the same for every model. Refund denials usually fail just one rule, often by a day, so a model has to check every rule.
Ranking: Accuracy, then confidence gap, then cost.
How it's asked: One request per ticket to OpenRouter's Decisions API: a pick-one question for team, a yes/no question for refund and a 0–4 scale for urgency.