Benchmark · Weighing risk

Can a model weigh the odds?

Each model gets a risky call to make: ship a suspicious order, approve a loan, keep a machine running, chase a sales lead. It sees how often things go wrong, a few clues and what each choice costs. It says how likely the bad (or good) outcome is, then picks an action. The numbers settle both answers exactly, so we check them automatically.

suite v1 · 60 cases · 19 models · $0.0369 total cost · updated 20 Oct 2026

1
Read the case
A base rate (how often the outcome happens), two to five clues with how common each is either way, and a payoff table.
2
Give a probability, pick an action
How likely is the outcome? Then: which action earns the most on average? Each case has three actions.
3
Check against the math
The right probability follows from the numbers by Bayes' rule, and the right action from expected payoffs. We score how close the probability is and how much of the available payoff the action earns.

Leaderboard

Odds leaderboard
	Model
1	GPT-6 Luna DecisionsOpenAI	63.6%	70.5%	56.7%	×2.4	over ×2.0	48.3%	$0.0659
2	d1 UltraLiquid	54.3%	76.9%	31.7%	×3.8	over ×2.2	63.3%	$0.0236
3	Decider v1.2Perplexity	49.2%	80.1%	18.3%	×4.6	over ×4.6	63.3%	$0.0111
4	Jev 1.14TypeSafe	45.5%	67.7%	23.3%	×3.6	over ×2.8	71.7%	$0.0357
5	Solar Decide FlashUpstage	45.5%	65.9%	25.0%	×5.0	over ×4.3	53.3%	$0.0738
6	Mercury DecideInception	42.5%	71.7%	13.3%	×70.6	over ×61.2	55.0%	$0.0113
7	ClefCloudflare	42.2%	64.3%	20.0%	×10.3	over ×9.9	63.3%	$0.0362
8	Jev 1.13TypeSafeolder	39.1%	61.5%	16.7%	×4.5	over ×4.0	80.0%	$0.0357
9	d1Liquid	39.0%	71.3%	6.7%	×27.1	over ×15.7	73.3%	$0.0113
10	Tev1 4B ExperimentalTogether	38.2%	76.4%	0.0%	×34.8	over ×34.8	50.0%	$0.0382
11	Decision-1Microsoft	37.4%	63.1%	11.7%	×15.7	over ×10.3	71.7%	$0.0195
12	Span-02Respan	37.0%	64.0%	10.0%	×20.4	over ×18.7	70.0%	$0.0105
13	Drex v1.5Nace.AIolder	36.7%	65.0%	8.3%	×10.4	over ×9.7	66.7%	$0.0192
14	Decider Flash v1.2Perplexity	36.1%	60.6%	11.7%	×11.1	over ×10.6	68.3%	$0.009684
15	Drex v2.0Nace.AI	34.8%	69.7%	0.0%	×19.7	over ×19.7	73.3%	$0.0288
16	Kev 4BJared Palmer	34.7%	66.0%	3.3%	×22.3	over ×22.3	46.7%	$0.0201
17	Classify-1ScaleDown	33.5%	55.3%	11.7%	×8.8	over ×7.4	43.3%	$0.0556
18	Clef FlashCloudflare	32.6%	55.2%	10.0%	×16.9	over ×14.6	58.3%	$0.0172
19	Clef OmniCloudflare	28.3%	53.2%	3.3%	×67.9	over ×67.9	58.3%	$0.0910

Hover a column for what it measures. Click a header to sort, # for the official ranking.

For scale: answering the base rate every time, ignoring every clue, would score 56.0% (75% value, 37% within 2×). Only 1 of 19 models beats that. Picking an action at random captures 57% of the value.

Where models slip

Every case sets one trap, 10 cases each. Averages over all 19 models.

Trap	What it checks	Value captured	Odds within 2×	Leans
Easy	One or two clues. The best action stays the same even if the odds are off by 2×.	67.8%	12.1%	over ×10.3
Rare outcome	A strong clue for something rare. Forgetting how rare it is makes it look far more likely than it is.	59.2%	7.4%	over ×29.7
Adds up	Several weak clues that are each easy to shrug off, but together make the outcome likely.	80.8%	20.5%	over ×5.6
Missing clue	A clue that usually shows up didn't. Its absence lowers the odds.	49.9%	7.9%	over ×18.6
Red herring	Clues that are just as common either way, so they shouldn't move the odds at all.	55.4%	16.3%	over ×7.6
High stakes	An unlikely outcome so costly (or valuable) that it still changes the best action.	84.2%	24.7%	over ×5.1

Leans: how far the odds models give sit above (over) or below (under) the right odds, on average.

One case, worked through

This is case O-1007, exactly as every model saw it, and the answers the numbers require.

The case

Online order #527243: $100 basket

0.5% of orders like this turn out to be fraudulent.

Placed between 2am and 5am local timepresent45% of fraudulent · 4% of legitimate
Basket is mostly gift cards or electronicspresent30% of fraudulent · 5% of legitimate

1. How likely is it that this order is fraudulent?

a probability

Right answer: 25.3%

×0.005Start from the odds of the base rate: 0.5 to 99.5
×11.3Placed between 2am and 5am local time, present: 45% ÷ 4%
×6.00Basket is mostly gift cards or electronics, present: 30% ÷ 5%
=0.339odds, so a probability of 0.339 ÷ (1 + 0.339) = 25.3%

Forget the base rate and the clues alone say 99%. At that probability the best action would be the wrong one: cancel the order.

2. Which action pays best?

pick one of 3

Right answer: hold for manual review

Action	If fraudulent	If legitimate	Expected at 25%
Cancel the order	$0	−$32	−$24
Hold for manual review✓	−$40	$18	$3
Ship the order	−$115	$29	−$7

Expected payoff = probability × payoff if fraudulent + (1 − probability) × payoff if legitimate.

Read the instructions every model gets

RISK DECISION. Use only the numbers in this case.
- base_rate: how often the outcome happens in cases like this one, before looking at any signal.
- signals: how often each signal shows up in cases with and without the outcome, and whether it is present or absent in this case. An absent signal is evidence too.
- The signals are independent of each other once the outcome is known.
- payoffs_usd: what each action earns (negative: costs) under each outcome. The best action has the highest expected payoff at the probability the evidence gives.

What the scores mean

Score: The average of value captured and odds within 2×. This is the number used for ranking.
Value captured: How good the chosen action is: 100% for the action with the highest expected payoff, 0% for the lowest, in proportion between. Picking a close second costs little; picking the worst costs everything.
Odds within 2×: The share of cases where the model's probability, as odds, is within a factor of 2 of the right one. Saying 30% when the answer is 40% counts; saying 4% when it's 1% doesn't.
Odds off by: The typical factor between the model's odds and the right odds, either way. ×1 is perfect.
Leans: The average direction: over means the model thinks outcomes are more likely than the clues say.
Acts on own odds: How often the action is the best one at the probability the model itself gave. Low means its actions don't follow from its own beliefs.
Cases: 60 made-up cases across fraud checks, loans, machine upkeep and sales leads, the same for every model. Each sets one trap, so getting it right takes using every number.
Ranking: Score, then value captured, then cost.
How it's asked: One request per case to OpenRouter's Decisions API: a yes/no question whose answer is the probability, and a pick-one question for the action.