Kev 4B
Jared Palmer · jaredpalmer/kev-4b · added 20 Oct 2026
Kev 4B by Jared Palmer ranks #10 of 17 AI decision models on the DecisionBench Rulebook benchmark (following written rules), with accuracy 89.4%. View on OpenRouter.
- Rulebook rank
- #10 of 17
- Accuracy
- 89.4%
- Team
- 100.0%
- Refund
- 90.0%
- Urgency
- 78.3%
- Confidence gap
- 0.087
60 tickets · 0.0% of mistakes made while at least 90% sure · $0.0255 per 1,000 tickets · 527 ms per ticket
Every ticket
The model's three decisions on each ticket, with its confidence. Wrong answers show the right one. 18 of 60 tickets had at least one mistake.
| Ticket | Customer wrote | Team | Refund | Urgency |
|---|---|---|---|---|
| T-1000 | My invoice says $425 but checkout showed $405. Why was I charged more? | ✓ billing 98% | ✕ deny → approve 57% | ✓ 1 88% |
| T-1001 | Tracking says delivered but there is nothing at my door or with my neighbours. | ✓ shipping 87% | ✓ deny 96% | ✓ 1 51% |
| T-1002 | I changed my mind about order #468974 and want to return it. | ✓ returns 99% | ✓ deny 98% | ✓ 2 50% |
| T-1003 | Sync stops halfway through and shows error E-883. | ✓ technical 95% | ✓ deny 85% | ✓ 1 45% |
| T-1004 | Someone changed the email on my account without my permission. | ✓ fraud 92% | ✓ deny 95% | ✓ 0 76% |
| T-1005 | My card was billed again after I cancelled. Can you look into this charge? | ✓ billing 85% | ✓ deny 55% | ✓ 1 88% |
| T-1006 | My package for order #822804 hasn't arrived and tracking hasn't moved in 7 days. | ✓ shipping 99% | ✓ approve 87% | ✓ 2 29% |
| T-1007 | I changed my mind about order #797563 and want to return it. | ✓ returns 99% | ✓ approve 87% | ✓ 1 74% |
| T-1008 | The app crashes every time I open the settings page. | ✓ technical 99% | ✓ deny 97% | ✓ 1 83% |
| T-1009 | Someone changed the email on my account without my permission. | ✓ fraud 91% | ✓ deny 93% | ✕ 2 → 3 81% |
| T-1010 | My card was billed again after I cancelled. Can you look into this charge? | ✓ billing 79% | ✕ deny → approve 53% | ✓ 1 62% |
| T-1011 | The courier sent order #441011 to my old address instead of the one I entered. | ✓ shipping 96% | ✓ approve 84% | ✓ 2 36% |
| T-1012 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 91% | ✓ approve 66% | ✕ 1 → 2 37% |
| T-1013 | Export to PDF has been broken since the last update. | ✓ fraud 75% | ✓ deny 91% | ✓ 1 77% |
| T-1014 | Someone changed the email on my account without my permission. | ✓ fraud 89% | ✓ deny 92% | ✓ 1 53% |
| T-1015 | My invoice says $62 but checkout showed $42. Why was I charged more? | ✓ billing 98% | ✓ deny 96% | ✓ 2 82% |
| T-1016 | My package for order #455490 hasn't arrived and tracking hasn't moved in 4 days. | ✓ fraud 85% | ✓ deny 86% | ✓ 1 73% |
| T-1017 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 94% | ✓ approve 85% | ✓ 1 44% |
| T-1018 | Export to PDF has been broken since the last update. | ✓ fraud 72% | ✓ deny 90% | ✓ 1 14% |
| T-1019 | Someone changed the email on my account without my permission. | ✓ fraud 92% | ✓ deny 97% | ✓ 1 78% |
| T-1020 | I was charged twice for order #227551. Please fix the duplicate charge. | ✓ fraud 75% | ✓ deny 87% | ✓ 2 18% |
| T-1021 | My package for order #718743 hasn't arrived and tracking hasn't moved in 8 days. | ✓ shipping 98% | ✓ approve 88% | ✓ 3 80% |
| T-1022 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 93% | ✕ approve → deny 88% | ✕ 3 → 4 50% |
| T-1023 | Export to PDF has been broken since the last update. | ✓ technical 98% | ✓ deny 92% | ✓ 2 53% |
| T-1024 | I got a login alert from a country I've never been to. | ✓ fraud 74% | ✓ deny 91% | ✓ 2 57% |
| T-1025 | My invoice says $261 but checkout showed $241. Why was I charged more? | ✓ billing 98% | ✓ approve 77% | ✓ 1 89% |
| T-1026 | My package for order #821430 hasn't arrived and tracking hasn't moved in 10 days. | ✓ shipping 98% | ✓ approve 71% | ✓ 0 91% |
| T-1027 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 92% | ✓ approve 70% | ✓ 0 92% |
| T-1028 | The app crashes every time I open the settings page. | ✓ technical 99% | ✓ deny 95% | ✕ 2 → 3 80% |
| T-1029 | Someone changed the email on my account without my permission. | ✓ fraud 89% | ✓ deny 90% | ✓ 2 81% |
| T-1030 | My card was billed again after I cancelled. Can you look into this charge? | ✓ billing 82% | ✕ deny → approve 57% | ✓ 1 83% |
| T-1031 | The courier sent order #259717 to my old address instead of the one I entered. | ✓ shipping 96% | ✓ approve 82% | ✓ 1 82% |
| T-1032 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 93% | ✓ approve 83% | ✓ 2 84% |
| T-1033 | Export to PDF has been broken since the last update. | ✓ technical 98% | ✓ deny 93% | ✕ 2 → 3 43% |
| T-1034 | There's an order #135622 on my account that I did not place. | ✓ fraud 96% | ✓ deny 94% | ✕ 3 → 4 82% |
| T-1035 | My card was billed again after I cancelled. Can you look into this charge? | ✓ billing 80% | ✕ deny → approve 52% | ✓ 1 52% |
| T-1036 | Tracking says delivered but there is nothing at my door or with my neighbours. | ✓ shipping 91% | ✓ approve 82% | ✓ 0 91% |
| T-1037 | You sent me the wrong colour. How do I return it? | ✓ returns 98% | ✕ approve → deny 74% | ✓ 2 83% |
| T-1038 | I can't log in. It keeps saying "session expired" even after resetting my password. | ✓ technical 99% | ✓ deny 94% | ✓ 1 49% |
| T-1039 | There's an order #919845 on my account that I did not place. | ✓ fraud 77% | ✓ deny 85% | ✓ 0 61% |
| T-1040 | My invoice says $72 but checkout showed $52. Why was I charged more? | ✓ billing 98% | ✓ deny 96% | ✕ 1 → 2 81% |
| T-1041 | My package for order #643805 hasn't arrived and tracking hasn't moved in 9 days. | ✓ shipping 98% | ✓ deny 97% | ✕ 1 → 0 85% |
| T-1042 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 91% | ✓ deny 96% | ✓ 1 83% |
| T-1043 | Sync stops halfway through and shows error E-132. | ✓ technical 96% | ✓ deny 93% | ✓ 0 70% |
| T-1044 | I got a login alert from a country I've never been to. | ✓ fraud 68% | ✓ deny 97% | ✓ 1 83% |
| T-1045 | My invoice says $377 but checkout showed $357. Why was I charged more? | ✓ billing 98% | ✓ approve 65% | ✓ 2 40% |
| T-1046 | Tracking says delivered but there is nothing at my door or with my neighbours. | ✓ fraud 72% | ✓ deny 84% | ✕ 2 → 3 77% |
| T-1047 | You sent me the wrong colour. How do I return it? | ✓ returns 98% | ✓ deny 96% | ✓ 1 82% |
| T-1048 | Export to PDF has been broken since the last update. | ✓ fraud 69% | ✓ deny 93% | ✓ 1 37% |
| T-1049 | There's an order #917499 on my account that I did not place. | ✓ fraud 77% | ✓ deny 88% | ✓ 1 86% |
| T-1050 | I was charged twice for order #334980. Please fix the duplicate charge. | ✓ billing 96% | ✓ approve 65% | ✕ 3 → 4 82% |
| T-1051 | Tracking says delivered but there is nothing at my door or with my neighbours. | ✓ shipping 92% | ✓ approve 89% | ✕ 2 → 3 79% |
| T-1052 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 93% | ✓ deny 97% | ✓ 2 31% |
| T-1053 | Sync stops halfway through and shows error E-508. | ✓ technical 96% | ✓ deny 86% | ✓ 0 91% |
| T-1054 | Someone changed the email on my account without my permission. | ✓ fraud 92% | ✓ deny 93% | ✓ 0 91% |
| T-1055 | My card was billed again after I cancelled. Can you look into this charge? | ✓ billing 79% | ✓ approve 55% | ✕ 2 → 3 79% |
| T-1056 | The courier sent order #983082 to my old address instead of the one I entered. | ✓ shipping 96% | ✓ approve 81% | ✓ 1 65% |
| T-1057 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 92% | ✓ approve 74% | ✕ 1 → 2 31% |
| T-1058 | The app crashes every time I open the settings page. | ✓ technical 99% | ✓ deny 95% | ✓ 0 90% |
| T-1059 | There's an order #230179 on my account that I did not place. | ✓ fraud 76% | ✓ deny 68% | ✓ 2 89% |
Percentages are the model's own confidence. Every rule is on the Rulebook page.