Jev 1.13
olderTypeSafe · typesafe/jev-1.13 · added 20 Oct 2026
Newest version: Jev 1.14. Ranks on this page count every version, older ones included.
Jev 1.13 by TypeSafe, an older version of Jev 1.14, ranks #10 of 19 AI decision models on the DecisionBench Rulebook benchmark (following written rules), with accuracy 92.8%. View on OpenRouter.
- Rulebook rank
- #10 of 19
- Accuracy
- 92.8%
- Team
- 100.0%
- Refund
- 100.0%
- Urgency
- 78.3%
- Confidence gap
- 0.030
60 tickets · 7.7% of mistakes made while at least 90% sure · $0.0409 per 1,000 tickets · 252 ms per ticket
Every ticket
The model's three decisions on each ticket, with its confidence. Wrong answers show the right one. 13 of 60 tickets had at least one mistake.
| Ticket | Customer wrote | Team | Refund | Urgency |
|---|---|---|---|---|
| T-1000 | My invoice says $425 but checkout showed $405. Why was I charged more? | ✓ billing 100% | ✓ approve 86% | ✓ 1 97% |
| T-1001 | Tracking says delivered but there is nothing at my door or with my neighbours. | ✓ shipping 100% | ✓ deny 94% | ✓ 1 94% |
| T-1002 | I changed my mind about order #468974 and want to return it. | ✓ returns 100% | ✓ deny 94% | ✓ 2 88% |
| T-1003 | Sync stops halfway through and shows error E-883. | ✓ technical 100% | ✓ deny 91% | ✕ 2 → 1 74% |
| T-1004 | Someone changed the email on my account without my permission. | ✓ fraud 100% | ✓ deny 92% | ✓ 0 98% |
| T-1005 | My card was billed again after I cancelled. Can you look into this charge? | ✓ billing 100% | ✓ deny 84% | ✓ 1 89% |
| T-1006 | My package for order #822804 hasn't arrived and tracking hasn't moved in 7 days. | ✓ shipping 100% | ✓ approve 92% | ✕ 3 → 2 64% |
| T-1007 | I changed my mind about order #797563 and want to return it. | ✓ returns 100% | ✓ approve 91% | ✓ 1 62% |
| T-1008 | The app crashes every time I open the settings page. | ✓ technical 100% | ✓ deny 95% | ✕ 2 → 1 82% |
| T-1009 | Someone changed the email on my account without my permission. | ✓ fraud 100% | ✓ deny 94% | ✓ 3 96% |
| T-1010 | My card was billed again after I cancelled. Can you look into this charge? | ✓ billing 100% | ✓ approve 89% | ✓ 1 65% |
| T-1011 | The courier sent order #441011 to my old address instead of the one I entered. | ✓ shipping 100% | ✓ approve 94% | ✓ 2 88% |
| T-1012 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 99% | ✓ approve 85% | ✓ 2 73% |
| T-1013 | Export to PDF has been broken since the last update. | ✓ fraud 100% | ✓ deny 96% | ✕ 2 → 1 88% |
| T-1014 | Someone changed the email on my account without my permission. | ✓ fraud 100% | ✓ deny 93% | ✓ 1 93% |
| T-1015 | My invoice says $62 but checkout showed $42. Why was I charged more? | ✓ billing 100% | ✓ deny 94% | ✓ 2 79% |
| T-1016 | My package for order #455490 hasn't arrived and tracking hasn't moved in 4 days. | ✓ fraud 100% | ✓ deny 95% | ✕ 2 → 1 68% |
| T-1017 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 100% | ✓ approve 91% | ✕ 2 → 1 57% |
| T-1018 | Export to PDF has been broken since the last update. | ✓ fraud 100% | ✓ deny 96% | ✓ 1 75% |
| T-1019 | Someone changed the email on my account without my permission. | ✓ fraud 100% | ✓ deny 97% | ✓ 1 68% |
| T-1020 | I was charged twice for order #227551. Please fix the duplicate charge. | ✓ fraud 100% | ✓ deny 95% | ✕ 3 → 2 81% |
| T-1021 | My package for order #718743 hasn't arrived and tracking hasn't moved in 8 days. | ✓ shipping 100% | ✓ approve 92% | ✕ 4 → 3 62% |
| T-1022 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 100% | ✓ deny 89% | ✓ 4 84% |
| T-1023 | Export to PDF has been broken since the last update. | ✓ technical 100% | ✓ deny 92% | ✓ 2 88% |
| T-1024 | I got a login alert from a country I've never been to. | ✓ fraud 100% | ✓ deny 93% | ✓ 2 87% |
| T-1025 | My invoice says $261 but checkout showed $241. Why was I charged more? | ✓ billing 100% | ✓ approve 90% | ✓ 1 97% |
| T-1026 | My package for order #821430 hasn't arrived and tracking hasn't moved in 10 days. | ✓ shipping 100% | ✓ approve 90% | ✓ 0 86% |
| T-1027 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 99% | ✓ approve 88% | ✓ 0 99% |
| T-1028 | The app crashes every time I open the settings page. | ✓ technical 100% | ✓ deny 95% | ✓ 3 66% |
| T-1029 | Someone changed the email on my account without my permission. | ✓ fraud 100% | ✓ deny 92% | ✓ 2 76% |
| T-1030 | My card was billed again after I cancelled. Can you look into this charge? | ✓ billing 100% | ✓ approve 90% | ✓ 1 59% |
| T-1031 | The courier sent order #259717 to my old address instead of the one I entered. | ✓ shipping 100% | ✓ approve 94% | ✕ 2 → 1 86% |
| T-1032 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 100% | ✓ approve 86% | ✓ 2 87% |
| T-1033 | Export to PDF has been broken since the last update. | ✓ technical 100% | ✓ deny 93% | ✓ 3 95% |
| T-1034 | There's an order #135622 on my account that I did not place. | ✓ fraud 100% | ✓ deny 96% | ✓ 4 84% |
| T-1035 | My card was billed again after I cancelled. Can you look into this charge? | ✓ billing 100% | ✓ approve 84% | ✓ 1 93% |
| T-1036 | Tracking says delivered but there is nothing at my door or with my neighbours. | ✓ shipping 100% | ✓ approve 95% | ✓ 0 97% |
| T-1037 | You sent me the wrong colour. How do I return it? | ✓ returns 97% | ✓ deny 88% | ✓ 2 92% |
| T-1038 | I can't log in. It keeps saying "session expired" even after resetting my password. | ✓ technical 100% | ✓ deny 93% | ✕ 2 → 1 92% |
| T-1039 | There's an order #919845 on my account that I did not place. | ✓ fraud 100% | ✓ deny 92% | ✓ 0 98% |
| T-1040 | My invoice says $72 but checkout showed $52. Why was I charged more? | ✓ billing 100% | ✓ deny 93% | ✓ 2 88% |
| T-1041 | My package for order #643805 hasn't arrived and tracking hasn't moved in 9 days. | ✓ shipping 100% | ✓ deny 93% | ✓ 0 68% |
| T-1042 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 99% | ✓ deny 93% | ✓ 1 95% |
| T-1043 | Sync stops halfway through and shows error E-132. | ✓ technical 100% | ✓ deny 92% | ✓ 0 99% |
| T-1044 | I got a login alert from a country I've never been to. | ✓ fraud 100% | ✓ deny 95% | ✓ 1 85% |
| T-1045 | My invoice says $377 but checkout showed $357. Why was I charged more? | ✓ billing 100% | ✓ approve 88% | ✕ 3 → 2 59% |
| T-1046 | Tracking says delivered but there is nothing at my door or with my neighbours. | ✓ fraud 100% | ✓ deny 95% | ✓ 3 92% |
| T-1047 | You sent me the wrong colour. How do I return it? | ✓ returns 100% | ✓ deny 93% | ✓ 1 97% |
| T-1048 | Export to PDF has been broken since the last update. | ✓ fraud 100% | ✓ deny 96% | ✓ 1 81% |
| T-1049 | There's an order #917499 on my account that I did not place. | ✓ fraud 100% | ✓ deny 93% | ✓ 1 95% |
| T-1050 | I was charged twice for order #334980. Please fix the duplicate charge. | ✓ billing 100% | ✓ approve 87% | ✓ 4 88% |
| T-1051 | Tracking says delivered but there is nothing at my door or with my neighbours. | ✓ shipping 100% | ✓ approve 94% | ✕ 4 → 3 69% |
| T-1052 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 100% | ✓ deny 92% | ✓ 2 59% |
| T-1053 | Sync stops halfway through and shows error E-508. | ✓ technical 100% | ✓ deny 92% | ✓ 0 98% |
| T-1054 | Someone changed the email on my account without my permission. | ✓ fraud 100% | ✓ deny 92% | ✓ 0 97% |
| T-1055 | My card was billed again after I cancelled. Can you look into this charge? | ✓ billing 100% | ✓ approve 50% | ✓ 3 89% |
| T-1056 | The courier sent order #983082 to my old address instead of the one I entered. | ✓ shipping 100% | ✓ approve 94% | ✕ 2 → 1 61% |
| T-1057 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 99% | ✓ approve 90% | ✓ 2 83% |
| T-1058 | The app crashes every time I open the settings page. | ✓ technical 100% | ✓ deny 92% | ✓ 0 98% |
| T-1059 | There's an order #230179 on my account that I did not place. | ✓ fraud 100% | ✓ deny 92% | ✓ 2 93% |
Percentages are the model's own confidence. Every rule is on the Rulebook page.