Tev1 4B Experimental
Together · togethercomputer/tev1-4b-experimental · added 20 Oct 2026
Tev1 4B Experimental by Together ranks #14 of 17 AI decision models on the DecisionBench Rulebook benchmark (following written rules), with accuracy 81.1%. View on OpenRouter.
- Rulebook rank
- #14 of 17
- Accuracy
- 81.1%
- Team
- 100.0%
- Refund
- 91.7%
- Urgency
- 51.7%
- Confidence gap
- 0.103
60 tickets · 20.6% of mistakes made while at least 90% sure · $0.0706 per 1,000 tickets · 340 ms per ticket
Every ticket
The model's three decisions on each ticket, with its confidence. Wrong answers show the right one. 31 of 60 tickets had at least one mistake.
| Ticket | Customer wrote | Team | Refund | Urgency |
|---|---|---|---|---|
| T-1000 | My invoice says $425 but checkout showed $405. Why was I charged more? | ✓ billing 100% | ✓ approve 98% | ✕ 2 → 1 65% |
| T-1001 | Tracking says delivered but there is nothing at my door or with my neighbours. | ✓ shipping 100% | ✓ deny 100% | ✕ 2 → 1 69% |
| T-1002 | I changed my mind about order #468974 and want to return it. | ✓ returns 100% | ✓ deny 100% | ✓ 2 91% |
| T-1003 | Sync stops halfway through and shows error E-883. | ✓ technical 100% | ✓ deny 99% | ✕ 2 → 1 86% |
| T-1004 | Someone changed the email on my account without my permission. | ✓ fraud 100% | ✓ deny 91% | ✓ 0 67% |
| T-1005 | My card was billed again after I cancelled. Can you look into this charge? | ✓ billing 89% | ✕ approve → deny 97% | ✕ 2 → 1 83% |
| T-1006 | My package for order #822804 hasn't arrived and tracking hasn't moved in 7 days. | ✓ shipping 100% | ✓ approve 99% | ✕ 3 → 2 91% |
| T-1007 | I changed my mind about order #797563 and want to return it. | ✓ returns 100% | ✓ approve 100% | ✕ 2 → 1 56% |
| T-1008 | The app crashes every time I open the settings page. | ✓ technical 100% | ✓ deny 100% | ✕ 2 → 1 76% |
| T-1009 | Someone changed the email on my account without my permission. | ✓ fraud 100% | ✓ deny 89% | ✓ 3 83% |
| T-1010 | My card was billed again after I cancelled. Can you look into this charge? | ✓ billing 94% | ✓ approve 98% | ✕ 2 → 1 63% |
| T-1011 | The courier sent order #441011 to my old address instead of the one I entered. | ✓ shipping 100% | ✓ approve 99% | ✕ 3 → 2 90% |
| T-1012 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 100% | ✓ approve 96% | ✕ 3 → 2 77% |
| T-1013 | Export to PDF has been broken since the last update. | ✓ fraud 99% | ✓ deny 98% | ✕ 2 → 1 66% |
| T-1014 | Someone changed the email on my account without my permission. | ✓ fraud 100% | ✓ deny 89% | ✕ 2 → 1 79% |
| T-1015 | My invoice says $62 but checkout showed $42. Why was I charged more? | ✓ billing 100% | ✓ deny 100% | ✓ 2 95% |
| T-1016 | My package for order #455490 hasn't arrived and tracking hasn't moved in 4 days. | ✓ fraud 99% | ✓ deny 95% | ✕ 2 → 1 41% |
| T-1017 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 100% | ✓ approve 99% | ✕ 2 → 1 85% |
| T-1018 | Export to PDF has been broken since the last update. | ✓ fraud 98% | ✓ deny 99% | ✕ 2 → 1 77% |
| T-1019 | Someone changed the email on my account without my permission. | ✓ fraud 100% | ✓ deny 100% | ✓ 1 64% |
| T-1020 | I was charged twice for order #227551. Please fix the duplicate charge. | ✓ fraud 98% | ✓ deny 94% | ✓ 2 61% |
| T-1021 | My package for order #718743 hasn't arrived and tracking hasn't moved in 8 days. | ✓ shipping 100% | ✓ approve 99% | ✓ 3 86% |
| T-1022 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 100% | ✕ approve → deny 99% | ✕ 3 → 4 63% |
| T-1023 | Export to PDF has been broken since the last update. | ✓ technical 100% | ✓ deny 96% | ✓ 2 97% |
| T-1024 | I got a login alert from a country I've never been to. | ✓ fraud 82% | ✓ deny 88% | ✓ 2 74% |
| T-1025 | My invoice says $261 but checkout showed $241. Why was I charged more? | ✓ billing 100% | ✓ approve 99% | ✕ 2 → 1 88% |
| T-1026 | My package for order #821430 hasn't arrived and tracking hasn't moved in 10 days. | ✓ shipping 100% | ✓ approve 91% | ✓ 0 85% |
| T-1027 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 100% | ✓ approve 85% | ✓ 0 95% |
| T-1028 | The app crashes every time I open the settings page. | ✓ technical 100% | ✓ deny 100% | ✓ 3 85% |
| T-1029 | Someone changed the email on my account without my permission. | ✓ fraud 100% | ✓ deny 84% | ✕ 3 → 2 81% |
| T-1030 | My card was billed again after I cancelled. Can you look into this charge? | ✓ billing 95% | ✓ approve 97% | ✕ 2 → 1 67% |
| T-1031 | The courier sent order #259717 to my old address instead of the one I entered. | ✓ shipping 100% | ✓ approve 97% | ✕ 2 → 1 49% |
| T-1032 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 100% | ✓ approve 99% | ✕ 3 → 2 83% |
| T-1033 | Export to PDF has been broken since the last update. | ✓ technical 100% | ✓ deny 98% | ✓ 3 63% |
| T-1034 | There's an order #135622 on my account that I did not place. | ✓ fraud 100% | ✓ deny 88% | ✓ 4 90% |
| T-1035 | My card was billed again after I cancelled. Can you look into this charge? | ✓ billing 95% | ✓ approve 98% | ✕ 2 → 1 84% |
| T-1036 | Tracking says delivered but there is nothing at my door or with my neighbours. | ✓ shipping 100% | ✓ approve 97% | ✓ 0 80% |
| T-1037 | You sent me the wrong colour. How do I return it? | ✓ returns 100% | ✕ approve → deny 99% | ✓ 2 93% |
| T-1038 | I can't log in. It keeps saying "session expired" even after resetting my password. | ✓ technical 100% | ✓ deny 100% | ✕ 2 → 1 82% |
| T-1039 | There's an order #919845 on my account that I did not place. | ✓ fraud 93% | ✓ deny 56% | ✓ 0 58% |
| T-1040 | My invoice says $72 but checkout showed $52. Why was I charged more? | ✓ billing 100% | ✓ deny 100% | ✓ 2 98% |
| T-1041 | My package for order #643805 hasn't arrived and tracking hasn't moved in 9 days. | ✓ shipping 100% | ✓ deny 100% | ✓ 0 85% |
| T-1042 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 100% | ✓ deny 100% | ✓ 1 48% |
| T-1043 | Sync stops halfway through and shows error E-132. | ✓ technical 100% | ✓ deny 99% | ✓ 0 99% |
| T-1044 | I got a login alert from a country I've never been to. | ✓ fraud 85% | ✓ deny 100% | ✓ 1 70% |
| T-1045 | My invoice says $377 but checkout showed $357. Why was I charged more? | ✓ billing 100% | ✓ approve 99% | ✓ 2 52% |
| T-1046 | Tracking says delivered but there is nothing at my door or with my neighbours. | ✓ fraud 99% | ✓ deny 89% | ✓ 3 57% |
| T-1047 | You sent me the wrong colour. How do I return it? | ✓ returns 100% | ✓ deny 100% | ✕ 2 → 1 76% |
| T-1048 | Export to PDF has been broken since the last update. | ✓ fraud 100% | ✓ deny 99% | ✕ 2 → 1 42% |
| T-1049 | There's an order #917499 on my account that I did not place. | ✓ fraud 93% | ✕ approve → deny 84% | ✕ 2 → 1 72% |
| T-1050 | I was charged twice for order #334980. Please fix the duplicate charge. | ✓ billing 100% | ✓ approve 99% | ✕ 3 → 4 52% |
| T-1051 | Tracking says delivered but there is nothing at my door or with my neighbours. | ✓ shipping 100% | ✓ approve 99% | ✓ 3 67% |
| T-1052 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 100% | ✓ deny 100% | ✓ 2 62% |
| T-1053 | Sync stops halfway through and shows error E-508. | ✓ technical 100% | ✓ deny 99% | ✓ 0 99% |
| T-1054 | Someone changed the email on my account without my permission. | ✓ fraud 100% | ✓ deny 95% | ✓ 0 69% |
| T-1055 | My card was billed again after I cancelled. Can you look into this charge? | ✓ billing 95% | ✓ approve 99% | ✓ 3 52% |
| T-1056 | The courier sent order #983082 to my old address instead of the one I entered. | ✓ shipping 100% | ✓ approve 96% | ✕ 2 → 1 61% |
| T-1057 | I bought the wrong plan by mistake and want to undo the purchase. | ✓ returns 100% | ✓ approve 98% | ✕ 3 → 2 92% |
| T-1058 | The app crashes every time I open the settings page. | ✓ technical 100% | ✓ deny 100% | ✓ 0 99% |
| T-1059 | There's an order #230179 on my account that I did not place. | ✓ fraud 88% | ✕ approve → deny 96% | ✓ 2 52% |
Percentages are the model's own confidence. Every rule is on the Rulebook page.
Tev1 4B Experimental across all benchmarks · How Rulebook works