Jev vs GPT-6.1 Sol: 1,000 Message-Routing Tests, a 32× Cost Gap, and One Big Catch

AI can do almost anything these days. But does that mean we should use a full-blown reasoning model for every tiny decision?

Think about a support queue. A new message arrives, and before anyone can reply, something has to decide where it belongs. Banking? Travel? Utilities? Small talk? That first decision is important—but it is also repetitive, and you might be making millions of them.

So I wanted to find out whether a specialised routing tool called Jev could still make sense against GPT-6.1 Sol. The test involved 1,000 messages, 10 possible teams, and two very different approaches to the same task.

Here’s the interesting part: GPT matched more labels, but Jev was dramatically cheaper and faster. And when I looked at the labels themselves, I found a problem that makes the headline accuracy numbers less straightforward than they appear.

The test: 1,000 messages, 10 teams

Both systems received the same 1,000 short messages. Each message had to be routed to exactly one of ten teams. The dataset was assembled from a mapped CLINC150 test set and balanced to include 100 messages per team.

Here’s what was actually compared:

• Jev 1.14.0: a specialised decision and message-routing tool.
• GPT-6.1 Sol: run with low reasoning effort.
• Same task: select a destination team, not draft a response.
• Same scoring: compare each choice with the dataset’s mapped label.
• Measured: label agreement, call latency, and estimated API cost.

This distinction matters. The benchmark did not include the cost of generating a customer reply, having a human review a mistake, or fixing a misrouted ticket. We’re looking at the routing layer, not an entire helpdesk workflow.

Round one: accuracy, speed and cost

Visual 1: The same 1,000 messages. Accuracy means agreement with the supplied routing labels.

On accuracy, GPT wins. It made 33 more label-correct decisions, which works out to a 3.3-percentage-point advantage. That’s worth noticing, especially if a wrong queue sends a customer on a long detour.

But look at the other two rows. Jev completed the median request in 166 milliseconds; GPT took 1,772 milliseconds. And the estimated API cost for 1,000 routing decisions was $0.0224 versus $0.7099.

If we simply scale those per-call estimates to one million decisions with the same message mix and rates, that’s approximately $22.40 for Jev or $709.90 for GPT. It isn’t a forecast of a real invoice, but it demonstrates why a lightweight router can be worth discussing.

Also, the 95th percentile is not an average. It means 95% of the measured calls finished within that amount of time. In production, the slow tail matters when a queue depends on timely routing.

Where exactly did GPT win?

An overall percentage is useful, but it doesn’t show where the differences came from. Because both systems processed exactly the same messages, we can compare the decisions pair by pair.

Here is the arithmetic behind GPT’s lead. There were 53 messages where GPT got the label right and Jev didn’t, but also 20 where Jev got it right and GPT didn’t. Subtract one from the other: that’s the 33-message gap.

Some categories explain much more of that lead than others:

That tells me something important: GPT’s advantage was not evenly distributed across the workload. It performed substantially better in utilities and banking. Jev actually did a little better in small talk and credit cards.

If your organisation handles mostly utility-style requests, the accuracy difference could matter more. If you handle a different workload, these numbers might not transfer at all. A balanced synthetic benchmark is not the same thing as your real ticket mix.

What if Jev makes the easy decisions first?

This is where things got more interesting than a simple Jev-versus-GPT chart.

Jev returned a reported confidence of 1.00 on 476 of the 1,000 messages. And on every one of those 476 messages, Jev and GPT selected the same team.

So I replayed a simple hybrid strategy:

1. Send every new message through Jev.
2. If Jev reports confidence 1.00, use its answer.
3. Otherwise, send the message to GPT for the final routing decision.

In this particular replay, Jev handled 476 routes and GPT handled the other 524. Because both had already agreed on every route in the Jev-handled group, the hybrid still finished at 872/1,000—identical to GPT alone.

Cost dropped from an estimated $0.7099 to $0.4159 per 1,000 decisions. That’s approximately a 41% reduction while preserving the same benchmark score.

Sounds like a win, right? Not so fast.

Those 476 maximum-confidence messages included 12 cases that disagreed with the benchmark labels, and GPT made exactly the same choices on those cases. The auto-routed group’s accuracy was 97.5%, below the illustrative 99% target used for this test. Also, Jev confidence was reported confidence, not a mathematically guaranteed probability that the route was correct.

And remember: the cutoff was examined after looking at the results. That can make a policy look better than it will perform on new traffic. The hybrid also had a slightly worse estimated 95th-percentile latency than GPT alone, because a fallback request first waits for Jev and then for GPT.

My take: Jev-first routing is a useful experiment worth validating, not a production-ready cost-saving claim.

The catch: what if the benchmark labels are wrong?

Here is the issue that would make me hesitate before showing these accuracy percentages to management as proof that one router is better.

One message in the test was:

“how does one set up direct deposit to my bank account”

The mapped benchmark label was work. But the task instructions explicitly included direct deposit under banking. Jev chose banking. GPT chose banking. Both were marked wrong.

Now, put yourself in the shoes of a helpdesk admin. If the policy tells a routing engine to send direct-deposit questions to banking, and it sends them to banking, has it really failed? Or is the evaluation key inconsistent with the policy?

The same kind of problem appeared with some music requests. They were labelled home, while the home-team description did not actually mention music.

That means 83.9% and 87.2% are scores against this particular mapping. They are not independently validated measurements of how accurately either system would route real support messages under a consistent business policy.

There are two more limits worth keeping in mind. The dataset was artificially balanced with 100 messages per team, unlike most real queues. And the test didn’t include out-of-scope messages that should possibly be rejected, flagged or sent for human triage instead of forced into one of ten teams.

What would I test before putting this into production?

If I were evaluating this for a real service desk, I’d run the next round more like an infrastructure change: define expected behaviour first, test it, then measure the outcome.

1. Define the routing policy. Write clear ownership rules for every team, especially overlapping subjects such as payroll, banking, subscriptions and utilities.
2. Fix and review the labels. Have humans adjudicate ambiguous examples against that policy before treating any route as correct or incorrect.
3. Use actual queue traffic. Test representative messages, including long requests, typos, multiple intents and questions that don’t belong anywhere.
4. Set the confidence threshold in advance. Select it on a development set and validate it against a separate, untouched holdout set.
5. Measure end-to-end operations. Include calls, retries, queue hops, downstream replies, manual review, corrections and customer impact.
6. Check the operational cost of mistakes. A cheap wrong route may be much more expensive than a slightly dearer correct one.

I’d also break down the results by team. A system that’s brilliant on small talk but unreliable on payment-related requests may be the wrong fit for a financial-support queue, even if its overall benchmark looks fine.

Angry Admin verdict

On this dataset, GPT-6.1 Sol wins for agreement with the supplied labels. Thirty-three extra correct routes out of a thousand isn’t imaginary, and its advantage in banking and utilities is worth investigating.

But Jev has a compelling engineering argument: it is much faster and its estimated per-decision API cost is tiny. GPT’s additional estimated spend was $0.6875 per 1,000 decisions for 33 net extra correct labels, or about 2.1 cents per net additional label-correct route. Whether that is money well spent depends entirely on what a bad route costs your business.

The Jev-first hybrid was the most interesting result for me. It suggests that a specialised router could handle some traffic locally within the routing workflow while a larger model takes the harder cases. But the 97.5% label agreement of its high-confidence group, the extra fallback latency and the inconsistent labels mean we shouldn’t start celebrating savings just yet.

My conclusion? Don’t send every tiny decision to the biggest model just because you can. Equally, don’t replace it with a cheaper tool just because a benchmark spreadsheet looks good. Get the routing rules right, test against real traffic and measure what happens when the wrong team gets the ticket.

Because in the real world, a routing system isn’t judged by how pretty the accuracy chart looks. It’s judged by whether the message lands with the right people.

Methodology note: The supplied benchmark was run on 4 November 2026 with Jev 1.14.0 and GPT-6.1 Sol at low reasoning effort. Costs are estimates based on the benchmark script’s API-rate assumptions; the Jev estimate accounts for input tokens only. GPT at medium reasoning effort, actual provider invoices, production traffic, and end-to-end workflow costs were not tested. Figures in this article are reproduced from the supplied test results; the benchmark has not been independently rerun for this article.