Chon-1.0-12B Beat Larger Models on JevBench. But One Number Caught My Attention

[aapanel]

I've seen plenty of smaller models claim to outperform much larger ones. Most of the time, the headline sounds more impressive than the actual results.
But Chon-1.0-12B is worth a closer look.
It's a 12B decision model that recently scored 85.3 on JevBench's Capability metric, outperforming several larger models on structured decision tasks.
The October 20 benchmark results showed:

• Capability Score: 85.3
• Intelligence: 79.9
• Calibration: 90.8

Under the leaderboard's default cost and median-latency limits, Chon also reached the top of the open-weight Capability ranking.
Not bad for 12B parameters.
There's a catch, though. Chon isn't beating larger LLMs at coding, general reasoning, or writing. JevBench focuses on a much narrower problem: given a situation and a set of possible answers, can the model make a good decision and assign useful probabilities to those options?
Chon is built for that kind of task.
And I think there's a good reason to pay attention to it, especially if you're building AI agents.

We might be using LLMs for too many things​

I've noticed that a lot of AI applications follow the same pattern.
Need to understand a user request? Call an LLM.
Need to choose a tool? Call the same LLM.
Need to check whether something succeeded? Ask the LLM again.
It works, but I don't think it's always a good design.
Imagine a coding agent working on a GitHub issue. Understanding the issue and generating a patch are relatively difficult tasks. A capable LLM makes sense there.
But checking whether a test passed? That's just an exit code.
Choosing between a few predefined tools? That might not require a large general-purpose model either.
Chon takes a more focused approach. It handles typed decisions such as multiple-choice questions, yes/no judgments, and scoring tasks.
It's still built on an LLM. The evaluated 12B version uses a LoRA adapter on Gemma 4 12B, along with its own inference setup.
What's different is the output.
Instead of writing a paragraph about which option seems best, the model produces a decision with probabilities attached.
For applications where the output needs to be consumed by another program, that's useful.
Of course, general LLMs can return structured JSON too. The difference isn't simply that Chon can output something a program understands. It's that decision quality and probability calibration are central to what it's designed to do.

The calibration score interests me more than the model size​

Here's something I keep running into with LLM-based automation: a model can be wrong and sound completely sure of itself.
That's usually manageable in a chatbot. It's a bigger problem when the answer becomes the next action in a workflow.
Say a model labels 100 incidents with roughly 90% confidence.
If only 65 of those predictions are correct, that confidence score isn't particularly useful.
A well-calibrated model should have its predicted probabilities line up reasonably well with actual outcomes over many similar cases.
That's why Chon's Calibration score of 90.8 caught my eye.
But I wouldn't treat it as a guarantee. A good calibration score on a benchmark doesn't mean the model will be equally reliable on your application's data.
Production inputs are messy.
Logs are incomplete. Users describe problems badly. Sometimes two different failures produce almost identical symptoms.
And sometimes the right answer isn't even included in the available choices.
If I were building a decision system around Chon, I'd want an explicit "not enough information" option.
I'd rather have a model decline a questionable decision than confidently choose the least-wrong answer.

Then I looked at the latency numbers​

This was probably the most surprising part of the benchmark for me.
Chon has a mechanism called "think-when-unsure." When it's uncertain about a decision, it can spend additional compute reasoning before returning an answer.
That sounds reasonable.
But look at the raw GPU latency measurements from the published evaluation:

• P50: approximately 134 ms
• P95: approximately 33.4 seconds

That's quite a gap.
These are benchmark measurements under a specific setup, not production API latency numbers. Still, I wouldn't ignore them.
A fast median response is nice. But if a meaningful percentage of requests take much longer, that affects how you design your system.
Imagine using this for tool routing in an interactive agent. Most decisions might feel nearly instantaneous, while the harder ones could introduce a noticeable delay.
Maybe that's an acceptable trade-off. Maybe it isn't.
It depends on what you're building.
The benchmark also estimates around $0.05 per 1,000 decisions. That's based on reference pricing, not a measured cloud hosting bill.
So I wouldn't call Chon cheap to deploy just because it's a 12B model.
I'd want to see actual GPU utilization, throughput, memory requirements, and P95/P99 latency under the workload I'm planning to run.
A good benchmark score gets my attention. It doesn't make the deployment decision for me.

Where I'd actually use a model like this​

Tool routing seems like an obvious place to experiment.
So does issue classification, support ticket routing, code review triage, or deciding which diagnostic check to run next.
Take a server incident.
A few regions are reporting connection failures. CPU and memory look normal. Some requests return 502 errors.
You could ask a general LLM to analyze everything and write a troubleshooting plan.
Or you could give a decision model a few options:

• Check DNS.
• Investigate CDN or regional network issues.
• Inspect the origin and reverse proxy.
• Collect more evidence before deciding.

That seems like a reasonable use case.
But I wouldn't assume Chon is better at it until I'd tested it against a decent baseline.
A simple classifier might do just as well. A few deterministic rules might do better.
And if the evidence doesn't support a particular root cause, the system shouldn't pretend otherwise.
This is also where I think the distinction between making a decision and executing an action matters.
I'm fine with an agent automatically collecting logs, checking service status, or running read-only diagnostics.
I'm much less comfortable with an agent restarting databases, changing firewall rules, or modifying production configurations simply because a model reports 97% confidence.
Confidence isn't permission.
For risky operations, I'd still want independent policy checks, audit logs, rollback options, and human approval when appropriate.

My take​

I haven't run Chon on my own workloads yet, so I'm not going to claim it's ready to replace existing routing or classification systems.
But I like the direction.
We've spent a lot of time trying to make general-purpose models better at everything. Meanwhile, plenty of software tasks don't require broad intelligence. They need a reliable choice from a small set of options.
That doesn't automatically make a specialized model the right answer. Sometimes a rule engine is enough. Sometimes a small classifier is better. Sometimes you really do need a powerful LLM.
I'd rather build a system that makes those distinctions than send every decision to the biggest model available.
Chon is interesting because it gives us another option to evaluate.
And for me, that's a more useful takeaway than the usual "small model beats big model" headline.

 

[xoai]

Thanks for the write-up. I'm the author of Chon, and you've hit on the two things we care about most.

On the P95: the JevBench run you're quoting (20 Oct) was before our 22 Oct serving update, which went after exactly that number. A few percent of questions are ones the model is unsure about. On those, it writes a short piece of reasoning (up to 512 tokens) before giving its answer, and that writing step was plain transformer generation. Two changes:

• The reasoning step now runs on vLLM with FP8 weights. `chon.serve` starts a vLLM server itself when there's enough GPU memory left, or uses one you already run (`CHON_VLLM_URL`). Without vLLM it falls back to a static KV cache, so nothing breaks; it's just slower.
• We lowered the "unsure" thresholds, so about 11% of questions take the reasoning path instead of 20%.

On one H100 80 GB, one request at a time: p95 went from 24.3 s to 4.8 s, mean from 5.0 s to 0.6 s, median 0.11 s, about $0.042 per 1,000 decisions. Our setup differs from the board's, so I wouldn't read 4.8 s as the number the board will show, but the drop should carry over.

The honest trade-off: fewer questions get the reasoning step, so capability dropped a little on our own measurements (held-out 86.7 to 84.9, public 85.8 to 84.7). The adapter and calibration are unchanged. You need `chon` 1.1.0 or later; vLLM goes in its own environment (`vllm==0.32.0`), or use the Docker image in the repo. On a fresh machine, vLLM compiles its kernels at first start, so give it about 5 minutes before `/health` answers.

If tail latency matters more than the last points of accuracy, there's also Chon-1.0-12B-Fast: same adapter with the reasoning step off, one forward pass per decision, p95 about 0.1 s on the same H100, and somewhat lower accuracy.

On calibration, I agree: a benchmark score says nothing about your data. Every answer comes back with a probability for each option, so it's cheap to check yourself. Label a couple hundred of your own incidents, group the answers by stated confidence (say 0.6-0.7, 0.7-0.8...) and see whether each group is right about that often. If you try it, I'd genuinely like to hear how it comes out.

Model card and changelog: huggingface.co/ninhnv/chon-1.0-12b · code: github.com/xoai/chon