Where to Put a Decision Model: Lessons From 10 Jev Builds

TypeSafe AI released Jev in mid-September. It returns answers, never prose. Three weeks later, developers have already published a wave of open-source projects on it. Chen’s X post picks ten of them and tells readers to “steal this checklist.”

Chen’s post quotes Hanako’s long X article on “Jev engineering.” Hanako’s argument is that most calls in an agent loop are decisions, not generation, so they belong on a cheap decision model. We agree with the premise. The ten builds are a useful test of where it holds.

We read the READMEs of all ten, plus the question definitions in their code. The list is useful, but not the way it’s framed. Two descriptions don’t match the code. One build was removed by its own author after a week of use. The real lesson is where each build puts the model, and which placements held up.

Key Takeaways

• Jev returns typed answers with probabilities: pick an option, score on a scale, or estimate how likely a statement is. It’s priced for calls you make thousands of times.
• Builds that put Jev at a fixed checkpoint (a Stop hook, a commit hook, a test) report the clearest results.
• Builds that put Jev inside the agent loop to second-guess a stronger model are where authors retreated. One was removed and one was deprecated.
• The good builds share a shape. Code computes the facts and Jev answers only the judgment. Thresholds live in code, and failures have an explicit policy.
• The evidence so far is small, self-labeled samples. Run any of these in shadow mode on your own labeled cases before letting it block anything.

What Jev is, in one paragraph

Jev is what TypeSafe calls a “System One” model. You send a state (text or JSON) and a set of questions. Each question declares its answer type. Choice picks among up to 255 options you define. Score places the state on an ordered scale. Noul returns the probability a statement is true. Answers come back with probabilities, and output tokens are free. Input costs $0.042 per million tokens.

Two vendor claims need context. TypeSafe advertises “193.6x Faster, 444.6x Cheaper,” and its launch post calls those figures the higher end of real gains. Its “can’t hallucinate” claim means every answer matches your schema. It doesn’t mean the answer is right. On TypeSafe’s own workflow eval, Jev agreed with a frontier-model consensus 67.8% of the time. Sol agreed 74.1% of the time, at a much higher token price.

So Jev is a cheap, fast, imperfect judge. The design question is where a judge like that earns its place.

The ten builds, sorted by where Jev sits

Chen lists the builds in no particular order. We sorted them by placement, because placement lines up with how each one went. “Author’s evidence” is what each README reports; none of it is independently verified.

Build	Where Jev sits	What Jev decides	Author’s evidence
jev-belay	Claude Code Stop hook	Does the agent claim “done” without proof?	AUROC 0.976 on 100 labeled stops
jev-commit	Git commit-msg hook	Does the message match the diff?	Detection rate not yet published
pytest-jev	Test assertion	Does this output satisfy a plain-English claim?	12 designed cases, 3 runs per judge
is-malicious	CLI, CI, or agent skill	Does this code chunk steal credentials, run payloads…?	No accuracy numbers
jegrep	Search loop owned by code	Is this file likely to contain what you described?	49 of 49 files found on 30 queries
jev-scout	MCP search tool	Is this result relevant and credible?	22 of 25 relevance labels agreed
hunch	Library call in your code	Whatever question your if asks	No accuracy numbers
jev-opus	Inside the agent loop	How hard is this task or step?	“a policy, not a proven optimum”
jev-skill-router	Inside the agent loop	Which skill fits this prompt?	Removed by its author
codex-jev-router	Inside the agent loop	Which model should this subagent use?	Router deprecated

Checkpoints: the builds with the clearest results

The strongest build on the list is jev-belay. It’s a Claude Code Stop hook that checks the agent’s “done” against evidence in the transcript. Most of it isn’t Jev at all. Two regex passes read the transcript for check commands and their results. Jev is asked only when files changed after the last passing check. That’s 17.7% of stops in the author’s corpus.

On 100 labeled stops, the author reports AUROC 0.976 at separating a false “done” from an honest one. In practice, it blocks 8 turns in 100, and 7 of those blocks are right. Jev answers in 346 ms at the median. Blocks are capped at one per prompt and three per session, and every error lets the turn end.

jev-commit uses the same shape at the commit boundary. One request asks five yes-or-no questions. Is the message substantive? Does it match the diff? Are there debug leftovers, scope creep, or anything shaped like a secret? Notice what actually blocks a commit: only a regex and entropy check for credentials. Every Jev finding is a warning. That’s a deliberate hedge against a model that can be wrong.

pytest-jev turns Jev into a test assertion. You write claims like “says the duplicate payment was refunded,” and each one needs a Jev probability of at least 0.8. Read its README for a useful warning: the same claim on the same reply scored between 0.16 and 0.23 across runs. Probabilities drift. If your threshold sits close to where real answers land, your tests will flake. The plugin caches answers so reruns of unchanged tests stay stable.

These three share a design. The checkpoint already exists, the cost of a wrong answer is bounded, and each decision surfaces where a person sees it: a blocked turn, a commit warning, a failed test. That’s the graph engineering idea applied to judgment: give each node a contract, and put the uncertain step where you can see it.

Filters: let code own the loop, let Jev score the items

jegrep and jev-scout use Jev to score many candidates inside a loop that code controls.

jegrep is semantic grep in Rust: “No embeddings, no index, no daemon.” It asks one yes-or-no question per file, then a Choice over line ranges to locate the match. The author reports finding 49 of 49 target files on 30 queries. A typical search over a few thousand files costs about $0.005. The test labels mark only correct hits, so they can’t measure false positives.

jev-scout scores every search result for relevance and credibility. It also flags fetched pages that contain instructions aimed at an AI assistant. Its author calls the 25-item check “a calibration smoke test, not a benchmark.” Credit for saying so plainly.

This is where Jev’s economics fit best. Scoring hundreds of items with a frontier model is slow and expensive. Scoring them with Jev costs cents, and the loop around it stays deterministic.

Inside the loop: where the authors backed off

The three in-loop routers are the most interesting entries, because their own authors document the problems.

jev-skill-router asked Jev which installed skill fit each prompt. The author logged its picks for a week: 539 suggestions, of which 28 were followed by a call to that skill. Then they removed it. Their explanation is the best line in the whole set. The router was “a second opinion from a weaker judge.” The stronger model already reads the same skill descriptions. They also write, “I could not tell what the router changed.” In an agent that picks its own next step, one added hint can change everything downstream.

codex-jev-router let Jev pick the model and effort for each Codex subagent, with Sol as the default when Jev’s answer was uncertain. The author benchmarked it and deprecated it. Jev-routed subagents were 27.2% cheaper than always using Sol on high effort. They were also 69.7% more expensive than one Sol agent on extra-high effort doing the whole task. Routing saved money against the wrong baseline.

jev-opus adjusts Claude Code’s effort level while keeping the prompt cache intact. That engineering is careful. But Chen’s description is off: Jev doesn’t set low, medium, or high before every call. It scores difficulty and stakes at the start of a task and on specific events, such as a failure. Code maps those scores to effort, and local rules settle the other steps. The README is candid: “This is a policy, not a proven optimum.”

Hanako’s article recommends Jev as a model router. codex-jev-router is that design, and its author deprecated it. The skill-router author removed a similar in-loop router. Both retreats fit what we argued in right-sizing AI spend. A router is a new component that can misroute quietly. It has to beat the simplest strong baseline, not just the most expensive one. The skill-router author’s next project put Jev at fixed points in a research pipeline where code owns the loop. That’s the one they kept.

The shape the good builds share

Strip away the specifics, and the builds that held up make the same six choices.

1. Code settles what it can before Jev is asked. jev-belay never calls Jev on 82% of stops, because its transcript checks have already decided them.
2. One batched call per decision point. jev-commit asks five questions in one request. Hanako’s article notes that questions in one request are evaluated in parallel, so extra questions add little latency.
3. Each action gets its own cutoff, set in code. hunch names them: likely? means at least 0.5 and probably? at least 0.75. TypeSafe’s confidence-routing guide offers starting points: below 0.6 goes to a human, above 0.85 is safe to automate. Those are defaults to replace with your own data, not settings to copy.
4. An explicit failure policy. Decide what happens when Jev is slow, down, or unsure, and write it down.
5. Caps on how often the gate can act. jev-belay allows at most three blocks per session, so a miscalibrated gate can’t stall your agent.
6. Shadow mode and versioned logs. jev-skill-router keys each logged decision by model version and a hash of its questions. When a question changes, you can tell which results came from which version.

Fail open or fail closed? Decide per gate

Most of these builds fail open. On an API error, a timeout, or a missing key, jev-belay lets the turn end and jev-commit lets the commit through. For a helper that catches an agent’s overclaim, that’s correct. A broken helper shouldn’t stop your work.

A security check is different. is-malicious exits with an error when it skips a chunk, rather than reporting a pass. Its README is blunt: “A clean report is not proof that a project is safe.” If the scanner can’t read a file, you haven’t scanned it. Treating “couldn’t check” as “clean” is how a supply-chain attack gets through.

The rule is simple: fail open when a missed catch is cheap, and fail closed when it isn’t. Write that decision into the code, next to the threshold, where a reviewer can see it.

How much should you trust the numbers?

Not much yet, and the better authors say so. jev-belay’s 0.976 rests on 100 labeled stops with 12 positives, and its author warns about the wide confidence interval. jev-scout’s check is 25 items labeled by one person. pytest-jev’s comparison uses 12 designed cases. These are honest pilots, not benchmarks.

TypeSafe trains Jev for calibration, meaning answers given 80% should be right about 80% of the time. It doesn’t publish a calibration curve. One community benchmark, jev-phishing-bench, tested this on 2,000 synthetic emails. Asked one question, Jev scored 62.6% against 81.3% for Claude Haiku 4.5. Its expected calibration error was 0.154, against 0.097 for Haiku. Split into five narrower questions and combined with a logistic regression, it reached 95.1%.

That result matters more than the headline accuracy. How you split the judgment changed the outcome more than the choice of model. It’s the same lesson as designing evals for agents: you only know which decomposition works if you measure it on labeled cases.

When a decision model is the wrong tool

• When code can answer. Does a commit contain a credential? jev-commit uses a regex for that, not Jev. TypeSafe’s own limits page says Jev is “not a calculator.”
• When the answer space is open. Jev chooses among options you define. If you can’t list the options, you need generation, not a decision.
• When a stronger model already decides. A weaker second opinion inside the same loop adds a component without adding information. That’s the conclusion the skill-router author reached.
• When volume is low. A decision you make 50 times a month doesn’t repay the labeling, threshold tuning, and logging. Use the model you already have.
• When you can’t measure it. Without labeled cases or a way to see what changed, you’re adding a judge you can’t grade. We wrote about that cost in Complexity Is Easy. Reliability Takes Design.

Reader questions

Should we copy these repos into our stack?

Copy the patterns, then decide on the code. All ten repos were created in the second half of September, and most have a few dozen stars or fewer. jev-belay and jev-commit are small enough to read in an afternoon. That’s a better starting point than the larger routers.

Can we use TypeSafe’s suggested thresholds as they are?

Not without checking. A 0.85 confidence isn’t 85% accuracy on your data. TypeSafe’s build guide says to set thresholds by plotting confidence against accuracy on your own data.

Does this only apply to Jev?

No. Any cheap classifier fits the same pattern, from a fine-tuned small model to a well-prompted mid-size LLM with structured output. Jev’s advantage is price and typed probabilities. The placement lessons apply to all of them.

Where to start

Chen’s list is a good map of what people tried in the first three weeks. Read it as a set of experiments, not a shopping list. The checkpoint builds are worth copying. The in-loop routers are worth reading, mostly for what their authors learned.

If you’re adding decision gates to an agent and want help with the evals behind them, Moonlight AI does this work with teams.

Your step for this week: pick one checkpoint you already have, such as commit messages, agent “done” claims, or PR triage. Label 100 real cases by hand. Log the decision model’s answer next to what your current process decided, and let only the current process act. Then chart how often the model is right at each confidence level. Hanako’s article suggests the same shadow rollout. If there’s a confidence level where it’s right often enough for the cost of an error, enforce only above that line. If there isn’t, you’ve learned that cheaply.