A question with defined answers
The integration sends the context and available options to the model. Jev returns an answer and a confidence measure.
Our own work / Open source
What does AI need to make a small decision?
A coding agent chooses a tool or decides whether to ask for review before continuing. We built OpenJev to connect those decisions to TypeSafe's Jev model and examine how it responds.
The question
The agent has to choose before continuing.
Choosing a tool, routing a request, and reviewing a command have a defined set of possible answers. We wanted to examine whether a decision model could handle those tasks and flag answers that needed review.
How we built it.
A review point
The code compares confidence with a configured threshold. Answers below it are flagged for review. Actions with consequences need their own approval rules.
An evaluation record
The test set covers tool selection and other agent decisions. The evaluation records errors so the approach's failures can be examined.
The evaluation
The limitations belong in the result.
The published evaluation uses 115 cases written and labeled internally. It includes classification and review decisions, among other tasks. Severity scoring was one of the weaker areas.
The general-purpose language model comparison uses three sequential calls per case in its final configuration. That choice affects the time and cost comparison.
The dataset is small and labeled by one person. Its figures describe that test. A decision for another team requires representative cases from that team's own work.
Darwin Tech Lab built the integration and evaluation code. TypeSafe provides the Jev model used in these tests. This project and benchmark are our own work; they do not represent client results.
Material you can examine
Does your workflow have a similar decision?
We can help define a task and its acceptance criteria, then evaluate whether AI is suitable. The first step is understanding your process.
Book an introductory call