When to use this local approach
To replace a label-only LLM call, evaluate JevStyle or another suitable local method on the same held-out inputs. The migration checklist connects threshold selection, accepted-request errors, coverage and fallback cost; the tool comparison helps define the baselines.
Start with representative labels
Collect examples from the task that will actually use the model. Include rare classes, ambiguous inputs, your working languages, and cases that should go to a person. Use human-reviewed outcomes where possible. If the labels come from an existing LLM, the result measures agreement with that LLM rather than independent correctness.
Reserve an untouched final test set before choosing question wording, labels, or thresholds. Keep duplicates and closely related items in the same split. A hundred decisions per question is a useful starting sample, but rare errors need considerably more evidence.
Use the request format, plus a label
Each line of a JSONL file is one request. Add label inside each question you want scored: an option name for choice, true/false for noul, or a level label or zero-based level index for score.
{"state":"I was charged twice.","questions":{"team":{"type":"choice","instructions":"Which team should handle this ticket?","criteria":{"billing":"Charges and refunds","shipping":"Parcels and delivery"},"label":"billing"}}}
{"state":"My parcel is missing.","questions":{"team":{"type":"choice","instructions":"Which team should handle this ticket?","criteria":{"billing":"Charges and refunds","shipping":"Parcels and delivery"},"label":"shipping"}}}Run the model and evaluation
Use Python 3.10 or newer in a virtual environment. Install the backend for your machine, then keep the server running in this terminal.
# Apple silicon
python -m pip install "jev-style[mlx]"
# Linux, Windows, or an Intel Mac: use this instead
# python -m pip install "jev-style[torch]"
jev-style serve --release 0.8bThe first run downloads the model. Once it is ready, open the local Playground. Run the Python examples in a second terminal with the same environment.
In a second terminal, evaluate your labeled file. Strict mode stops on a label that does not name a valid option.
jev-style eval data.jsonl --url http://127.0.0.1:8765 --strict --json eval.jsonThe JSON output records the summary and per-question results. Save the input data version, model revision, package version, hardware, and backend alongside it. The example commands below use a running server so evaluation reuses the loaded model.
Read quality and coverage together
| Metric | What it tells you |
|---|---|
| Accuracy | How often the selected answer matches the label. |
| Brier score | How close predicted probabilities are to the labeled outcome. Lower is better. |
| ECE | A binned summary of calibration error. Inspect it alongside sample size and class-level errors. |
| Automation at 1%, 5%, 10% error | Observed coverage at a threshold whose selected examples meet the error budget in the scored sample. |
The automation threshold uses the top option probability, or max(p, 1-p) for a yes/no question. It does not use the API’s separate confidence field. A reported 5% error budget describes the observed sample; it is not a guarantee for future requests.
The evaluation command also fits a temperature on part of the data and reports its effect on the held-out portion. It does not automatically rewrite model weights or change the running server’s output.
Freeze a threshold, then test it
- Use development data to define the question and options.
- Use separate calibration data to choose the threshold and any temperature adjustment.
- Freeze those choices. Apply the fixed rule to the untouched test set and report both error rate and coverage, including counts.
- Repeat the analysis by language, class, and important failure type. Keep a fallback for low probability, out-of-scope cases, and runtime errors.
If you select a threshold from the same rows used to report its error, the result is exploratory. Do not describe it as independent validation. For small selected samples, report uncertainty instead of treating zero observed errors as zero risk.
Compare builds on the same requests
Start another build on a different port, then evaluate both endpoints together. For example, with servers already running on ports 8765 and 8799:
jev-style eval data.jsonl \
--server current=http://127.0.0.1:8765 \
--server candidate=http://127.0.0.1:8799 \
--strict --json comparison.jsonThe comparison scores common questions: if one engine rejects a row, that row is excluded for all engines and counted as failed. Report that count so an apparently good score does not hide unsupported inputs. Include cold start, warm request time, and fallback cost when deciding which model to deploy.