Published results, with their limits
JevStyle publishes decision-model results on JevBench and the Decision Index. The values below are project self-runs, not independent certifications or official leaderboard ranks. They describe particular model revisions, datasets, and runtimes.
The latest 2B v4 reports 74.9% accuracy on 231 public JevBench v1.4.1 items, a 44.37 Decision Index 0.3 public score, and a 57.55 public Vision score. These are different metrics: only the first is a percentage accuracy. Sources: 2B v4 model card, pinned revision and 0.8B v3 model card, pinned revision.
Model repository creation dates are 27 October 2026 for 2B v4 and 24 September 2026 for 0.8B v3. They are release metadata, not benchmark run timestamps. This page consolidates existing reports; it does not claim that a new evaluation was performed on the source-review date.
Accuracy and benchmark scores
| Model / evaluation | Result | Sample and metric | Runtime and interpretation |
|---|---|---|---|
| 2B v4 · JevBench v1.4.1 | 74.9% 173 / 231 correct | 231 public items; accuracy; 95% Wilson interval 68.9–80.0% | Official harness via the local systemone server; float32, Apple MPS. One self-run; no official rank. |
| 0.8B v3 · JevBench v1.4.1 | 64.1% 148 / 231 correct | 231 public items; accuracy; 95% Wilson interval 57.7–70.0% | Vendored official harness, commit 24b9b5c. Card reports zero training-pool hits; device is not specified in this result's protocol note. |
| 2B v4 · Decision Index 0.3 public | 44.37 | 140,178 scoreable requests; composite public index, not percentage accuracy | Kit 25c6ec7, its runner and scorer; bfloat16, A100 80 GB, fast path enabled. All requests answered. Some benchmark train splits were used in training. |
| 2B v4 · Decision Index public Vision | 57.55 | 12,210 rows; weighted, chance-corrected public index | Kit 25c6ec7, vision runner; bfloat16, A100 80 GB. Up to 1,638,400 pixels / 1,600 image tokens per image. CORD training data was used, so KIE is not zero-shot. |
Decision Index Full scores include private tests run by the maintainers. A public self-run cannot establish that Full score or its board rank. The 0.8B and 2B JevBench rows also come from separate runs, so this table is a starting point for model selection, not a controlled head-to-head on your workload.
Latency with hardware and measurement scope
The 2B v4 card reports a sample of 750 Decision Index 0.3 public requests, sampled uniformly with seed 20261027, on one otherwise idle A100 80 GB in bfloat16, one request at a time.
| Runtime | Median | Mean | p95 |
|---|---|---|---|
| CUDA graphs + fast path | 16.4 ms | 58.8 ms | 211 ms |
| Fast path only | 71.5 ms | 104.0 ms | 214 ms |
These are in-process request wall times. Model loading and CUDA graph recording are excluded. They do not measure HTTP transport, client-side work, queueing, or fallback to another model, and are not an estimate for your CPU, Mac, or hosted endpoint. See the pinned speed protocol.
For deployment decisions, time cold start, warm p50/p95, concurrent load, and the complete route including fallbacks on your own hardware. Compare quality and latency on the same input distribution.
Sources and reproduction
- 2B v4 protocol and training provenance: pinned source for all 2B rows above.
- 2B v4 published Decision Index evaluation adapter: use with the kit version stated in the model card.
- JevBench v1.4.1 harness: retain the public-item split and request format when comparing a new run.
- 0.8B v3 JevBench chart data and full source card: published counts, uncertainty, and provenance.
- Evaluate your own labeled requests: save the data version, model revision, backend, precision, hardware, and error/coverage counts with the report.
For a new run, pin both the model revision and the evaluation tool revision. Do not reuse the headline numbers for quantized weights, another backend, altered prompts, or a different dataset without checking them again. A model's published global calibration is not evidence that its probabilities remain calibrated in every domain.
Benchmark questions
Is 44.37 an accuracy percentage?
No. It is the Decision Index 0.3 composite public score. Use the per-benchmark metric and protocol when interpreting an individual task.
Do these numbers prove 2B v4 will work better on my tickets?
No. Evaluate the language, labels, ambiguous cases, and class mix you expect to see. Measure the error rate and the share of requests sent for review under a fixed threshold.
Was every task evaluated zero-shot?
No. The source card marks training overlap and distinguishes held-out evaluation rows from whether a benchmark's training split was used. Consult each dataset's protocol before claiming zero-shot performance.
Where is the hosted API's response-time guarantee?
These local model experiments provide no hosted API latency guarantee. Check current service information separately.