Local support routing: JevStyle versus a trained text classifier

What did the comparison find?

For this six-intent task with labeled training data, TF-IDF plus logistic regression was more accurate and faster in the observed test run: 229/240 correct (95.4%) versus JevStyle's 224/240 (93.3%). JevStyle ran without task fine-tuning; the classifier learned from 738 labeled examples. This result does not establish equal-supervision superiority or a general ranking.

JevStyle remains a candidate to evaluate when a workflow has described choices and you want to start without training a task-specific classifier. When representative labeled data already exist, include a traditional classifier before selecting a larger local model. These measurements provide no production-quality or safety guarantee.

Held-out accuracy, latency and fallback

Latency includes the local prediction and output validation. Model loading, downloads, HTTP transport, queueing and human or remote-model fallback are excluded. No cloud cost saving or hosted API latency is measured. Both methods had zero inference errors.

95% Wilson accuracy intervals: TF-IDF classifier 92.0–97.4%; JevStyle 89.4–95.9%. This single small run does not establish a statistically reliable performance ranking across workloads.

The development error target did not carry over

The threshold rule was frozen before test scoring: maximize development coverage with at least 20 accepted rows and observed accepted error no higher than 5%, searching a fixed 0.00–1.00 grid. The classifier selected 0.29; JevStyle selected 0.00 after 114/120 correct on development. The JevStyle test error was then 16/240 (6.7%), above that target.

Do not treat a top option probability as a guaranteed correctness rate. This rule is a reproducible failure case, not a recommended production threshold. Fallback means the request would be sent for review; no reviewer or larger LLM was run, and no perfect fallback accuracy is assumed. The threshold uses the top option probability, not JevStyle's separately normalized confidence.

Data, model and fixed protocol

We selected these six support intents before evaluating predictions:

• card_arrival: The physical bank card has not arrived or the user asks when it will be delivered.
• card_not_working: The physical bank card does not work or is damaged.
• card_payment_fee_charged: The user asks about a fee charged for paying with a bank card.
• cash_withdrawal_charge: The user asks about a fee for withdrawing cash from an ATM.
• lost_or_stolen_card: The physical bank card has been lost or stolen.
• pending_transfer: A bank transfer is still pending or has not completed.

Original train and test files retain their upstream split. Normalized duplicate text and train/test overlap are removed. Within each class, SHA-256 ordering with seed 20261101 assigns 20 original training rows to development and the remainder to baseline training. Final sizes are 738 train, 120 development and 240 test. The prompts and descriptions never include gold labels for development/test examples.

JevStyle: jev-style==0.6.2, 0.8B v3, MLX 8-bit, model revision 7f14c9fa1491d168a7f70b16acf68baf9d4f7353. Baseline: scikit-learn 1.7.2, word TF-IDF (1–2 grams, 10,000 features), logistic regression C=1, one CPU thread. Python 3.12.13; macOS 15.7.5; Apple M1 Max, 64 GiB. Baseline fit: 0.060 s; JevStyle load: 2.438 s.

Reproduce and inspect every prediction

Complete reproduction instructions · Frozen protocol · Python runner · Dependency lock

Machine-readable report · Source row IDs and text hashes · All 720 development/test predictions · Frozen thresholds

python3.12 -m venv .venv
.venv/bin/python -m pip install -r experiments/banking-routing-v1/requirements.lock
.venv/bin/jev-style download --release 0.8b-v3 --backend mlx --precision 8bit
.venv/bin/python experiments/banking-routing-v1/run.py --download --data-dir /tmp/jev-banking-data --output /tmp/jev-banking-run-1

Run from a checkout of this website repository. Use a new output directory. Dataset hashes, script hash and result artifact hashes are included in the report. Numerical probabilities and timings can vary across hardware or runtime versions.

Scope and dataset attribution

• Project-run closed-set six-intent experiment, not the full BANKING77 task or an independent benchmark.
• Baseline uses labeled training data; JevStyle uses instructions and descriptions without task fine-tuning. Supervision differs.
• Public data may have appeared in model pretraining; absence of contamination is not established.
• No out-of-scope, multilingual, production traffic, image or downstream action evaluation.
• Development error target is empirical; it does not guarantee the held-out error rate.
• No test-driven prompt, class, hyperparameter or threshold tuning. All unfavorable outcomes retained.

BANKING77 is © PolyAI Limited, CC BY 4.0. We use fixed CSVs from the original PolyAI repository, select six intents, derive development rows and publish predictions without changing upstream labels. See the dataset license. Citation: Casanueva et al. (2020), Efficient Intent Detection with Dual Sentence Encoders.

Next: compare local routing tools · evaluate your own requests · other published evaluations.