Local LLM routing with a small decision model

When to use this local approach

JevStyle can make the first local, fixed-choice routing decision, while your application sends uncertain or failed requests to a larger model or a human. It scores a defined workflow; it does not choose the best LLM for every possible task. Compare routing tools and the local Semantic Router integration, or follow the measured migration checklist.

One request, a defined set of choices

For this tutorial, “routing” means a classification decision over named destinations. The local model supplies scores; your application remains responsible for dispatch and fallback.

If your application asks an LLM to return a label such as billing or shipping, that step can be evaluated separately from the rest of the conversation. JevStyle scores the choices you define in one forward pass.

This example proposes a destination. It does not invoke a tool or perform a customer action. Keep authorization and execution checks in your application.

Start a local decision server

Use Python 3.10 or newer in a virtual environment. Install the backend for your machine, then keep the server running in this terminal.

# Apple silicon
python -m pip install "jev-style[mlx]"

# Linux, Windows, or an Intel Mac: use this instead
# python -m pip install "jev-style[torch]"

jev-style serve --release 0.8b

The first run downloads the model. Once it is ready, open the local Playground. Run the Python examples in a second terminal with the same environment.

Score a request and keep a fallback

Save this as route.py and run python route.py. The route descriptions are part of the model input: keep them specific and mutually understandable.

import json
from jev_style import JevStyle, choice

# Illustrative threshold: select yours on held-out labelled examples.
threshold = 0.90

with JevStyle(base_url="http://127.0.0.1:8765") as model:
    result = model.decide(
        "I need to change the delivery address for order 4411.",
        {"route": choice("Which workflow should handle this request?", {
            "billing": "Charges, payments, invoices, and refunds",
            "shipping": "Delivery status and delivery address changes",
            "account": "Login, password, and profile settings",
            "other": "Requests outside the workflows above",
        })},
    )

answer = result["answers"]["route"]
label = answer["choice"]
top_probability = max(answer["probabilities"].values())
needs_review = label == "other" or top_probability < threshold
print(json.dumps({
    "proposed_route": label,
    "top_probability": top_probability,
    "next_step": "human_or_llm_review" if needs_review else label,
}, indent=2))

Move one decision before moving a workflow

1. Collect representative requests and label the correct route, including ambiguous and out-of-scope cases.
2. Run the local model alongside the existing process. Record route agreement, mistakes, latency, and the share of cases sent for review.
3. Use a held-out set to check a fixed threshold before letting local decisions affect routing.
4. Keep the existing LLM or human queue available for uncertain answers and runtime failures.

The adoption skill can help find label-only LLM calls and prepare a measured migration.

npx skills add lawrence3699/jev-style --skill jev-style-adopt

Use your existing Semantic Router workflow

Semantic Router 1.1.1 includes SystemOneDecisionModel, which can call JevStyle’s local endpoint without an external API key. Its DecisionRouter uses route descriptions as choice options.

With your JevStyle server running, install the integration dependency and run the complete example from a clone of this repository:

python -m pip install "semantic-router==1.1.1"
python examples/semantic_router_local.py "Where is my parcel?" --threshold 0.90

The complete example discovers the server’s model ID, returns a review step when no route clears the threshold, and reports connection failures without choosing a business action. Its 0.90 threshold is illustrative. It requires Python 3.10–3.13.

The request/response path and fallback behavior were checked with the published Semantic Router 1.1.1 client against a local JevStyle HTTP server using the fake engine. That verifies the integration contract; evaluate the real model separately before using its decisions.

Where this approach fits

Use it for defined choices, yes/no checks, and ordered scores. Keep open-ended writing, conversation, and complex reasoning in the system best suited to those jobs. Adding other is useful, but it does not guarantee that every unfamiliar request will be rejected.

Local execution avoids sending decision inputs to a hosted model after the weights have downloaded. The Hugging Face demo is hosted separately. Actual latency depends on your hardware, input length, backend, and model; measure end-to-end time including fallback calls.

Common questions

What does local LLM routing mean in this guide?

JevStyle classifies a request into routes you define, such as billing, shipping, or account support. It returns a probability for every choice. Your application then selects the workflow; the decision model does not generate the final response or execute the selected tool.

Is this the same as choosing between hosted language models?

This example routes requests between workflows. You can define model destinations as choices, but your application must implement those calls and evaluate that policy. JevStyle does not automatically compare providers, predict their costs, or know which model will answer best.

Can the router run offline, and does it need a GPU?

After installing dependencies and downloading the weights, local decision requests can run without the hosted JevStyle API. The package supports PyTorch on CPU or CUDA and MLX on Apple silicon. Memory use and latency depend on the model, quantization, input length, and machine. Compare the local models and backends before choosing one.

Which probability threshold should I use?

Choose a threshold on representative labeled examples, then evaluate it on a held-out test set. The 0.90 value in this tutorial is illustrative. Measure the error rate among accepted routes and the share sent for review. A high model probability is not a guarantee that the route is correct.

What happens if the request is unfamiliar or the server fails?

Include an out-of-scope route, but do not rely on that option alone to catch unfamiliar requests. Route low-confidence decisions, invalid responses, timeouts, and connection failures to your existing LLM or human queue. The short scoring example assumes a successful server response; add failure handling before using it in a live workflow. See the published evaluation evidence and test your own traffic.