Local text classification in Python: baselines, decisions and review

When to use this local approach

For instruction-driven categories without fitting a new classifier first, evaluate JevStyle as a local Python text-classification option. For stable labeled categories, keep a supervised baseline. Compare JevStyle, Semantic Router, scikit-learn and SetFit by their actual inputs and outputs.

Choose the classifier from the data you have

Local text classification is a workflow: define labels, run a model on your machine, and decide which results need review. A support message can become a team label; a document can become a topic; a request can become a workflow destination. You still need representative labeled examples to measure quality, even if the model can score categories without task-specific training.

The described-choice example below uses JevStyle’s 0.8B v3 text model through a local Python server. The supervised example uses scikit-learn. Both are starting points for evaluating your own labels, not claims that one approach will win on every dataset.

Starting points to evaluate on the same held-out requests. This table is not a performance ranking.
Approach	Useful starting situation	What you get	What to check
TF-IDF + logistic regression	Stable categories and representative labeled text.	A fitted classifier and per-class probability estimates.	Vocabulary drift, rare classes, calibration, and mistakes on an untouched test set.
Local text embeddings	Category descriptions or examples that can be compared by semantic similarity.	Similarity to label descriptions or a classifier fitted on embeddings.	A cosine-similarity score is not a calibrated class probability. Evaluate label wording and ambiguous matches.
JevStyle typed decision model	Choices described in words, or several yes/no, choice and ordered-score questions about one input.	Probabilities for your defined options in one forward pass.	Task-specific accuracy, probability calibration, rejection/fallback behavior, and hardware requirements.

For embedding similarity, the Sentence Transformers documentation explains how to encode text and compare vectors. For supervised baselines, see scikit-learn's text feature extraction and logistic regression references. JevStyle is another candidate to test; changing the model does not remove the need for labeled evaluation.

A small supervised baseline

Install scikit-learn in a virtual environment. This example fits a text vectorizer and classifier together, then prints predictions for new messages. All six training messages below are invented to demonstrate the code path.

bash

python -m pip install scikit-learn

python

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

# Toy examples for the API demonstration, not an evaluation dataset.
train_texts = [
    "My invoice has two charges for the same month.",
    "Please refund the duplicate payment on my bill.",
    "The parcel has not arrived at the delivery address.",
    "Where can I track the package that was shipped?",
    "I cannot sign in after resetting my password.",
    "My account login no longer accepts my password.",
]
train_labels = ["billing", "billing", "shipping", "shipping", "account", "account"]

classifier = make_pipeline(
    TfidfVectorizer(ngram_range=(1, 2)),
    LogisticRegression(max_iter=1000, random_state=0),
)
classifier.fit(train_texts, train_labels)

messages = ["Please check a charge on my invoice.", "I need help tracking my parcel."]
for message, label, probabilities in zip(
    messages, classifier.predict(messages), classifier.predict_proba(messages)
):
    scores = dict(zip(classifier.classes_, probabilities.tolist()))
    print({"label": label, "probabilities": scores})

Do not report accuracy from these toy rows. In a real project, fit the pipeline on the training split, choose settings on development data, and evaluate once on a held-out test split. Keep related messages in the same split. Compare class-level errors as well as overall accuracy.

Use described choices with a local decision model

JevStyle scores named options from your instructions. You do not need to train a classifier on these example messages just to run the request, but you do need labels from your intended task before deciding how much to automate. The 0.8b release is a text-only starting point. Compare its backends and weight sizes before installing.

Use Python 3.10 or newer in a virtual environment. Choose one installation command for your machine, then keep the local server running.

bash

# Apple silicon: MLX text backend
python -m pip install "jev-style[mlx]"

# CPU or NVIDIA CUDA: use this command instead
# python -m pip install "jev-style[torch]"

jev-style serve --release 0.8b
# Local Playground: http://127.0.0.1:8765

The first model start downloads weights. These commands do not use the hosted JevStyle API. After downloading and installing the required assets, local inference can process text on your machine. An online demo or remote API endpoint is a different deployment.

Save the following as classify_local.py and run it in a second terminal in the same environment. The code proposes a label or a human-review result; it does not send a ticket to a team or perform another business action.

python

import json
import math

import httpx
from jev_style import JevStyle, JevStyleError, choice

ROUTES = {
    "billing": "Charges, invoices, payments, and refund requests",
    "shipping": "Package tracking, delivery, and shipping-address changes",
    "account": "Sign-in, passwords, and account access",
    "other": "Messages outside the three categories above",
}


def classify_local(text, threshold=0.90):
    # Illustrative threshold. Select a fixed value using separate labeled data.
    if not math.isfinite(threshold) or not 0 <= threshold <= 1:
        raise ValueError("threshold must be between 0 and 1")
    try:
        with JevStyle(base_url="http://127.0.0.1:8765", timeout=15.0) as model:
            result = model.decide(
                text,
                {"team": choice("Which team should review this message?", ROUTES)},
            )
        raw = result["answers"]["team"]["probabilities"]
        if set(raw) != set(ROUTES):
            raise ValueError("unexpected options")
        probabilities = {label: float(value) for label, value in raw.items()}
        if not all(math.isfinite(p) and 0 <= p <= 1 for p in probabilities.values()):
            raise ValueError("invalid probabilities")
        label = max(probabilities, key=probabilities.get)
        probability = probabilities[label]
        review = label == "other" or probability < threshold
        return {
            "proposed_label": label,
            "top_probability": probability,
            "next_step": "human_review" if review else "candidate_for_routing",
        }
    except (JevStyleError, httpx.HTTPError, KeyError, TypeError, ValueError):
        return {"next_step": "human_review", "reason": "runtime_or_response_error"}


if __name__ == "__main__":
    print(json.dumps(classify_local("I was billed twice for the same month."), indent=2))

The client request and error types are documented in the versioned Python client source. The support-ticket example adds separate refund and urgency questions. For dispatch into a workflow, continue with local request routing and fallback.

Decide what can be automated

Run candidate approaches on the same held-out messages and the same labels. Record failed requests, out-of-scope inputs, and the proportion sent for review. Useful measures include per-class precision and recall, probability calibration where probabilities are produced, error among accepted decisions, and coverage.

Use development examples to refine the label definitions and resolve ambiguous categories.
Choose the review threshold on separate calibration examples. State the sample count and errors behind the choice.
Freeze the threshold, then evaluate on the untouched test split. Check individual classes and languages, not only an average.
Measure end-to-end latency and memory on the intended hardware, including fallback work. Published weight-file size is not runtime memory.

In the JevStyle setup, use the evaluation guide to score a labeled JSONL file. The published benchmark page explains existing results and their limits; it does not replace validation on your messages. This page makes no claim that one of the three approaches is universally more accurate, faster, or cheaper.

Common questions

Can this work without a GPU?

A scikit-learn text baseline runs on CPU. JevStyle supports a PyTorch CPU backend, and Apple silicon has an MLX option. Performance and available memory still determine whether a particular model and input length are practical. See the local decision model guide.

Are local embeddings already a classifier?

An embedding is a vector representation. You can compare it to label descriptions or train a classifier on vectors, but those are additional decision rules. Similarity alone is not evidence that a score is a calibrated probability of a class.

Do I need labeled data when I use described choices?

You can run a pretrained decision model with options described in words. Representative labeled examples are still needed to measure errors, choose a review threshold, and check that changes improve your task.

Does local execution mean that nothing can leave the machine?

It describes where this inference step runs. Package installation and initial model downloads use the network; an application's logging, monitoring, or downstream services may also send data elsewhere. The Python example sends inference requests only to the loopback address shown in its code.

What should happen when the model or server fails?

Keep an explicit failure result, and route it to human review or an independently designed fallback. Do not silently map a timeout or malformed response to a valid business category. The example returns human_review for those errors.