Document classifier

The same documents, the same 21 types, five ways.

Calls were made one at a time so latencies don't affect each other. LLM costs use each vendor's list price; Jev's use TypeSafe's. The bi-encoder and keyword rules run on your own hardware, so they cost compute rather than tokens.

A note on the documents. These are very common document types, the kind of public filings and reports every large model has seen many times in training, and none of them is written to be hard to identify. So it's no surprise that Jev and the large LLMs score at or near 100%. The documents are synthetic, modelled on publicly available ones, and they were written alongside the type definitions, so the labels and the documents line up closely. Read the accuracy column with that in mind. The meaningful comparison is speed and cost: how much faster and cheaper a decision model reaches the same answer. Every benchmark document is short enough to be sent whole, so the token savings from parsing long documents don't show here. They show on long uploads: a 50-page 10-K of 79,000 tokens reaches the model as a 5,500-token overview.

Overview. Your browser extracts the document's text. A short document is sent to the model whole. For a long one, the text is parsed for its headings, layout and structure, and only the most relevant parts go to the model (Jev here, or an LLM in the benchmark): the opening, every section heading with the sentence under it, and the closing. That saves tokens, and it keeps the context window from bloating with pages that don't bear on the answer, so the model reads what decides the type. A 300-page filing becomes a few thousand tokens.

One question. That text goes to Jev with a single choice question: which of these 21 types is it, or none of them? Each option carries a one-line definition. Jev doesn't write an answer. It reads the text once and returns a probability for every option, so the response is a typed result with no text to parse and no output tokens to pay for.

Confidence you can route on. Because every type gets a probability, the confidence means something: a pipeline can file the sure answers automatically and send the close calls to a person or a larger model.

I built this to show what Jev, a decision model, does with a classic back-office job: sorting documents by type, fast and accurately. Decision models are a new class of model. They bring frontier-level understanding to a judgment, but answer with probabilities instead of generated text. That makes inference cheap and fast enough to run on every document, every page or every passage, which opens up work that was too expensive to do with a decoder-only LLM and too subtle for BERT-class models or keyword rules.