Tool Selection with
Jev-Style Decision Models

A personal assistant operating on a local computer may need to condition its next action on the user request, conversation history, previous tool results, and an available tool catalog. As this decision context grows, encoding cost and memory requirements become relevant constraints. A research objective is therefore to identify compact models that retain tool-selection accuracy under long contexts and can produce a decision without autoregressively generating a complete tool call.

The proposed architecture assigns action selection to a decision model, argument generation and response synthesis to a chat model, and execution and permission enforcement to a harness. Selection is repeated after each tool observation, with fallback to the chat model when the decision policy abstains. The present experiment evaluates only the initial selection step.

1.1 Decision interface and prefill

Here, Jev-style describes a decision interface over a finite set of supplied actions. For input context x and action set A, the system selects the highest-scoring action. The evaluated systems return action scores or probabilities rather than a generated answer sequence. This terminology describes the interface; the underlying architectures and scoring implementations differ.

Single-prefill tool selection is the deployment target: encode the decision context and obtain a bounded action readout without decoding a multi-token tool call. The current comparison does not establish that every backend performs one shared physical forward pass across all candidates. For example, candidate-conditioned encoders and direct answer-logit readouts use different computation paths [7][10].

Removing autoregressive answer generation addresses one source of inference cost. The model must still process its input context, and the chat model may require some of the same evidence to construct arguments. Any overall latency benefit must therefore be measured across the complete assistant workflow.

1.2 Scope and related work

This study evaluates existing open decision systems and scoring methods. Related BFCL dispatch evaluations and decision-model comparisons have been released by Blobfish, OD1, and LangWatch [11][12][13]. The contribution here is a documented BFCL V1 conversion, a frozen evaluation subset, and an inspectable comparison of seven systems on identical cases. No new model architecture is introduced.

2.1 Source data and selection

The evaluation derives from the BFCL V1 data in the Gorilla repository [1], pinned to tag v1.0 and commit 9df5c346ee0556c8a7cb09fd7206a39aadd904c2. From a 2,000-case converted corpus, 250 examples were selected using category quotas, deterministic hash ranking, and normalized-request deduplication. Eligibility required an unambiguous next action and a usable reference label. Parallel-call cases were excluded because they do not define a unique first action.

The selected set contains 120 multiple_function, 30 executable_multiple_function, 35 simple, 15 executable_simple, and 50 relevance examples. The relevance category was filtered through a reviewed 55-case allowlist before selection. The exact seed, review decisions, retained source references, and selection manifest are included in the dataset release [2].

2.2 Decision inputs and systems

Each case supplies a user request, one to four function options, and three non-tool actions: no_tool, clarify, and cannot_answer. Function descriptions are capped at 120 characters. Full argument schemas and reference answers are retained in the release for inspection but are excluded from the decision input. The semantic inputs are shared; prompt serialization and tokenization are backend-specific. Laya’s rendered inputs span 104–271 tokens.

The evaluated systems are Laya typed-decisions [3], Kev-0.8B [4], Bespoke Nimble 9B [5], Qwen3.5-4B with SemIf [6][7], Rizzo Flow 4B Q8_0 [8], Von 1.3 with decision chains disabled [9], and NanoJev unified-games-v1 [10]. SemIf scores answer-option logits from frozen Qwen weights; it is a readout method, not an additional trained checkpoint. No model is fine-tuned in this experiment. All seven systems returned predictions for all 250 cases, without recorded evaluation errors or retries.

2.3 Selection metric

For a tool-required case, a prediction is correct only if the selected function matches the reference. For a no-call case, any of the three non-tool actions is accepted. This aggregation follows BFCL’s distinction between function calls and no-call responses; its labels do not separately supervise clarification and inability. The aggregate metric is:

Results are also reported separately for the 200 tool-required cases, the 150 multiple-tool cases, and both 50-case controls. These scores evaluate the selected action. Argument values, execution outcomes, and final responses are outside the protocol.

On the combined 250-case metric, Bespoke Nimble 9B and Qwen3.5-4B with SemIf each obtain 248 correct selections (99.2%). Laya obtains 240 (96.0%). On the 200 tool-required cases, the corresponding counts are 200, 198, and 197. The no-call subset accounts for the difference between these views.

Figure 1 reports the aggregate and subset results. As deterministic controls on the same 250 cases, always selecting the first offered function obtains 106/250 (42.4%), while always selecting a non-tool action obtains 50/250 (20.0%). These controls are provided in the selection manifest [2].

The leading tool-required scores differ by only a few cases. These point estimates do not establish statistical equivalence or a general model ranking. The subset analysis is particularly relevant because the 50 positive one-tool controls do not require distinguishing among multiple candidate functions.

The hardware identifier is not disclosed, limiting reproducibility of the latency measurements. Each plotted value is computed from the saved per-case wall-time measurements, with one recorded decision per case and no uniform warm-up or repeated-trial protocol. The measurements include first recorded calls. They use different precision settings and serving paths, including HTTP services and in-process scorers.

Figure 2 pairs selection accuracy with latency over the same selected subset. Median latency reduces the influence of isolated first-call overhead; the mean remains available to expose that sensitivity. These observations characterize the tested configurations. They do not isolate architecture effects or quantify an on-device speedup.

For a future assistant evaluation, the relevant measurement is time to a correctly completed task, including selection, argument generation, tool execution, fallback, retries, and response synthesis. A reduction in selection latency is useful only if it survives this complete workflow.

The released report contains all 250 requests and 1,750 predictions, including offered actions, selected actions, recorded score distributions, and wall times. The interface below supports comparison on identical cases and filtering by a selected system’s errors. Calibration on this evaluation distribution has not been established; the recorded probabilities should not be interpreted as empirical correctness guarantees.

This experiment establishes a reproducible conversion and an inspectable short-context selection baseline for seven existing decision systems. Several configurations achieve high selection accuracy under the defined BFCL-derived protocol. The results support further evaluation of a separate tool-selection stage in a local assistant; they do not establish the effectiveness of the complete architecture.

The next study should test whether selection accuracy is retained as the decision context and catalog increase, and whether avoiding autoregressive tool-call decoding produces a measurable benefit on constrained devices. The following extensions follow directly from the present limitations.