CS Paper Radar

Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters

Tardelli Ronan Coelho Stekel

cs.CL cs.LG ·submitted 2026-10-22 09:14 UTC · v1
benchmarkcode/data

Abstract

Many language-model pipelines end in a closed-set decision, such as selecting a tool. Prompting a large language model for it is slow and order-sensitive. We ask how small an encoder for such decisions can be and whether it should read the options jointly or separately. In a three-seed study at 32 million parameters, per-option scoring beats joint scoring on a held-out sample of a 37-benchmark decision suite by 5.8 points of chance-corrected skill at matched optimizer steps and by 2.8 at approximately matched training compute; two external benchmarks agree, though not significantly on their unseen tasks. Crossing reader with initialisation shows that the advantage requires a reranker initialisation, whose query-candidate pair input matches per-option reading; from the raw encoder the two readers are indistinguishable. Per-option scoring needs 5 to 13 times more encoder passes: at most twice the median latency, but more than four times the mean. Of six released joint encoders (17 million to 1 billion parameters), the 68- and 150-million-parameter models exceed a 435-million-parameter DeBERTa-v3 zero-shot classifier by 4.9 and 5.9 accuracy points on 209 overlap-audited System1-Arena tasks and are at least on par with it on the 145 tasks unseen in training. The smallest answers a median request in 14 milliseconds on 16 processor threads, against 728 for the classifier. Reordering options changes 12 to 23% of the joint models' answers (per-option scoring: none). Up to 68 million parameters, they lead their size range on a public leaderboard's private, mainly in-distribution tests.

Comment
18 pages, 8 figures, 8 tables. Code: https://github.com/tardellirs/crivo-decision-encoders ; models: https://huggingface.co/tardellirs

Closest papers

Nearest neighbours by meaning across the whole index.

MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface

cs.CL cs.AI ·2026-10-08 ·Jianpeng Cheng, Guangyu Sun, Aashu Singh, Benyu Zhang, Haixing Dai +9 more
benchmarkhighly cited similarity 0.69
Abstract

System One models prioritize fast, low-latency decision-making and calibrated probability distributions over token generation. While prevailing systems rely on structured schemas and typed outputs, candidate-scoring trade-offs remain poorly understood. We systematically investigate the scaling dynamics, accuracy, and inference efficiency of three scoring architectures adapted from a pre-trained decoder: bi-encoders, cross-encoders, and constrained decoders. Experiments show clear performance scaling with model size across all three architectures. Cross-encoders consistently achieve the highest accuracy across model sizes, while bi-encoders and constrained decoders perform similarly as training tokens scale. Efficiency-wise, bi-encoders maintain near-constant inference latency as candidate pools grow because candidate embeddings can be cached offline, whereas cross-encoder and constrained-decoder costs scale quadratically with candidate size. This efficiency advantage makes bi-encoders uniquely suited to bridge small closed-set decisions and massive open-set retrieval. For closed-set tasks, including all candidate options in the request prompt is critical for bi-encoder competitiveness. Guided by these insights, we introduce MetaEncoder, a multimodal encoder fine-tuned from a Muse-Glimmer decoder via unidirectional contrastive learning to match requests and candidates. We present packed contrastive training to maximize efficiency for dynamically allocated negative candidates, enabling memory-efficient full fine-tuning of a 30B decoder into an encoder. Evaluated across 12 benchmark suites spanning closed-set decision-making, multimodal understanding, and open-set retrieval, MetaEncoder outperforms state-of-the-art multimodal embedding baselines on decision tasks and MMEB-v3 image/video retrieval, while scaling to open candidate spaces far too large for current decision models.

0.44

Verdict: Encode Once, Decide Many

cs.CL cs.AIcs.LG ·2026-10-14 ·Avdhesh Singh Chouhan, Jack Perry
benchmarkcode/datahot direction similarity 0.66
Abstract · 19 pages, 2 figures, 14 tables. Code: https://github.com/LEC-AI/verdict

Automating organizational work means answering many small typed questions about one piece of state (does this email need a reply, how urgent is it, who owns it) with probabilities good enough to act on without a human. The open typed-decision model we compare against, like a standard cross-encoder, answers each question with a separate pass over the input. We present Verdict-MQ, an open encoder that reads a case once and answers all of its questions: given identical retained state tokens after truncation (verified on every reported case), a structured attention mask makes the state encoding independent of the questions, keeps questions from influencing each other, and makes option order irrelevant by construction. On the typed-decisions benchmark, a 395M Verdict-MQ matches the soft gold labels significantly better than the released 421M Laya specialist on all five fresh seeds (KL 0.080 vs 0.117, Brier 0.045 vs 0.061), an advantage that survives per-system temperature scaling; its higher accuracy (0.801 vs 0.767) lies past the benchmark's own saturation point, so we read that margin as closer agreement with the benchmark's labelling teacher. On RACE, BoolQ and MultiRC, against a matched fine-tuned baseline it wins on MultiRC and loses on RACE and on BoolQ, where its training is unstable; we measure zero-shot references ourselves, including a closed general system (Jev) that beats every fine-tuned encoder. A controlled comparison measures what the shared state costs: giving each question its own question-aware copy of the state gains 0.9-3.5 points on RACE and MultiRC at 1.4-2.5x the per-case latency, and nothing measurable on typed decisions, with the invariance guarantees unchanged.

0.69

Small Heads, Strong Decisions: A Controlled Study of Typed-Decision Models

cs.AI ·2026-11-11 ·Ritvik Rastogi
similarity 0.65
Abstract

In September 2026, TypeSafe announced Jev as the first model in its System One category: unstructured state goes in, and typed, calibrated decisions that software can use directly come out. Jev's architecture and training data are unpublished, but many open typed-decision models have followed, including Nimble, the Decision 1.0-3.0 family and Clef. They answer structured questions about a piece of text, such as a yes/no check, a choice among labelled options or a graded score, by reading the text with a language model and scoring the options with a small readout. Each pairs a different readout with a different backbone, training mix and recipe, so it is unclear which design choices matter. We re-implement five readouts and train all of them under one protocol: the same backbone, the same 15 public English training sources, the same recipe with a per-head learning-rate sweep, and the same seed. Our experiments demonstrate that: The complexity of the head does not affect accuracy: on Qwen3-0.6B every readout reaches 81.3-82.0% in-domain accuracy, and a 0.5M-parameter cosine head, which scores each option by its cosine similarity to the question, matches the 27M-parameter joint schema head adapted from Clef in-domain at half its inference time. Reading all of a text's questions from one prompt also matches one prompt per question in accuracy, with 1.6 times faster inference and 2.1-2.5 times faster training on one H100. The backbone's size matters more than either choice: moving from Qwen3-0.6B to Qwen3-4B adds 4 points in-domain and 11 points on held-out sources, while the base, instruct and thinking checkpoints of the same model tie. We recommend spending effort on the backbone and keeping the head minimal.

0.06

Jeva: System One Models from First Principles

cs.AI ·2026-10-13 ·Qi Li, Xinchao Wang
positioncode/datahighly citedcommunity pick similarity 0.64
Abstract

System one models like Jev enable direct, low-latency decision-making. Yet recent open-source practice of predicting an answer from serialized text does not, by itself, respect the structure of a decision problem. We argue that current designs underemphasize four important structural requirements. (i) null-aware: Models should distinguish answer absence from uncertainty among candidates; (ii) permutation-invariant: Reordering alternatives should not change the decision; (iii) option-scalable: Growing candidate sets should not require redefining the output space; (iv) cacheable: Request-independent computation over recurring alternatives should be reusable across requests. We introduce Jeva, a decision-native architecture that addresses these requirements jointly. A shared encoder independently represents the request and each option, while a compact decision stack refines only the request state against immutable option memory. Symmetric aggregation yields permutation-equivariant probabilities; shared scoring accommodates variable-size candidate sets. A separate coverage estimate distinguishes answer absence from candidate competition, enabling explicit null outcomes. Request-independent option representations and layer projections support reuse across newly assembled sets. These mechanisms unify Choice, Noul, and Score outputs. Across evaluations against Jev and open source models, Jeva combines strong typed-decision performance with near-perfect null detection and no decision flips under any tested permutations. It scales to thousands of candidates and substantially reduces latency through cross-request reuse without changing decisions.

0.66

UltraBERT-JEV: Encoder models for System One Applications

cs.CL cs.AI ·2026-11-13 ·Astarag Mohapatra
similarity 0.64
Abstract

We study encoder models that pick one of several options in a single forward pass, without generating text. This technical report compares recipes for fine-tuning UltraBERT-380M with one mask per option. The four recipes differ in how an option is scored and how the scores become probabilities; one follows the expert gate of DeepSeek-V3. Trained on the same data as Qwen3-0.6B with the LLM-as-Jev recipe (67.6% test accuracy), the encoder reaches 69.7% to 72.1%. Scoring an option by "yes" minus "no" instead of "yes" alone gives the larger gain, from 69.7% to 71.3% on the test set and from 57.8% to 59.6% on external benchmarks (56.1% for the decoder). Sigmoid normalisation adds 0.8 test points; a square-root softplus adds none. One request at a time on an NVIDIA A100, the encoder takes 29 ms per decision against 68 ms, but uses more peak memory (2.8 against 2.2 GiB). An auxiliary head, trained on human-marked evidence and on labels selected with a reward adapted from SelfCite, predicts in the same pass the sentences annotated as support for the answer.

0.19

From Retrieval to Typed Decisions: Calibrated System One Models from Biomedical Sentence Encoders

cs.CL cs.AI ·2026-10-01 ·Pritam Deka
highly cited similarity 0.64
Abstract

Typed decision models answer schema-constrained questions about a text in one forward pass and return probabilities meant to be thresholded. We ask whether biomedical sentence encoders trained for retrieval are good starting points for such models. We present SBERT2S1, which converts Sentence-Transformers encoders into bi-encoder, cross-head (C) and prior-fused residual (PFR) decision models, together with BIODECIDE, a biomedical typed-decision suite, and MEDLINE-S1, 243k training decisions derived from NLM indexing. Across six parent-retriever pairs, retrieval training improves zero-shot matching of content-bearing options. After fine-tuning, its effect depends on the head: across five pairs and three training-set sizes, retrieval training significantly helps PFR, which keeps the retrieval prior, in 10 of 15 comparisons, but helps C in one and hurts it in five. A matched grid of two heads and five training objectives shows that C outperforms PFR under every objective, and that the released RLCD recipe of open System One models trails cross-entropy by 2.5-3.0 points. The deficit stems mainly from its reward normalisation, which inflates the noisy score-function term 3.6-15-fold; an unbiased leave-one-out estimator recovers most of the gap. After temperature scaling, no objective is clearly better calibrated than cross-entropy. We release the code, the MEDLINE-S1 labels and a model.

0.44

LEV: Thinking Fast and Deep with Looped Decision Models

cs.AI cs.CLcs.LG ·2026-10-14 ·Dingjie Song, Zhiling Yan, Yuxuan Zhang, Lichao Sun
code/datacommunity pick similarity 0.63
Abstract · 28 pages. Code: https://github.com/OpenLAIR/lev

Decision models answer typed questions about an input with one probability per option, computed in a single forward pass without generating text. Like the fast System 1 of dual-process theory, they answer at once, but their depth is fixed at training time, so they spend the same computation on easy and hard questions. Models that reason in generated text can think longer, but they pay for every extra step in decoded tokens. We present LEV, a decision model that thinks deeper by rerunning its layers in a loop instead of generating tokens. LEV builds on a language model pretrained as a loop, reads the options after every loop step and trains all steps jointly, so one checkpoint can answer at any depth. On three looped backbones, accuracy rises with depth on the same weights, mostly on hard questions, and a 2.6B LEV goes from 63.2% at depth one to 76.2% at depth four on JevBench. On-demand depth, set per question by a small stopper with a threshold fitted on unlabelled requests, saves 15% to 24% of the computation at a cost of under one accuracy point, level with or slightly above a fixed mix of depths. On a locked test set, the 1.4B and 3B models are 5.4 to 8.8 points more accurate than dense (non-looped) models and cascades of the same latency below 50 ms.

0.68

Preference Is Not Position: Evaluating Reranking on NevIR in Fixed Hybrid Candidate Pools

cs.IR ·2026-10-19 ·Yuning Bian, Lele Xu, Yifan Yuan, Xinyu Yin, Chengyan Zhan
publishedcode/data similarity 0.63
Abstract · 6 pages, 2 figures, 3 tables. Accepted at the IEEE BigData 2026 Undergraduate and REU Consortium. Code: https://github.com/ql-link/LinkRag-E…

Near-identical passages can differ in a decisive condition, such as negation. A ranker may order such a pair correctly while placing the preferred passage far down the retrieved list. We examine this distinction using official NevIR pairs jointly covered by fixed candidate pools from dense, learned sparse, and BM25 retrieval. On 2,736 Test query directions, a pairwise-trained ranker improves strict per-query preference over fixed fusion (56.62% versus 52.30%) while reducing preferred-passage mean reciprocal rank (MRR) from 0.5764 to 0.3055. An exploratory fixed-model replay of a variant with background sampling and revised supervision yields MRR of 0.5859 and strict preference of 54.53%. Bounded pointwise reranking of the fusion top 20 improves both outcomes. Qwen3-14B-AWQ with fusion tie-breaking reaches 77.45% strict preference, 58.99% NevIR-style paired accuracy across 1,363 complete pairs, and preferred-passage MRR of 0.7509. A BGE cross-encoder reaches 69.77% strict preference and MRR of 0.7404. A post-hoc analysis finds that the branch where equal Qwen grades for the two designated passages trigger fusion tie-breaking accounts for 27% of evaluated directions and contains most remaining strict-preference failures. These findings support joint evaluation of pairwise preference, designated-passage position, and computational cost, while distinguishing model judgments from the combined ranking procedure.

0.50