AutoTrust-JEV-27B v1.1: A decision model that only answers multiple-choice questions

AutoTrust Labs released the 270 billion parameter text decision model v1.1 (named AutoTrust-JEV-27B v1.1 in the model card). The upstream statement included in the repository indicates that it is built on top of d3 (Decision 3.0) from the vLLM Semantic Router team, with Qwen3.8-27B as the base model. It does not generate free-form answers: questions and candidate options are fed in using a fixed format, and the model outputs only a single token restricted to the option codes; the engine then converts log probabilities into probabilities for each option. It supports only single-choice and yes/no questions, and accepts only text input. Although the repository name includes "Vision," the model card specifies that the input is text-only.

Original title: autotrust/AutoTrust-Decider-Pro-Vision-27B-v1.1

AutoTrust Labs released the 270 billion parameter text decision model v1.1 (named AutoTrust-JEV-27B v1.1 in the model card). The upstream statement included in the repository indicates that it is built on top of d3 (Decision 3.0) from the vLLM Semantic Router team, with Qwen3.8-27B as the base model. It does not generate free-form answers: questions and candidate options are fed in using a fixed format, and the model outputs only a single token restricted to the option codes; the engine then converts log probabilities into probabilities for each option. It supports only single-choice and yes/no questions, and accepts only text input. Although the repository name includes "Vision," the model card specifies that the input is text-only.

Why it matters For agent tasks such as tool selection, intent classification, and deciding whether to call an API, it outputs only an option code without generating lengthy reasoning.

Rising score 112 · Likes 1561 · Downloads 151462

Reasons

  • The authors claim a public index score of 67.31 on the public Decision Index 0.3 suite (37 benchmarks, 140,178 scorable requests), with the highest score in the Tools and Automation domain at 85.19 skill points.
  • In the per-item scores, CLINC150+OOS intent classification, BFCL tool selection, and BANKING77 banking intent classification all score above 93. Compared to JEV-27B in the same table, it scores higher on most benchmarks, but still scores lower than Jev 1.14 on reasoning tasks such as MMLU-Pro, GPQA Diamond, and HLE.
  • Evaluation results and score archives are available in the dataset autotrust/jev-decision-index-results for cross-verification.

Usage

  • Install vllm==0.31.0, transformers==5.17.0, and the fixed version of the StartLux-Decision package specified in the model card. After downloading the model, start the service using serve_vllm.sh in the repository (set MAX_LEN=32768 for 80GB GPUs).
  • The vLLM endpoint itself is not a decision interface: you must use jev_vllm_engine.py in the repository to format questions according to jev_format.py, and convert the log probabilities of the option codes into probabilities for each option.
  • After installing the Decision Index evaluation suite, you can rerun evaluations using python -m decision_index run --engine jev_vllm_engine:JevVllmEngine. The download command in the model card uses the old name AutoTrust-JEV-27B-v1.1, which redirects to this repository on Hugging Face.

Risks

  • The scores are self-reported results run locally by the authors on a single NVIDIA B300 GPU. The model card states that these results are still pending verification by leaderboard maintainers, and no private or complete scores are reported.
  • The authors caution that these scores come from adapted versions of tasks, do not represent reliability in real-world scenarios, and do not indicate that probabilities are calibrated. Validate with your own data before using for high-stakes decisions. HLE performance is close to random chance.
  • It depends on fixed versions of vLLM and the StartLux plugin (bidirectional reading of the entire prompt, disabling prefix caching, etc.). Results may differ if the inference framework is changed. This entry has not been run.

Open

Hugging Face Trending

Explore the latest AI Radar updates