SingProbe is an attention probe embedded within DeepSeek-V4.1-Flash-1017, which reads the hidden states of layers 12, 25, and 39 to provide per-token scores for 8 question-intent categories, answer safety, and hallucination risk during generation. The model card reports an F1 score of 0.8756 for question-intent classification, an F1 score of 0.8668 for answer safety classification, a streaming answer-level AUC of 0.9907 for safety, and a hallucination detection AUC of 0.7951, which is close to the 0.8000 of the baseline method DRIFT; the method and training code are available in the technical report and the SingProbe repository.
Why It Matters The probe directly reads hidden states from inside the model with a very small parameter count, eliminating the need to run a separate guardrail model. The authors claim that its performance across multiple safety classification tasks is comparable to or better than dedicated guardrails in the 8B class.
Upvotes 2 · Likes 2 · Downloads 0
Reasons
- MIT license, uploaded on 11/19; includes an arXiv technical report and a training code repository
- The model card lists baseline comparisons (such as Qwen3Guard and YuFeng-XGuard) and the release-time acceptance thresholds
Usage
- Requires an SGLang branch with token-probe functionality and 8 H20 GPUs: use --probe-ckpt to point to this repository at startup, read scores from meta_info.token_probe_probs returned by /generate, where label_8 indicates answer unsafety and label_9 indicates hallucination risk
Risks
- The SGLang branch referenced for quick start is hosted on code.alipay.com; external accessibility is unknown, and the example commands also use the authors' internal paths
- All evaluation metrics are self-reported by the authors; it is only adapted for version 1017