zaxiu88888/issue-rag

issue-rag is a project that performs retrieval-augmented question answering on issues in open-source repositories. It was evaluated on vitejs/vite using 95 manually reviewed questions (with deepseek-chat used for both generation and grading). The author reports that closed-book accuracy was 8.4%, and the best retrieval configuration achieved 64.2%, but the proportion of responses judged as hallucinations also increased from 1.1% to 20%; the differences among the four retrieval configurations fall within the noise range of the grader itself. The grader was audited from four perspectives: switching models, re-running with the same model, changing scoring criteria, and 30 manually blind-graded responses.

issue-rag is a project that performs retrieval-augmented question answering on issues in open-source repositories. It was evaluated on vitejs/vite using 95 manually reviewed questions (with deepseek-chat used for both generation and grading). The author reports that closed-book accuracy was 8.4%, and the best retrieval configuration achieved 64.2%, but the proportion of responses judged as hallucinations also increased from 1.1% to 20%; the differences among the four retrieval configurations fall within the noise range of the grader itself. The grader was audited from four perspectives: switching models, re-running with the same model, changing scoring criteria, and 30 manually blind-graded responses.

Why it matters Provides a reproducible evaluation template for RAG practitioners: retrieval can improve accuracy, but it can also turn "I don't know" into "confidently wrong."

Stars 250 (+49 / 24h)

Reasons

  • 49 new stars in 24 hours (indicating attention only)
  • All 95 evaluation questions, maintainer reference answers, responses from both systems, and judgments are available on an online results page with search and filtering capabilities

Usage

  • After running pip install -e ".[models,dev]", follow the Quickstart to fetch issues and comments, build the corpus, and run evaluations; setting GITHUB_TOKEN can improve fetching speed

Risks

  • Retrieval supports English only; asking questions in Chinese will retrieve noise, a limitation acknowledged by the author
  • Evaluation was conducted on a single repository; it is unknown whether the conclusions generalize to other projects

Open

GitHub

Explore the latest AI Radar updates