issue-rag is a project that performs retrieval-augmented question answering on issues in open-source repositories. It was evaluated on vitejs/vite using 95 manually reviewed questions (with deepseek-chat used for both generation and grading). The author reports that closed-book accuracy was 8.4%, and the best retrieval configuration achieved 64.2%, but the proportion of responses judged as hallucinations also increased from 1.1% to 20%; the differences among the four retrieval configurations fall within the noise range of the grader itself. The grader was audited from four perspectives: switching models, re-running with the same model, changing scoring criteria, and 30 manually blind-graded responses.
Why it matters Provides a reproducible evaluation template for RAG practitioners: retrieval can improve accuracy, but it can also turn "I don't know" into "confidently wrong."
Stars 250 (+49 / 24h)
Reasons
- 49 new stars in 24 hours (indicating attention only)
- All 95 evaluation questions, maintainer reference answers, responses from both systems, and judgments are available on an online results page with search and filtering capabilities
Usage
- After running pip install -e ".[models,dev]", follow the Quickstart to fetch issues and comments, build the corpus, and run evaluations; setting GITHUB_TOKEN can improve fetching speed
Risks
- Retrieval supports English only; asking questions in Chinese will retrieve noise, a limitation acknowledged by the author
- Evaluation was conducted on a single repository; it is unknown whether the conclusions generalize to other projects