# RAGScore

Turns your docs into a question set that tests a RAG endpoint, then reports accuracy and why answers failed. Runs on a local model too.

- MCP server · `type=mcp` · Claude Code, Codex, Cursor, GitHub Copilot, Windsurf (Devin Desktop), Cline, Zed, Gemini CLI · ★ 15
- Author: HZYAI
- License: Apache-2.0 · policy=link-only · description only; the file stays with the author
- Source: https://github.com/HZYAI/RagScore
- Updated: 2026-05-29
- HTML: https://affpapa.org/en/nevibecode/mcp/ragscore

## What it does

RAGScore works in two stages. It builds question-answer pairs from the files you point it at, then sends each question to your RAG over HTTP and scores what comes back. The report shows overall accuracy and the pairs that went wrong.

Add `--detailed` and every answer gets five scores: correctness, completeness, relevance, conciseness, faithfulness. The `--diagnose` flag sorts failures into four buckets — the retriever came back empty, the generator invented something, the reply was partial, or the content was misread. That breakdown leans on the `support_span` saved with each pair.

Scoring runs on OpenAI or Anthropic, and both keys are passed as environment variables when you register the server. Ollama gives you a local path with no cloud keys at all. Pick a provider and supply its key, or evaluation stops at the first call.

## Requirements
- Needs: Python and uv (commands run via uvx)
- Environment variables: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` — keys and tokens: ask the human for the values, never commit them or put them in shared files.

## Install

### Claude Code (native platform)

**Platform CLI**
```bash
claude mcp add ragscore -e OPENAI_API_KEY=<OPENAI_API_KEY> -e ANTHROPIC_API_KEY=<ANTHROPIC_API_KEY> -- uvx ragscore
```
Replace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from ragscore <command>

### Codex

**Platform CLI**
```bash
codex mcp add ragscore --env OPENAI_API_KEY=<OPENAI_API_KEY> --env ANTHROPIC_API_KEY=<ANTHROPIC_API_KEY> -- uvx ragscore
```
Replace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from ragscore <command>

### Cursor

**Config file**
File `.cursor/mcp.json`:
```json
{
    "mcpServers": {
        "ragscore": {
            "command": "uvx",
            "args": [
                "ragscore"
            ],
            "env": {
                "OPENAI_API_KEY": "<OPENAI_API_KEY>",
                "ANTHROPIC_API_KEY": "<ANTHROPIC_API_KEY>"
            }
        }
    }
}
```
Replace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from ragscore <command>

### GitHub Copilot

**Config file**
File `.vscode/mcp.json`:
```json
{
    "servers": {
        "ragscore": {
            "type": "stdio",
            "command": "uvx",
            "args": [
                "ragscore"
            ],
            "env": {
                "OPENAI_API_KEY": "<OPENAI_API_KEY>",
                "ANTHROPIC_API_KEY": "<ANTHROPIC_API_KEY>"
            }
        }
    }
}
```
Replace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from ragscore <command>

### Gemini CLI

**Platform CLI**
```bash
gemini mcp add -e OPENAI_API_KEY=<OPENAI_API_KEY> -e ANTHROPIC_API_KEY=<ANTHROPIC_API_KEY> ragscore uvx ragscore
```
Replace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from ragscore <command>

## When to use it
- build a question set from project docs
- measure accuracy of a RAG endpoint
- list the questions your RAG answers badly
- tell a retrieval miss from a hallucination
- score a RAG without sending docs to the cloud

## Example prompts
- Generate QA pairs from `docs/` and score my RAG at `http://localhost:8000/query`
- Run the evaluation with `--detailed --diagnose` and count retriever misses
- Create a QA set for the `developers` audience with the `api-integration` purpose

## Similar
- [Front-End Checklist](https://affpapa.org/en/nevibecode/mcp/front-end-checklist.md) — MCP server · ★ 74559
- [Keploy](https://affpapa.org/en/nevibecode/mcp/keploy.md) — MCP server · ★ 18727
- [SeleniumBase MCP](https://affpapa.org/en/nevibecode/mcp/seleniumbase-mcp.md) — MCP server · ★ 13169
- [mobile-mcp](https://affpapa.org/en/nevibecode/mcp/mobile-mcp.md) — MCP server · ★ 9946
- [e2e](https://affpapa.org/en/nevibecode/mcp/e2e.md) — MCP server · ★ 11912
- [mockserver](https://affpapa.org/en/nevibecode/mcp/mockserver.md) — MCP server · ★ 4996

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md