RAGScore — Claude Code MCP server: what it does, how to install
Turns your docs into a question set that tests a RAG endpoint, then reports accuracy and why answers failed. Runs on a local model too.
claude mcp add ragscore -e OPENAI_API_KEY=<OPENAI_API_KEY> -e ANTHROPIC_API_KEY=<ANTHROPIC_API_KEY> -- uvx ragscoreWhat RAGScore does
RAGScore works in two stages. It builds question-answer pairs from the files you point it at, then sends each question to your RAG over HTTP and scores what comes back. The report shows overall accuracy and the pairs that went wrong.
Add --detailed and every answer gets five scores: correctness, completeness, relevance, conciseness, faithfulness. The --diagnose flag sorts failures into four buckets — the retriever came back empty, the generator invented something, the reply was partial, or the content was misread. That breakdown leans on the support_span saved with each pair.
Scoring runs on OpenAI or Anthropic, and both keys are passed as environment variables when you register the server. Ollama gives you a local path with no cloud keys at all. Pick a provider and supply its key, or evaluation stops at the first call.
What it helps with
- build a question set from project docs
- measure accuracy of a RAG endpoint
- list the questions your RAG answers badly
- tell a retrieval miss from a hallucination
- score a RAG without sending docs to the cloud
Example prompts for the agent
- Generate QA pairs from
docs/and score my RAG athttp://localhost:8000/query - Run the evaluation with
--detailed --diagnoseand count retriever misses - Create a QA set for the
developersaudience with theapi-integrationpurpose
How to install RAGScore
Platform CLI
claude mcp add ragscore -e OPENAI_API_KEY=<OPENAI_API_KEY> -e ANTHROPIC_API_KEY=<ANTHROPIC_API_KEY> -- uvx ragscoreReplace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from ragscore <command>
Platform CLI
codex mcp add ragscore --env OPENAI_API_KEY=<OPENAI_API_KEY> --env ANTHROPIC_API_KEY=<ANTHROPIC_API_KEY> -- uvx ragscoreReplace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from ragscore <command>
Config file
{
"mcpServers": {
"ragscore": {
"command": "uvx",
"args": [
"ragscore"
],
"env": {
"OPENAI_API_KEY": "<OPENAI_API_KEY>",
"ANTHROPIC_API_KEY": "<ANTHROPIC_API_KEY>"
}
}
}
}Replace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from ragscore <command>
Config file
{
"servers": {
"ragscore": {
"type": "stdio",
"command": "uvx",
"args": [
"ragscore"
],
"env": {
"OPENAI_API_KEY": "<OPENAI_API_KEY>",
"ANTHROPIC_API_KEY": "<ANTHROPIC_API_KEY>"
}
}
}
}Replace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from ragscore <command>
Platform CLI
gemini mcp add -e OPENAI_API_KEY=<OPENAI_API_KEY> -e ANTHROPIC_API_KEY=<ANTHROPIC_API_KEY> ragscore uvx ragscoreReplace <…> with your own values. uvx runs the command named after the package; if the README names another one, use uvx --from ragscore <command>
Step by step, with commands for each platform: How to add an MCP server to Claude Code, Codex, Cursor, VS Code, Gemini CLI and more
Questions about RAGScore
How do I install RAGScore in Claude Code?
Run claude mcp add ragscore -e OPENAI_API_KEY=<OPENAI_API_KEY> -e ANTHROPIC_API_KEY=<ANTHROPIC_API_KEY> -- uvx ragscore with your own keys in place.
What does RAGScore do?
It generates question-answer pairs from your documents and tests your RAG endpoint: one accuracy figure, five per-answer metrics, and a breakdown of failure causes.
Can I skip the Anthropic and Anthropic keys?
Yes, by scoring with a local Ollama model. Your documents and questions then stay on your machine.
