Skip to main content
command CX Codex

obsidian-retrieval-eval — Claude Code command: setup and eval

obsidian-retrieval-eval evaluates Obsidian vault search quality using recall@k and MRR metrics. It tests search ranking against natural-language questions before changing indexing.

MIT ★ 5,412 updated 2026-10-24

Install into Claude Code · Plugin marketplace All methods and platforms ↓
/plugin marketplace add eugeniughelbur/obsidian-second-brain
/plugin install obsidian-second-brain@obsidian-second-brain
Open on GitHub

What obsidian-retrieval-eval does

The command evaluates the search engine in integrations/obsidian-mcp-server/vault_ops.py against questions with known target notes. It runs scripts/eval/retrieval_eval.py via uv run and computes recall@1, recall@3, recall@5, recall@10 alongside MRR. The JSON output highlights missed targets and notes buried below third rank, often caused by verbose logs or files in raw/ outranking canonical pages.

Evaluation datasets live in scripts/eval/retrieval_cases.jsonl. Without an XAI_API_KEY, generating new test questions falls back to verbatim sentence extraction instead of natural paraphrasing. Passing the report argument writes baseline metrics directly into the vault at wiki/concepts/.

What it helps with

  • measure Obsidian retrieval accuracy via recall and MRR
  • detect noisy logs and raw files that outrank canonical notes
  • test search ranking adjustments against a consistent benchmark
  • record baseline retrieval metrics directly inside the vault

Example prompts for the agent

  • /obsidian-retrieval-eval
  • Run /obsidian-retrieval-eval 30 to test search quality
  • /obsidian-retrieval-eval report

How to install obsidian-retrieval-eval

1Plugin marketplace

/plugin marketplace add eugeniughelbur/obsidian-second-brain
/plugin install obsidian-second-brain@obsidian-second-brain

2Manually: put the files in place

.claude/commands/obsidian-retrieval-eval.md

Call it with /obsidian-retrieval-eval. For all projects use ~/.claude/commands/

Questions about obsidian-retrieval-eval

How to install obsidian-retrieval-eval in Claude Code?

Run /plugin marketplace add eugeniughelbur/obsidian-second-brain first, then execute /plugin install obsidian-second-brain@obsidian-second-brain.

What does obsidian-retrieval-eval do?

It evaluates Obsidian vault search using recall@k and MRR metrics on natural-language queries, pinpointing misranked notes.

What is required to generate new evaluation cases?

It needs uv to run the Python script and an XAI_API_KEY variable. Without the key, question generation uses exact sentence matches instead of paraphrases.

Source text

obsidian-retrieval-eval.md · MIT
---
description: Measure how well vault search finds the right note for a natural-language question - recall@k and MRR, with the concrete failures
category: meta
triggers_en: ["evaluate retrieval", "how good is my vault search", "retrieval eval", "test vault search quality", "measure find quality"]
triggers_de: ["Abrufqualität bewerten", "wie gut ist meine Vault-Suche", "Retrieval-Evaluierung", "Qualität der Vault-Suche testen", "Suchqualität messen"]
triggers_es: ["evalúa la búsqueda", "qué tal funciona la búsqueda de mi vault", "evaluación de recuperación", "prueba la calidad de la búsqueda", "comprueba si encuentra bien mis notas"]
triggers_pt: ["avalie a recuperação", "quão boa é a busca do meu vault", "avaliação de recuperação", "teste a qualidade da busca do vault", "meça a qualidade da busca"]
triggers_zh: ["测试知识库搜索效果", "评估检索质量", "看看我的笔记好不好找", "测量语义搜索召回率", "检查搜索能不能找到正确笔记"]
---

Use the obsidian-second-brain skill. Execute `/obsidian-retrieval-eval $ARGUMENTS`:

You cannot improve retrieval you have not measured. This scores the vault's REAL search (the term-frequency, title-weighted ranking in `integrations/obsidian-mcp-server/vault_ops.py`, the same engine behind `/obsidian-find` and the MCP connector) against natural-language questions whose correct answer note is known - so "should I add a vector index / better ranking?" becomes a number and a list of failures, not a hunch.

The optional argument is a number of cases to (re)generate first (without `XAI_API_KEY` the generator silently falls back to verbatim-sentence heuristics - fine for keyword-style cases, weak for paraphrase-style) (e.g. `30`), or `report` to also write the result to the vault. No argument: evaluate the existing cases.

1. Read `_CLAUDE.md` first if it exists in the vault root.

2. If the user asked to generate (or no cases file exists yet at `scripts/eval/retrieval_cases.jsonl`), bootstrap the eval set from the vault. Run from the skill root (its absolute path was given at session start as **Skill root**; substitute it for `SKILL_ROOT`):
   ```bash
   uv run --directory "SKILL_ROOT" python scripts/eval/retrieval_eval.py --generate 30
   ```
   This samples real notes and, for each, has an LLM write a question whose answer is in that note while AVOIDING the note's title words (so it tests retrieval, not string match). The note's path is the gold answer. Cases are gitignored - they contain private note paths.

3. Run the evaluation (from the same skill root):
   ```bash
   uv run --directory "SKILL_ROOT" python scripts/eval/retrieval_eval.py --json
   ```
   It prints recall@1/3/5/10, MRR, and per-case results: misses (gold note never in the top 10) and buried cases (gold ranked below #3, usually because a noisy high-mention note or a `raw/` transcript outscored the canonical note).

4. Interpret the numbers for the user in plain language: what fraction of natural-language questions surface the right note, and the dominant failure pattern (e.g. long `raw/` sources and `log.md` outranking short canonical notes; title-weighting helping only when the query reuses title words). Name the specific notes that wrongly ranked #1 from the JSON.

5. Turn failures into concrete, testable retrieval fixes ranked by leverage - e.g. exclude `raw/` and `log.md` from search, boost notes by `type:` (canonical concept/entity/project over transcripts), add alias/heading indexing, or add semantic matching. Each fix is a hypothesis: re-run this eval on the SAME cases after the change to confirm it actually raised recall, never assume.

6. If the user passed `report` (or asks to save), write an AI-first baseline note to `wiki/concepts/` (resolve per `references/folder-map.md`), `type: synthesis`, tagged `[retrieval, eval]`, linked from a retrieval-quality project note if one exists: record the engine, the case count, the recall@k / MRR numbers with the date as a recency marker, the top failure patterns with example notes, and the ranked fix hypotheses. This is the before-number every future retrieval change is measured against.

Measure, change one thing, measure again. That loop - not a guess about vector databases - is how retrieval actually improves.

---

**AI-first rule:** Every note created or updated by this command MUST follow `references/ai-first-rules.md` - `## For future agent` preamble, rich frontmatter (`type`, `date`, `tags`, `ai-first: true`, plus type-specific fields), recency markers per external claim, mandatory `[[wikilinks]]` for every person/project/concept referenced, sources preserved verbatim with URLs inline, and confidence levels where applicable. If that path does not resolve from your working directory, search upward for it; if you still cannot read it, say so before writing rather than producing a note that silently skips the rule. The vault is for future agent retrieval - not human reading.

**Anti-fabrication:** Report the eval numbers exactly as the harness emits them - never round a miss into a hit or invent a recall figure. If the cases are few or the gold labels look weak, say so rather than overclaiming. See the anti-fabrication and search-completeness hard rules in `references/ai-first-rules.md`.

© Eugeniu Ghelbur · MIT · eugeniughelbur/obsidian-second-brain

Similar slash commands

All slash commands for Notes, tasks and projects →

From the same set51

The whole set →
obsidian-decide obsidian-decide — a Claude Code command that extracts decisions from a conversation and files them in Obsidian, as dated lines or, with --formal, as a full ADR. CX Codex +1 ★ 5,412repo stars obsidian-save obsidian-save extracts decisions, tasks, ideas, and contacts from a Claude Code session directly into an Obsidian vault. Useful for archiving working context without manual note-taking. CX Codex +1 ★ 5,412repo stars obsidian-learn The obsidian-learn command reviews lessons in an Obsidian vault, marks stale or replaced findings, and surfaces repeated patterns for _CLAUDE.md. CX Codex +1 ★ 5,412repo stars obsidian-catchup obsidian-catchup — a command to review everything you dumped into the Telegram journal bot on the go—voice, text, images, PDFs, links—and integrate it into your Obsidian vault with Claude Code when you're back at the laptop. CX Codex +1 ★ 5,412repo stars obsidian-projects obsidian-projects is a Claude Code command that builds project status from Obsidian notes, Git, and local docs. Ask for an overview or name one project to inspect. CX Codex +1 ★ 5,412repo stars obsidian-brainstorm obsidian-brainstorm is a Claude Code command that interviews you about an idea one question at a time, then writes a design note with alternatives and a recommendation. CX Codex +1 ★ 5,412repo stars podcast Extracts podcast episode metadata, transcript, and structured summary directly into an Obsidian vault note. CX Codex +1 ★ 5,412repo stars youtube Pulls a YouTube transcript, metadata, and top comments into an Obsidian note. Add --visual when the video’s frames matter too. CX Codex +1 ★ 5,412repo stars

Project sponsors

Companies that keep this analytics open