Skip to main content
skill CX Codex

parallel-search-evaluation — Claude Code skill for search testing

The parallel-search-evaluation Claude Code skill measures search quality, cost, and latency across providers. It activates for Parallel evaluations, provider comparisons, benchmark runs, or troubleshooting evaluation results.

MIT 734 installs ★ 97 updated 2026-10-21

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add parallel-web/parallel-agent-skills --skill parallel-search-evaluation -a claude-code
Open on GitHub

What parallel-search-evaluation does

The skill turns an application's search workload into a repeatable provider comparison. It can produce an evaluation plan or, when implementation is requested, a runnable harness. Existing harnesses take priority over creating a new one.

The process uses real questions with independently checked answers where available, and records the dataset version and configuration. The model and other comparison conditions stay fixed across providers. For new integrations, the skill points to /v1/search and /v1/extract; REST Search requests need an explicit mode, since omission selects advanced.

Live runs require a Parallel API key in PARALLEL_API_KEY. The key should not be sent in chat. Without it, the skill can prepare the plan and test harness, but not run live evaluations.

What it helps with

  • compare Parallel with Exa, Tavily, or an existing search setup
  • build a repeatable search evaluation harness
  • measure search quality, cost, and latency
  • investigate SimpleQA or DeepSearchQA results

Example prompts for the agent

  • Compare Parallel with Tavily using questions from our application.
  • Plan a search evaluation with the SimpleQA benchmark.
  • Analyze our DeepSearchQA results and identify the main failure types.

How to install parallel-search-evaluation

1Skills CLI (Vercel)

npx skills add parallel-web/parallel-agent-skills --skill parallel-search-evaluation -a claude-code

2Manually: put the files in place

~/.claude/skills/parallel-search-evaluation/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/parallel-search-evaluation/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about parallel-search-evaluation

How do I install parallel-search-evaluation in Claude Code?

Run npx skills add parallel-web/parallel-agent-skills --skill parallel-search-evaluation -a claude-code. The skill file is at ~/.claude/skills/parallel-search-evaluation/SKILL.md.

What does parallel-search-evaluation do?

It helps compare search providers on application questions or public benchmarks. The skill can prepare an evaluation plan or a runnable harness, with quality, cost, and latency as comparison measures.

What is required for live evaluations?

A Parallel API key must be available as PARALLEL_API_KEY. It can go in the project's existing secret file; if there is none, the instructions say to create .env and add it to .gitignore.

Source text

SKILL.md · MIT
---
name: parallel-search-evaluation
description: Use for search evaluation, a POC or bake-off, Parallel vs Exa/Tavily comparisons, SimpleQA/DeepSearchQA benchmarks, or diagnosing search evaluation results. For product selection use parallel-api; for integration migration use migrate-to-parallel.
metadata:
  author: parallel
---

# Evaluate Parallel Search

Turn the user's workload into a reproducible end-to-end comparison. Deliver a runnable harness when implementation is requested, or an actionable plan when planning is requested. Reuse an existing harness before creating another one. Do not invent a Parallel CLI eval command.

To choose a Parallel API or build the integration, use `parallel-api`. To replace another provider's integration, use `migrate-to-parallel`. Use this skill to measure quality, cost and latency.

<!-- markdownlint-disable MD033 MD034 -->

## Onboarding

Follow this section when a Parallel setup prompt sends you here, or when the user is evaluating Parallel or comparing it with other providers.

- Work quietly: give progress updates only for decisions the user must make. Use the quoted messages below word for word, filling in <placeholders>. Anywhere else, write at most two short lines.
- Ask one question at a time, with your question UI if you have one.

1. When the comparison needs one of these decisions and the project doesn't answer it, ask:
   "What should Parallel be compared with? For example: Exa, Tavily, your current setup, or a public benchmark."
   "Do you have real questions from your app, with known answers?
   1. Yes: tell me where they are.
   2. No: start from a public benchmark."
2. API key: if PARALLEL_API_KEY is set in the shell or in the project's secret file, use that key (check without printing it). Otherwise, send:
   "Live runs need a Parallel API key:
   1. Sign in or sign up at https://platform.parallel.ai and create a key.
   2. Add PARALLEL_API_KEY=<your key> to <file>.
   3. Reply done, or reply skip to get the plan and the test harness without live runs."
   <file> is the project's existing secret file. If there is none, create .env and add .env to .gitignore first.
3. Ask: "Want a quick look first?
   1. Spot check: I'll run 5 of your questions through Parallel and <provider>, side by side.
   2. Full evaluation: a fixed question set, repeated runs, graded results.
   A spot check doesn't replace the full evaluation."
4. In its own message, report quality, cost, latency and the main failure types in at most ten lines, with the path to the full report.
5. If Parallel caused a problem (this skill, the docs, the API, the key or the skill install), ask in a separate message: "One issue came up during the evaluation: <one-line description>. Would you like a note to send to [email protected]?" Skip problems in the user's own project.

<!-- markdownlint-enable MD033 MD034 -->

## Start with the workload

Read the public [evaluation guide](https://docs.parallel.ai/search/evaluating-search), [Search best practices](https://docs.parallel.ai/search/best-practices), and [Extract best practices](https://docs.parallel.ai/extract/best-practices) before implementing requests. Check current [modes](https://docs.parallel.ai/search/modes) and [pricing](https://docs.parallel.ai/getting-started/pricing); record the documentation date and configuration rather than embedding price or latency promises.

Establish the application, question types, providers, existing agent/model, verified answers, and quality/cost/latency target. Infer these from supplied code and data; ask only for missing decisions that change the comparison. Prepare offline artifacts while credentials or a live-run budget remain unresolved. Use existing authorization for paid calls; otherwise establish scope and spend limits before execution. Never request keys in chat.

Choose the product that matches the application:

| Application | Evaluation boundary |
| --- | --- |
| User owns the agent loop | Search + Extract, including answer generation |
| One call should return a cited answer | Responses API |
| Schema-bound enrichment or asynchronous research | Task API |

Always set Search `mode` explicitly for REST requests: omitting it selects `advanced`, changing latency and price. Unless the user specifies otherwise, start with `fast` for interactive workloads or `advanced` for harder research, then test another relevant mode. The evaluation guide still recommends `basic`; record this discrepancy and the chosen rationale rather than silently inheriting that starting point. Honor workload constraints such as language support and endpoint availability. For Search MCP, consult its current routing and override rules: pin the mode for authenticated calls where supported; anonymous calls reject overrides, so record the actual server-selected mode or require authentication for a controlled mode comparison. Keep the model fixed within a comparison; a different model/mode pairing is a separate experiment.

## Build and freeze the comparison

Use [references/eval-contract.md](references/eval-contract.md) when creating dataset records, run manifests, traces, grading output, and reports. Resolve this reference relative to this skill's directory.

1. Prefer real production questions and independently verified answers. Match domain, wording, freshness, and task shape. Review synthetic labels before running. Pin question IDs and dataset version; separate pilot questions from held-out questions. Public benchmarks supplement this set and need a consistent policy for answer-key contamination.
2. Hold the model, research prompt, budgets, judge, and extraction policy constant across providers. Expose only the selected provider's search tool. Start with each provider's documented defaults and tool definition, including descriptions; change a setting only with evidence from the workload or pilot, and record the evidence and deviation in the manifest. Avoid arbitrary result caps or custom tool descriptions. If extraction differs as part of a provider stack, disclose that boundary. Keep gold answers and annotations out of agent prompts, query generation, and tool messages.
3. Have the model write an objective that restates the full information need with every relevant constraint, including entities, scope, dates, and source requirements. Pair it with focused keyword queries of 3–6 words each: one for a simple fact, up to three useful angles for richer questions. Do not duplicate the query text in the objective or copy the whole task into each query. Preserve result titles, URLs, and excerpts in the answering model's context. Use `/v1/search` and `/v1/extract` for new integrations; verify current request schemas rather than renaming old endpoints.
4. Default to multi-hop Search + Extract for applications; use single-hop only for an explicitly one-request benchmark. Fix total calls, deadline, retry policy, and spend bounds before dispatch. Use a fresh session per question, configuration, and repetition; reuse it across that trial's related Search and Extract calls.

Recommend a cost-parity comparison: align result count, content per page, product tier, and reasoning level where supported, and account for total Search/Extract/model cost including input and output tokens. Start with documented defaults as the baseline; any parity overrides need a recorded rationale and a separate configuration. Report requested and actual result counts and content volumes, residual mismatches, and settings that cannot be equated. Do not treat equal API-call counts as equal cost or imply parity when it was not achieved.

| Setting | Executor behavior |
| --- | --- |
| Single-hop | Allow exactly one Search API request, no Extract; disable tools for the final answer. Enforce the limit in code, including concurrent requests. Several keyword queries in one request still count as one call. |
| Multi-hop | Expose Search + Extract and allow evidence-driven follow-ups within the total budget. Dispatch independent calls from the same turn concurrently; reserve budget before dispatch, return results under their original call IDs, and preserve successes when siblings fail. |

Use the same research prompt across providers and settings; change tool availability and executor budgets separately. For public benchmarks, use the guide's research prompt and official grader where available. Record output-format overrides and label altered benchmark protocols as adapted evaluations.

## Pilot, grade, and interpret

Inspect a small fixed pilot: query quality, warnings, excerpt delivery, tool limits, concurrency, and trace completeness. Fix integration faults and rerun affected configurations; do not tune during a scored run. Freeze settings before evaluating held-out questions. Run each configuration at least three times and report variation.

Flag every failure to the user with affected trial IDs and impact: rate limits, HTTP errors, timeouts, partial results, token truncation, dropped results, and endpoint fallback (including `/v1` → `/v1beta`). Record each occurrence even if a retry succeeds. Never silently fall back or count a Parallel-side failure as a retrieval miss. Surface failures during execution and include their counts and trace references in the final report; preserve partial successes and the declared scoring denominator.

Grade final answers by task shape: factual correctness and citation support; list precision and recall; structured field accuracy; or an explicit research rubric. Require judge reasoning. For custom rubrics, rank honest inability to retrieve above confident misinformation. Hand-check at least 10% of judged trials, covering successes and failures. Keep citation audits separate from official benchmark scores.

Classify failures from traces: no search call, synthesis failure despite sufficient evidence, provider error, or retrieval miss despite a reasonable query. Do not attribute missing excerpts or other harness faults to retrieval quality. Distinguish account rate limits from provider capability.

Include failures and timeouts in primary results under the declared scoring policy. Bound retries and include all their time and cost. Report Search/Extract/model charges, tokens, calls, budget exhaustion, and search and end-to-end latency at p50/p95/p99. Time the whole task by wall clock, including query reformulation and answer generation; do not sum concurrent call durations. Cost per resolved task includes spend on unsuccessful trials; report it as undefined if nothing resolves.

Deliver the reproducible configuration, dataset and trace locations, grading rationale, failure breakdown, and quality versus cost/latency comparisons. Recommend configurations on the measured Pareto frontier that meet the user's targets, with uncertainty and workload slices. If calls were not run, label the deliverable as an unexecuted plan or harness and state what remains needed.

After results, propose 2–3 follow-up configurations tied to observed failures or bottlenecks: retrieval misses → `advanced`; thin excerpts → enable or increase Extract use; high latency → `fast`; high model-token cost → a smaller content budget. Change one factor per comparison, record the rationale, and use a new frozen manifest. Present these as proposals, not automatically launched runs; execute only when the user requests them within the authorized scope and spend.

© parallel · MIT · parallel-web/parallel-agent-skills

Similar skills

All skills for Testing and QA →

From the same set14

The whole set →
parallel-findall parallel-findall discovers companies, people, and products matching a natural-language description into a structured list. Useful when you need concrete entities rather than web pages or reports. CX Codex +1 13,000installs parallel-cli-setup parallel-cli-setup — a skill that walks the agent through installing parallel-cli, signing in, checking balance and installing the rest of the Parallel skills. CX Codex +1 6,600installs migrate-to-parallel migrate-to-parallel — a skill for Claude Code, Codex, Cursor and other IDEs that migrates Exa, Tavily, Perplexity or Firecrawl integrations to Parallel products while preserving application behavior. CX Codex +1 3,800installs parallel-memory parallel-memory — a skill that pulls past Parallel Task, Monitor and FindAll runs out of memory when they can help, and removes them on request. CX Codex +1 2,900installs parallel-mcp-setup parallel-mcp-setup walks through an authenticated Parallel Search MCP connection behind Bifrost: credentials, client config, tool access, and call verification. CX Codex +1 1,600installs choose-your-parallel-api choose-your-parallel-api guides an agent to the right Parallel API and settings for the job: live web data, a cited answer, reading a URL or PDF, or researching every row of a list. CX Codex +1 1,600installs parallel-search-setup parallel-search-setup walks through checking, using, and fixing the anonymous Parallel Search MCP server that ships with the plugin. Handy when search breaks or keeps returning 429. CX Codex +1 1,500installs parallel-api Integrates Parallel API search, page extraction, and web research into apps via Python or TypeScript SDKs. Runs when writing, debugging, or reviewing Parallel API code. CX Codex +1 280installs

Project sponsors

Companies that keep this analytics open