Skip to main content
skill CX Codex

mlflow-evaluation — Claude Code skill for GenAI evaluation

The mlflow-evaluation Claude Code skill configures MLflow scorers and evaluates an agent, dataset, precomputed outputs, or traces. It checks GenAI quality and inspects evaluation results.

Apache-2.0 184 installs ★ 91 updated 2026-10-23

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add mlflow/skills --skill mlflow-evaluation -a claude-code
Open on GitHub

What mlflow-evaluation does

The skill sets up MLflow scorers and runs GenAI evaluations on agents, datasets, supplied outputs, or traces. It uses MLflow’s dataset API and mlflow.genai.evaluate(). Before a full run, it evaluates three representative records, or a smaller limit set by the user.

For reruns, it inspects the agent entry point, makes a representative call, and fetches the saved trace. Precomputed outputs are evaluated without calling the agent again. The project needs MLflow 3.8 or later and configured provider credentials; when MLFLOW_TRACKING_URI and MLFLOW_EXPERIMENT_ID are set, the skill preserves those settings.

It checks the pilot results before proceeding and records the run ID and row-level outputs. Authorization, tool, or prediction failures stop the full evaluation until the issue is fixed.

What it helps with

  • score agent responses against agreed criteria
  • evaluate supplied outputs or traces without rerunning the agent
  • catch scorer and data issues in a pilot run
  • link scores to individual results and errors

Example prompts for the agent

  • Evaluate the agent’s responses against the quality criteria in the MLflow experiment.
  • Score these precomputed outputs without invoking the agent again.
  • Inspect the traces and scores, then show errors for each record.

How to install mlflow-evaluation

1Skills CLI (Vercel)

npx skills add mlflow/skills --skill mlflow-evaluation -a claude-code

2Manually: put the files in place

~/.claude/skills/mlflow-evaluation/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/mlflow-evaluation/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about mlflow-evaluation

How do I install mlflow-evaluation in Claude Code?

Run npx skills add mlflow/skills --skill mlflow-evaluation -a claude-code. The skill file is at ~/.claude/skills/mlflow-evaluation/SKILL.md.

What does mlflow-evaluation do?

It configures MLflow scorers and evaluates GenAI agents, datasets, supplied outputs, or traces. A pilot run is checked before the full evaluation, with results recorded for each row.

What does mlflow-evaluation require?

It requires MLflow 3.8 or later and configured provider credentials. The skill retains the settings in MLFLOW_TRACKING_URI and MLFLOW_EXPERIMENT_ID when both are set.

Source text

SKILL.md · Apache-2.0
---
name: mlflow-evaluation
description: Design MLflow scorers and run GenAI evaluation on an agent, dataset, precomputed outputs, or traces. Use for quality criteria, judges, MemAlign, datasets, evaluation runs, and results analysis.
---

# Agent evaluation

- Establish purpose, important outcomes, known failures, data/trace selection, and agreed scorers from the request, project, and configuration. Ask only about unresolved decisions.
- Reuse explicit criteria and implementations. For any scorer-design request, read [scorer design](references/scorer-design.md); also read [judge alignment](references/judge-alignment.md) when alignment is requested.
- For scorer-only requests, build and verify the scorer, then stop. Do not create a dataset or run an evaluation unless requested.

- Use `mlflow.genai.datasets`, built-ins / `@scorer` / `make_judge`, and `mlflow.genai.evaluate()`; no parallel evaluation framework.
- Accept existing records/DataFrames; persist new reusable datasets through the GenAI dataset API.

## Step 1: Reuse setup and select execution mode

- Preserve `MLFLOW_TRACKING_URI`, `MLFLOW_EXPERIMENT_ID`, provider credentials, and configured judge model. If
  both MLflow variables are set, skip setup: no replacement backend, new experiment, or `.env` override.
  For missing configuration, use `mlflow-onboarding` when available.
- Use the project environment (`uv run` for uv-managed projects). Check MLflow >=3.8 and installed signatures
  for the APIs needed. Keep stderr separate when capturing structured stdout.
- **Rerun mode:** inspect the real agent entry point and its configuration. Use
  `mlflow-trace-instrumentation` when available for instrumentation; reuse working tracing. Run a
  representative invocation, flush and fetch its persisted trace in the intended experiment. If tracing is
  missing/broken, repair and verify it before evaluation; do not add duplicate framework spans.
- **Precomputed mode:** evaluate the supplied `inputs` and `outputs`, or existing traces, **without
  `predict_fn`**. Do not invoke the agent, change stored records, or create a replacement dataset. Check
  scorer data requirements against existing evidence; a final-output scorer does not require new
  instrumentation.

## Step 2: Reuse and register scorers

- List registered scorers: `mlflow scorers list -x "$MLFLOW_EXPERIMENT_ID"`. Reuse the suite, models, and definitions; add only missing agreed scorers.
- Read [scorers.md](references/scorers.md) for model selection, registration, and retrieval. Register new scorers before evaluation; verify them in the intended experiment.

- Check installed `required_columns` and trace requirements: `Correctness` needs ground truth; retrieval scorers need `RETRIEVER`, not just `TOOL`, spans.
- Aggregate booleans, numbers, or `"yes"`/`"no"`. Stored `"pass"`/`"fail"` scores may disappear from aggregates.
- For custom scorers or missing scores, read [scorers-constraints.md](references/scorers-constraints.md).

## Step 3: Discover and bind data

- Use the installed dataset discovery API before creating data. Inspect candidates; reuse a suitable or explicitly selected dataset. Discovery failure does not mean none exists.
- For UC, use the intended fully qualified table and backend-supported discovery. Ask only if selection is ambiguous.

Read [dataset-preparation.md](references/dataset-preparation.md) when creating, adapting, or synthesizing
records. Inspect actual record keys and the predictor signature:

```python
# MLflow calls predict_fn(**record["inputs"]). Nested values stay nested.
record = {"inputs": {"query": "Where is order 42?", "context": {"order_id": 42}}}
def predict(query, context):
    return agent(query=query, context=context)  # bind project config here as needed
```

- Accept `inputs` as a wrapper argument only if the dataset has that input key.
- Put ground truth under `expectations`, e.g. `expectations.expected_response`. Preserve multi-key inputs and source IDs.
- Size data to the task/coverage; no mandatory 100-question expansion or synthesis dependency.

For requested corpus synthesis, use the [RAGAS recipe](references/dataset-preparation.md#ragas-corpus-synthesis).

## Step 4: Dry run: required before a full evaluation

Evaluate **three representative records**, or the user's smaller explicit budget, with the full run's mode, wrapper, and scorers. Use one pilot, not separate sanity/dry runs. Inspect every row:

- Did the predictor finish, required tools/retrieval execute, and usable output persist? Tools need not run
  for greetings or other cases that do not require them.
- Did every applicable scorer return a valid assessment without an error? Check both row-level
  values/rationales and aggregate inclusion; a successful trace status or run alone is insufficient.
- **Zero, `False`, and `"no"` are valid quality failures.** Investigate suspicious uniform scores against
  actual responses and known-good/bad cases; never require nonzero scores or change the rubric to manufacture
  passes.

- **Stop before the full run** on auth/tool failures, failed prediction, empty rerun responses, or missing/error-only scores. Diagnose, repair within scope, and repeat the pilot.
- Preserve broken precomputed outputs as evidence. Distinguish scored negatives from scorer/infrastructure failures.
- Keep pilot run IDs and label them as pilots when supported. Do not delete runs unless the user requests it.
- Report blockers if repair is unavailable. Never invent credentials or switch providers to retry.

## Step 5: Execute and inspect

- Reruns: `mlflow.genai.evaluate(data=selected_data, predict_fn=predict, scorers=registered_scorers)`. Precomputed: omit `predict_fn`.
- Estimate duration from pilot throughput. For expensive runs, read [throughput-guide.md](references/throughput-guide.md); honor resource limits and host-supported execution methods.
- Wait for completion. Keep exit status, run ID/URL, elapsed time, and row-level artifact path; launch alone is not success.

## Verification

- Verify a fresh run in the intended experiment, with row-level results for every selected record and applicable scorer.
- Preserve outputs, trace IDs, values, rationales, and assessment errors.
- Report evaluated, failed, missing, skipped, and errored counts; do not hide missing rows in pass-rate denominators. Metric-only runs and empty score tables are not success.

- Summarize criteria separately; link representative failures to evidence and state small-sample limits. Optional [report outline](assets/evaluation_report_template.md).
- Check summaries against raw typed scores/errors; generated grades/recommendations do not establish root cause.
- Requested behavior fixes: hand results to `mlflow-agent-improvement`.

## References

- Infrastructure, schema, or scorer failure: [troubleshooting.md](references/troubleshooting.md).
- Production-trace curation, version comparisons, or continuous evaluation:
  [advanced-evaluation.md](references/advanced-evaluation.md).
- Databricks production scoring:
  [databricks-online-monitoring.md](references/databricks-online-monitoring.md); registration alone does not
  start monitoring.
- Judge disagreement, domain-expert labels, Review App, or MemAlign: `mlflow-evaluation` and its [alignment
  reference](references/judge-alignment.md).
- Registered-prompt optimization / GEPA: [prompt-optimization.md](references/prompt-optimization.md).

- API details: [evaluation quickstart](https://mlflow.org/docs/latest/genai/eval-monitor/quickstart.md) or the reference's exact topic link. Check installed-version compatibility.
- Missing/moved page: [llms.txt](https://mlflow.org/docs/latest/llms.txt).
- Docs/skill unavailable: use local instructions and installed API introspection; stop and explain unresolved correctness. Do not silently substitute legacy APIs, backend, or provider.

© mlflow · Apache-2.0 · mlflow/skills

Similar skills

All skills for AI and ML →

From the same set17

The whole set →
mlflow-agent Routes MLflow requests to the right child skill for tracing, evaluation, debugging, or documentation. Use it when the task has no specific MLflow workflow selected. CX Codex +1 882installs sagemaker-mlflow sagemaker-mlflow connects the agent to SageMaker Managed MLflow via resource ARN (mlflow-app or mlflow-tracking-server), then hands off to other MLflow skills through MLFLOW_TRACKING_URI. CX Codex +1 394installs fix-agent-issue fix-agent-issue — a skill that walks the agent through EXPLORE → PLAN → IMPLEMENT → VERIFY so a behavior change rests on a trace and a test, not a guess. CX Codex +1 376installs build-a-scorer build-a-scorer helps design and iterate MLflow judges and scorers for LLM apps or RAG agents. It defines atomic criteria and picks the cheapest reliable check. CX Codex +1 339installs mlflow-trace-analysis The mlflow-trace-analysis Claude Code skill examines MLflow traces to explain agent failures, errors, and quality issues. It handles single traces and multi-turn sessions. CX Codex +1 178installs mlflow-trace-metrics The mlflow-trace-metrics Claude Code skill queries aggregate trace, span, and assessment metrics from MLflow. Runs telemetry summaries for latency, token counts, and evaluation scores. CX Codex +1 174installs mlflow-trace-retrieval The mlflow-trace-retrieval Claude Code skill retrieves MLflow traces by ID or searches them with filters. It runs for trace lookups based on experiments, runs, users, time windows, and other fields. CX Codex +1 174installs mlflow Equips Claude Code with workflows for tracing, debugging, and evaluating GenAI apps using MLflow. Helpful when instrumenting LLMOps. ★ 91repo stars

Project sponsors

Companies that keep this analytics open