scenario-model-comparison — Claude Code skill: bake off Scenario

---
name: scenario-model-comparison
description: "Use when comparing Scenario models on the same brief through MCP: a bake-off with shared prompts and inputs, cost, generation time and quality measured side by side, choosing a default model, testing a new release against the incumbent, or finding which model handles a style, edit, or reference image best. Keywords: model comparison, benchmark, bake-off, A/B test, contact sheet, cost per asset, latency."
license: MIT
---

# Scenario Model Comparison

## Overview

A comparison is one brief run unchanged across several models, then judged on criteria written down
before the first paid call. Every candidate has its own contract, so the comparison lives in the
normalization: what is held fixed, what each schema forces to differ, and where each number came
from. `recommend` shortlists, `dry_run` prices, the `jobs_wait` rows carry the billed cost, the grid
tool puts the results side by side, and a table delivers. Connection and the core loop: see the
`scenario` skill; a rubric for judging output against a brief: `scenario-refine-loop`. If a sibling
skill named here is missing from your available skills, ask the user to install it
(`npx skills add scenario-labs/skills --skill <name>`); unattended, proceed from tool schemas and
flag the gap.

## Quick reference

| Step         | Do                                                                                                                                                |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1. Frame     | One brief, its inputs (prompt text, reference asset ids), three to five pass/fail criteria, all written before running                            |
| 2. Shortlist | `recommend` with the brief as `prompt` (`limit` up to 10); `search` for candidates the user named; keep three to five                             |
| 3. Normalize | `model_schema_get` each candidate: the shared fields, the caps that differ, the flags that rewrite prompts                                        |
| 4. Price     | `model_run` with `dry_run: true` (top-level, beside `model_id`, never inside `parameters`) per candidate and utility model; apply the Budget gate |
| 5. Run       | `model_run` with top-level `wait: false` per candidate, launched back to back, then one `jobs_wait` over all the job ids                          |
| 6. Measure   | `cuCost` from the `jobs_wait` rows; `createdAt` to `updatedAt` from `job_get` for the seconds                                                     |
| 7. Judge     | The sheet from `model_scenario-grid-maker`, then each asset at full size, scored against the criteria                                             |
| 8. Deliver   | One table row per candidate; the assets filed in one collection and tagged by model id                                                            |

## Normalization

- Hold the brief fixed: the same prompt text byte for byte, the same reference asset ids, the same
  output count (one, unless every candidate exposes the same batch field; a model whose default
  sample count is above one multiplies its cost and must be pinned).
- Let the schemas differ only where they must. Sizes are enums or pixel pairs per model: pick the
  closest each allows to the brief's target and record the delivered `width` and `height` from
  `asset_get` beside the result, since a larger output costs more and reads sharper. Prompt caps
  differ (`max_length`): an overrun is a 400, never an implicit trim. Preserve an exact user brief:
  replace a candidate whose cap is too small, or ask before changing the brief for every candidate.
- Turn off what rewrites the prompt: a prompt-expansion or optimize flag left on for one candidate
  compares its rewrite, not the brief. Where a flag cannot be turned off, say so in the table.
- Seeds do not compare across models: a seed reproduces one model's own sampling and transfers
  nothing. Leave `seed` unset, or set it only for repeat runs of the same candidate.
- A Turbo and a Quality member of one family are two candidates, not one. A LoRA runs through its
  base (`runs_as` and `run_with`, see `scenario`), and its cost and time are the base run's.
- Where one family's skill prescribes tuning (a reference tag syntax, a strength dial), fairness
  means two rows for that candidate, defaults and tuned, rather than a tuned one against untuned
  rivals.

## Budget

Price generations and required grid, extraction, or analysis steps. For a hard total cap, count each
launched job once, including in-flight commitments. Before the first and every later paid call,
require: committed spend + the next call's quote + verified reservations for remaining required work
<= the cap. A substitute-input quote cannot verify a reservation unless documented pricing proves it
bounds the real payload. If the sheet needs outputs that do not yet exist and has no verified bound,
do not launch the generations: propose staged spending or a revised plan and wait for the user's
agreement. Do not silently drop requested candidates or artifacts to fit. `usage` reports consumed
CU, not remaining credits; it cannot certify a balance for this budget gate.

## Measurement

`recommend` reports modeled cost and latency, right for the shortlist and wrong as a result:
`dry_run` prices the exact payload, and the `jobs_wait` row's `cuCost` is what was billed. Time
comes from `job_get`, which returns `createdAt` and `updatedAt`; their difference on a finished job
is the wall-clock latency including queue time, comparable across candidates launched within the
same minute. Launch every candidate before waiting on any, within the team's concurrency ceiling
(the 429 rows in `scenario`). Use the returned `actionLimit` on a concurrency 429; never infer the
ceiling from this comparison's batch size. Re-call `jobs_wait` with `pending_job_ids` until every
row is terminal. Record per candidate: model id, the exact `parameters` sent, `cuCost`, seconds,
delivered dimensions, asset ids, and any normalization it forced (a size, a cap, a flag).

Video candidates are compared on contact sheets, never on a first frame: sweep each clip with
`model_scenario-video-to-image-seq` (a fixed first-party id, Scenario's single deterministic frame
extractor, so discovery would only re-derive it), wait for the extraction job, then sheet the frames
per candidate; the extractor's frame-order and stride contract and the sheet's 100-image cap are in
`scenario-video-editing`. Audio and 3D candidates are compared on the assets themselves through
`asset_display`.

## Judging

Pre-registered criteria are pass/fail statements about the output ("the label text is legible", "the
scar is on the left cheek", "no extra fingers", "the background is plain"), so a result is scored,
not admired, and a surprising winner cannot rewrite the test after the fact. Score the literal
criterion: a related feature or partial match is not a pass; ambiguous evidence is unverified. Build
the sheet with `model_scenario-grid-maker` (a fixed first-party id, Scenario's single deterministic
grid tool, so discovery would only re-derive it): `images` in candidate order, wrapped as an array
even for one, since the array order is the legend (the tool has no labels), `columns` equal to the
candidate count so one row is one brief, `cellRatio` matching the outputs, `padding` for a gutter.
Then `asset_display` each output at full size: a sheet hides fine text, edge halos and small
anatomy. Score every criterion for every candidate, and put cost and seconds in the same table so
the trade-off is read in one place. For a large set, `asset_analyze` (catalog, cost-bearing, and
write-class: `scenario_tools_search` then `scenario_tool_execute_write`, see `scenario`) can score a
batch: the criteria list verbatim as its `instruction`, the outputs as `images`, one verdict per
criterion per image, and a criterion the image cannot settle at output resolution (small lettering,
a fine edge) recorded as unverified, never as a pass; pre-register that instruction with the
criteria.

## Worked example: three image models on one prop brief

1. Frame: "a rusty iron key with a skull-shaped bow, hand-painted style, plain background", a 1024
   square, criteria: skull bow present, exactly one key, no lettering, plain background.
2. `recommend` with that brief as `prompt` and `limit: 5`, then `search` with `target="models"`,
   `public=true` for the two models the user named; keep three. Never hardcode the ids: catalogs
   differ per team.
3. `model_schema_get` on each: prompt cap, size fields, sample-count field, any prompt-expansion
   flag.
4. `model_run` with `dry_run: true` (top-level) on each, the same prompt, the count field at one,
   the closest 1024 square each allows, the expansion flag off; write the three prices down and
   apply the Budget gate, including the required sheet.
5. `model_run` three times with `wait: false`, then one `jobs_wait` with the three job ids,
   re-called with `pending_job_ids` on a timeout; read `cuCost` off each row and `createdAt` and
   `updatedAt` off `job_get` for each job.
6. `model_scenario-grid-maker` with `images` as the three asset ids in candidate order and
   `columns: 3`; `asset_display` the sheet, then each asset; score the four criteria.
7. Deliver the table (model, parameters that differed, cost, seconds, dimensions, criteria passed,
   notes) and file the three outputs and the sheet in a collection (`collection_create`, then
   `collection_add_assets`), tagging each output with its model id through `asset_add_tags`.

## Common mistakes

- Quoting `recommend`'s numbers as results: they are modeled. Bill from `jobs_wait`, time from
  `job_get`.
- Different sizes or sample counts across candidates, or one candidate's prompt expansion left on:
  the comparison then measures the payload difference.
- One sample per candidate on a brief whose output varies wildly: run two or three where the budget
  allows, and say how many in the table.
- Reading a winner as a constant: the ranking is per brief and per team catalog. Re-run when either
  changes, and never write the winning id into a skill or a pipeline as a fixed value.
- Comparing video by first frame or a single still: sweep the clips into contact sheets.
- Shipping the sheet as the deliverable: the table with costs and criteria is the result; the sheet
  is evidence.
- Pasting signed download URLs into the report: file the assets and share the asset ids.