test-value-evidence — скилл для Claude Code: что делает

---
name: test-value-evidence
description: >-
  Assess tests for an identified code or test change, feature, PR or commit. USE FOR: review
  changed behavior or regression tests, prove pre-fix/missing behavior is
  detected, revision-bound red-green evidence. Polyglot. DO NOT USE without an
  identified change: hypothetical mutations or general suite blind
  spots (test-gap-analysis). Also exclude per-test readiness/A-F grades (grade-tests),
  coverage (coverage-analysis), new suites (code-testing), or mutation engines.
license: MIT
---

# Test-value evidence

A passing test proves little unless it detects the relevant wrong behavior.
Assess the changed contracts and their directly related tests, not the entire
repository. Discover and follow the repository's instructions, pinned toolchain,
test commands, assertion conventions, and execution permissions. This skill
does not authorize edits in a read-only review, execution of untrusted code,
workflow dispatch, publication, or access to another session's checkout.

## Inputs and boundaries

Discover the requested diff/revision, production entry points, tests, fixtures,
and available execution artifacts. If the change cannot be identified, report
the missing scope instead of grading the workspace. Existing unchanged tests
can suffice; a source-only diff is not automatically untested.

Use `test-gap-analysis` for general existing-suite blind spots, `grade-tests`
for curated per-test readiness/quality, and existing repository mutation tooling
for mutation campaigns. Do not start a second mutation engine or require
full-suite mutation testing for every change. This skill is self-contained:
sibling skills are optional routing destinations, not required dependencies.

## Workflow

1. **Bind the contract to the tested revision.** For a PR, record its merge
   base and exact head SHA. For a commit, record the selected parent and commit;
   identify the parent explicitly for a merge commit. For uncommitted work,
   record the base SHA plus a reproducible patch identity (saved patch/hash,
   including relevant untracked files), not a fictitious committed revision.
   Evidence expires after relevant source, tests, helpers, configuration, or
   package changes unless revalidated.

   For each distinct behavior change, map:
   `requirement -> input/sequence -> expected public outcome -> production
   entry point -> named test/data case -> distinguishing assertion`.
   Derive the expected outcome independently from requirements or supported
   API/protocol semantics, not the implementation or the test's current value.
   Check build/package/configuration behavior, deleted/renamed code, removed
   tests, and shipping consumers as well as source. Include positive, negative,
   boundary, compatibility, cancellation, cleanup, and multi-module cases only
   where relevant to the changed contract.

   A mock of the changed component, self-comparison, or expected value computed
   by the same production algorithm does not provide an independent oracle.

2. **Choose the meaningful counterfactual.**

   | Change | Counterfactual | Required observation |
   |---|---|---|
   | Bug fix | Keep the regression test and restore the actual pre-fix defect. | The test fails for that defect, then passes with the fix. |
   | New feature | Disable/remove or corrupt the behavior, retaining the API shape and setup needed to build. | An assertion detects absent/wrong behavior, then passes with the feature. |
   | Stronger/test-only coverage | Introduce one plausible wrong public outcome. | The strengthened test detects it; the unmodified product passes. |
   | Behavior-preserving refactor | Compare the preserved contract before and after; optionally probe a relevant fault. | Preservation tests pass on both revisions; passing before is expected. |
   | Documentation/generated metadata only | Establish whether an executable contract changed. | Not applicable only with a concrete reason; generated or build metadata can change behavior. |

   For a fix, prefer the real pre-fix implementation with the same regression
   test, setup, dependencies, and inputs. If new APIs prevent that combination
   from compiling, restore only the defective behavior in the head version.
   Call this a **targeted counterfactual**, not a run on the base revision.
   Record its exact patch/hash. Never change assertions to manufacture red.

   For a feature, a missing-symbol compiler error is not behavioral proof.
   Choose a buildable no-op, disabled registration, old branch, or wrong output.
   A compiler/analyzer diagnostic can be valid behavior only when an executed
   harness asserts that exact diagnostic or its absence.

   Prefer small plausible faults: restore the old boundary, drop a required
   output, skip a callback, ignore an option, or return the wrong value.
   An equivalent mutation is not a gap. A crash, timeout, or exception counts
   only when it is the intended contract signal and diagnostics attribute it
   to the fault, not setup, discovery, flakes, or infrastructure.

3. **Execute paired runs only when authorized.** Implementation normally
   requires execution; read-only/advisory review normally requires inspection.
   A review request alone does not permit source edits or builds/tests.
   Never run untrusted code with privileged credentials.

   In an authorized implementation/verification environment:

   - Use an owned isolated checkout or experiment directory. Preserve the
     user's dirty files and other sessions/main checkout. Preflight concurrent
     processes and shared outputs/resources; do not kill unrelated processes.
   - Run the smallest relevant selection on the correct implementation.
     Confirm the named tests and data cases actually execute and pass.
   - Apply only the chosen defective behavior in the isolated experiment.
     Rebuild and rerun the same tests, assertions, data, dependencies, and
     applicable runtime configuration. Capture the named failure, assertion,
     expected/actual observation, process exit, and diagnostics.
   - Restore the correct implementation, rebuild, and rerun to confirm green
     again. Verify protected source/test bytes, remove only owned temporary
     resources, and never deliver the counterfactual patch in the product.

   Paired runs must not reuse stale build/test outputs or the other variant's
   effective caches. Use separate outputs/caches or clean only owned outputs
   and rebuild; do not rely on `--no-build` or a runner's success exit alone.
   Record selected/executed/passed/failed/skipped counts for each run. If a
   runner does not expose a count, say **unknown** rather than inventing it.
   Zero selection/execution, skips, missing tools, authentication, setup/build,
   unrelated failures, and unattributed timing flakes leave proof unverified.

   For cross-component/acceptance behavior, identify the producer-consumer
   contract first: ownership, lifecycle, handshake/no-handshake, exit semantics,
   fallback, compatibility, cancellation, and multi-module aggregation where
   applicable. Record the exact shipping binaries/packages/extensions and
   release branches. Rebuild/repack each variant, isolate effective package
   caches, prove which variant the real shipping consumer loaded, and run a
   representative composed command. Unit mocks or source-compatible compilation
   alone do not replace shipping-layout validation. Exercise applicable normal,
   suppressed, partial, legacy/incompatible, cancellation, nonzero-exit,
   zero-result, fallback, and multi-module paths. Keep rollout/servicing ownership
   explicit; a feature flag does not lower the evidence bar.

   In read-only/advisory mode, trace both revisions, setup/helpers, inputs, and
   assertions without edits or execution unless separately permitted. Label
   conclusions **static predictions**, never executed proof. Existing artifacts
   count only when their revisions/patches, loaded variant, selection, counts,
   and failure attribution match. Green CI or an author's claim alone is not
   red-green evidence. Say what is inaccessible or missing.

4. **Report only the evidence obtained.** Put a compact **Test-value evidence**
   section in the handoff or authorized review/description. State base/parent
   and head SHAs or tested patch identities. Use one row per changed contract,
   group tests protecting the same contract, and identify unassessed groups.

   | Contract / test case | Counterfactual | Without behavior | Correct implementation | Evidence / gap |
   |---|---|---|---|---|
   | Boundary defect / named case | Restore old comparison; patch identity | Executed: assertion failed | Executed: passed | Commands, exits, expected/actual, counts, artifacts |
   | New output / named case | Skip output write | Static: predicts failure | Not executed | Source/assertion locations; paired execution missing |

   Use **executed: detected**, **executed: survived**, **static: predicts
   detection**, **static: predicts survival**, **unverified**, **equivalent**,
   or **not applicable** accurately. Missing red/green or incomplete selection
   leaves paired proof unverified. Report **executed: survived** only for an executed,
   non-equivalent fault; do not count equivalent edits as gaps. Preservation
   passes for a refactor are valid evidence, not missing-red defects.

   Include exact commands/exits, toolchain/framework/configuration, revision
   and counterfactual patch identities, per-run counts, and relevant artifact
   paths. Distinguish execution performed now from verified existing artifacts.
   Never combine static predictions with measured mutation counts/scores.
   Evidence for a final PR head does not prove every intermediate commit;
   assess a requested commit against its parent separately.

   Missing evidence is not automatically a proven defect or an inline review
   thread. Publication follows the caller's existing authorization gate; do
   not create a new GitHub object just to store a report or claim instructions
   enforce hooks, CI, or publication.

## Validation

- The evidence identifies the exact change, independent oracle, test/data case,
  and meaningful buildable counterfactual.
- A claimed bug-fix proof fails for the actual old defect and passes with the
  fix; a feature proof detects absent/wrong behavior rather than absent symbols.
- Executed claims have attributable outcomes, commands/exits, counts, and
  matching artifacts/consumer provenance; unavailable details stay explicit.
- Static predictions, survival, equivalence, preservation, and missing evidence
  remain distinct. Existing tests are credited without demanding redundant ones.
- Scope, read-only restrictions, clean variant outputs, parallel safety, and
  owned-resource cleanup were preserved.