Back to all skills
What it does
An eval is your definition of done, written down and executable. This skill runs one and returns a plain verdict: pass, fail, or regression, with the exact criterion that failed and the reason that criterion exists. Builds 5 (Content) and 6 (Ads) now ship with a definition of done already written. Each quality grader that runs during those builds saves its verdict as a receipt file, and the build cannot be marked complete until those receipts and the required output files clear every check. Completion is earned rather than asserted. The checks are deterministic: file checks, JSON field checks, and house-rule checks such as no em-dashes and no leaked placeholders. Zero model calls, so effectively no added token cost and no flakiness.
Say this to start
This skill has no button. You start it by saying what you want. Any of these will do it:
> run the evals on this build
> is this ready to ship
> did my change break anything
> regression check
> baseline this
> turn this golden test into an eval
When to reach for it
- at the end of a build, as the thing that decides whether it is actually complete
- before handing a deliverable to a client
- after editing a prompt, a saved config, or a pipeline you run for a client
- weekly, as a drift check across the configs you have saved
- when a build says complete and you want to see the evidence rather than take its word
- when you have a golden test written in prose that nobody ever re-runs
When NOT to use it
| If you actually want | Use this instead |
|---|
| a one-off qualitative read on whether a deliverable is any good | critique |
| the copy fixed once a check has flagged it | legendary-copywriters |
| the content rebuilt after a failed content build | content-system-architect |
| the ad campaign rebuilt after a failed ads build | meta-lead-system |
| testing the graders themselves rather than the work they graded | the claim, hook and niche eval harnesses that ship with the plugin |
| tidying brand memory that has drifted | dream |
Before you start
| What you need | Why | |
|---|
| A finished build, or the file you want checked | the runner grades a real artifact on disk, so there has to be something to read before it can say anything | Required |
| A config that says what done means for that artifact | Builds 5 and 6 write theirs for you at completion. Everything else needs one, and a config with no checks in it is refused rather than passed | Required |
| The grader receipts from the build, saved in its gates folder | the Build 5 and 6 checks read those receipt files, so a build whose graders never ran has nothing to audit and fails | Required |
| A saved baseline from an earlier green run | without one the run tells you pass or fail today, but cannot tell you that today is quietly worse than last month | Optional |
| An existing golden test or gate rule written in prose | only needed if you want it converted into a config that runs on every change instead of sitting in a document | Optional |
How it runs
- Find the definition of doneIt locates the config for what you are checking. For a content or ads build that file was copied into the build folder when the build finished, so there is nothing for you to write or point at. For anything else you say which config to run.
- Refuse a broken config before trusting its resultA config with a scenario that contains no checks is refused outright. A score of 100 from an eval that tested nothing is worse than no eval, because it manufactures confidence you then act on.
- Read the artifact and the receiptsIt reads the build output and the receipt files the graders saved while the build ran. A receipt records what the claim, hook, and copy graders found at the time, so a verdict that used to vanish when the session ended is now a file any later run can re-check.
- Run the deterministic checksEvery check is mechanical: does this file exist, is this receipt valid JSON, does its fail count read zero, does this output contain an em-dash or a leaked placeholder. No model is involved, so the checks cost nothing to run and return the same answer every time.
- Apply the tripwiresSome checks are marked required. House rules such as no em-dashes and no leaked placeholders are required by default. A failed required check fails the whole run on its own, so a house-rule breach can never be outvoted by a pile of checks that passed.
- Compare against the baselineIf a baseline was saved from an earlier green run, this run is compared against it. A lower score, or any single check flipping from pass to fail, is reported as a regression and fails the run even when the total still clears the bar. Meeting the bar while sliding backwards is how quality erodes without anyone noticing.
- Return one verdict and the reasonYou get pass, fail, or regression, and for anything that failed, the check that failed and the sentence explaining why that check exists. That sentence is written to be the thing you read when it breaks, so the output tells you what to fix rather than showing you a rule number.
- Block the completion, or let it throughOn a Build 5 or Build 6 run, a failure stops the build being written down as complete. You fix the flagged item and run it again. Nothing is rewritten for you: the runner grades, and fixing is the job of whichever skill produced the artifact.
What you get
- A verdict of pass, fail, or regression on the build or file you checked
- The exact check that failed and the plain-language reason that check exists, so the failure message is the instruction
- A receipt audit for Builds 5 and 6: proof each grader actually ran, that its recorded fail count is zero, and that its verdict is on disk rather than lost with the session
- A required-artifacts check confirming every file that build was supposed to produce is really there
- House-rule enforcement on the finished output, including no em-dashes and no leaked placeholders
- A baseline file, when you ask for one, that later runs are measured against
- A regression report naming what got worse when a later run slips below that baseline
- An executable config converted from a golden test or a gate rule you had written in prose
Honest limits
Read this before you rely on it
- It grades. It never fixes. A failing run hands you the flagged item and stops, and the repair goes back to the skill that produced the work.
- An eval only tests what its config says to test. A build can pass every check and still be the wrong content for your audience, which is a judgement call and belongs to critique.
- The deterministic checks are mechanical by design. They can confirm a grader receipt says zero failures. They cannot re-argue whether that grader was right.
- A missing or unreadable receipt fails the run. That is deliberate: a build whose graders never ran, or whose verdict was never saved, is not a build you can call complete.
- A baseline can be moved. After a real improvement, moving it is correct. Moving it to make a regression disappear defeats the whole mechanism, and nothing in the tool can stop you doing that.
- You do not write configs, and you are not meant to. The builds that need one ship with it. If you want a config for something else, ask for it in plain English and describe what done means to you.
Where people go wrong
| The mistake | Do this instead |
|---|
| Treating a refusal as the tool being broken | Read the failure. A refusal is the product working: it is the build telling you which specific item is not finished, before a client finds it instead. |
| Trying to learn the config format before running anything | Just run it. The builds that need a config already carry theirs. Your job is to read the failure message and fix the one thing it names. |
| Marking a build complete because the files exist | Let the run decide. A file existing is not the same as a file clearing the bar, and the whole point of the receipts is that the second question now has an answer. |
| Re-saving the baseline to clear a regression | Fix the output. Only move the baseline after a change you meant to make, and be clear with yourself that you are moving the bar. |
| Expecting it to replace a critique | Run both. An eval is repeatable and mechanical, a critique is a single judgement pass by something that can reason. They catch different failures. |
Worth knowingThe reason this matters more than it sounds: before receipts, a grader ran during a build, said everything was clean, and that verdict disappeared the moment the session ended. There was no way to check later whether it had ever really run. Now the verdict is a file, and the build checks those files before it is allowed to call itself finished. That is the difference between a build that claims it is done and a build that can show you.
