Most Claude skills run on vibes, with no Claude skill evals behind them. You write a prompt, run it once, and it works. Two days later a new input breaks it, and you spend 45 minutes tweaking.

Each round proves nothing about the next input. Anthropic now ships evals inside its Skill Creator, and one A/B test cut a skill's token use by 17% with no quality loss.

TLDR

A skill is a folder with three parts: SKILL.md for the prompt, scripts/ for code, references/ for examples.
A good skill asks for every input before it acts.
Three Pass/Fail checks over five real inputs show the failure before anyone else sees it.
A/B tests show which reference files earn their tokens. One test cut 17% with no quality loss.
A scorer in a hook blocks bad output, and a script prints the pass so the model never types it.

The same skill on vibes loses 45 minutes per break, and with a scorer gate it stopped 21 bad comments.

Take one skill that drafts my LinkedIn comments.

On vibes: I read each output by eye. A new input breaks it, and the fix proves nothing about the next one.
With evals: three written checks grade every output. A scorer runs on every save.
The result: in one session the scorer stopped 21 comments. None of them posted.

Six terms carry the rest of this post. Each one hangs off the skill.

Six terms hang off the skill: Claude Code, CLAUDE.md and MCP run it, and evals, scorers and hooks prove it.

Step 1. Build the Skill as a Folder With Three Parts

Every skill I run has the same three parts. Only SKILL.md is required.

A skill folder holds SKILL.md, a scripts folder and a references folder, and only SKILL.md is required.

SKILL.md holds the prompt. Its description line tells Claude when to trigger. Its body sets the steps and the output format.
scripts/ holds code the skill runs. Code gives the same answer on every run, and a prompt does not.
references/ holds examples. Two real outputs you liked teach Claude your register faster than a paragraph of adjectives.

To start one, pick a task where you paste the same instructions every week. Create a folder named after it in ~/.claude/skills/. Write the name, the trigger description, the steps and the output format into SKILL.md.

Tip. Keep the references short. Every reference file costs tokens on every run, so two good examples beat ten average ones.

Step 2. Make the Skill Ask for Its Inputs Before It Acts

A weak skill starts writing the moment it loads. A good skill asks first, and it cannot reach step 2 until step 1 has an answer.

Take a skill that sends a Slack DM to a list of people.

It asks for the message.
It asks for the channel.
It asks for a confirmation.
Only then does it send.

A lazy prompt such as "blast this to the team" still produces a safe result, because the skill fills the gaps before Claude acts. This step turns a 20 prompt job into a 2 prompt job.

My Substack Writing Harness, a skill that drafts and scores my essays, asks seven questions before the first draft. It asks for the hot take, the proof only I have, the opening story, the reader and the goal. When an answer comes back empty, the skill asks again rather than inventing an opinion for me.

Substack Writing Harness triages the idea, drafts the essay and scores it before you publish.

Step 3. Write Three Pass/Fail Checks and Grade Five Real Runs

The Skill Creator is the Anthropic skill that builds and tests other skills. It ships three folders for testing.

Eval Viewer: shows the test results.
Agents: subagents that analyze, compare and grade outputs.
Scripts: benchmarks and structured reports.

A subagent is a second Claude instance that starts with an empty context and does one job. Each test run produces a report with every tool call, every step and a Pass/Fail grade.

The Skill Creator ships agents, assets, an eval viewer and scripts, with grader.md open on the right.

Simon Willison argues that skills may matter more than MCP.

Simon Willison’s Newsletter

Claude Skills are awesome, maybe a bigger deal than MCP

In this newsletter…

a year ago · 355 likes · 13 comments · Simon Willison

Good prompts take 5 to 10 rounds of changes. That count has nothing to do with a skill issue, because language models behave that way. The Skill Creator runs those rounds in parallel with subagents, and each variation reports four numbers.

Speed: how long the skill takes to run.
Token usage: whether it costs more than it should.
Output quality: whether it matches your voice, style and format.
Tool call accuracy: whether its MCP tools fire correctly.

The Skill Creator automates the testing and iterating half of the loop that improves a skill.

Your own version of this takes four moves.

Write three Pass/Fail checks for a good output, such as "under 300 words".
Put the checks at the bottom of SKILL.md, so Claude reads them on every run.
Run the skill on five different inputs from last month.
Grade each output in a fresh chat, or ask the Skill Creator to run evals.

Where a check fails, fix SKILL.md and rerun all five inputs. Style becomes a test too. Style match, word count and tone each become a Pass/Fail condition the grader checks.

Claude grades five draft variations on style match, em dashes, word count and personal stories, and two score 4/4.

Alex Lieberman recorded a 38-minute evals lesson with the person who leads Labs at LangChain.

Step 4. A/B Test Which Reference Files Earn Their Tokens

The A/B test compares models, and its sharper use is context engineering. Context engineering means you decide what goes into the prompt. The test shows whether a reference file improves the output or only makes the prompt bigger.

Version A: 93,000 tokens.
Version B: a leaner prompt at 77,000 tokens, with the same output quality.
The result: a 17% token cut with no quality loss.

Anthropic splits skills into two kinds, and each kind asks its eval a different question. A capability uplift skill helps Claude do something the base model cannot do reliably. An encoded preference skill puts steps Claude can already do into your order.

A capability skill's eval asks if the model caught up, and a preference skill's eval asks if output still matches.

Evals catch the silent failures in both kinds. Anthropic's own PDF skill used to fail on forms with no fillable fields, because Claude could only guess where each answer belonged.

The fix: evals isolated the failure, and new text now anchors to coordinates pulled from the document.
The tuning: Anthropic ran its description tuning tool across its public document skills.
The result: triggering accuracy improved on five of the six. (Source: Anthropic, "Improving skill-creator: Test, measure, and refine Agent Skills," March 3, 2026.)

Claude rewrites a skill description on 80% of queries and picks the best one by the hidden 20% test set.

Two Anthropic engineers who built Skills explain the design here.

Step 5. Make a Scorer Refuse the Bad Output

A grade you read is a reminder. A grade that stops the save is a gate. Move every check you can from the first kind to the second.

The self-evaluation loop runs in six steps, and the cap stops it from running forever.

Define good and bad output as written checks.
Generate the output.
Run the scorer first, because a script is cheap and repeatable.
Send the output to a fresh subagent that grades it against the checks.
On a fail, regenerate with the failing checks attached.
Stop at 3 to 5 loops and hand the output to a person, with the checks it still fails.

Watch out. The grader in step 4 must be a new subagent with no memory of the draft. An agent rarely rejects an answer it wrote itself.

Anatoli Kopadze shares an Anthropic talk on agents that check each other's work and catch their own mistakes.

Code cannot score tone. For a check like that, run two fresh graders and pass the output only when both agree. Keep the loop cap, because two graders that disagree five times are telling you the check is vague.

The scorer is the part I sell as its own skill. The Deslop Audit scores a draft for machine tells, then reviews it or rewrites it clean. In one session my scorer refused 21 LinkedIn comments, and no prompt reminder would have stopped those 21.

Step 6. Put the Scorer in a Hook the Agent Cannot Skip

A rule in CLAUDE.md or SKILL.md is text, and Claude weighs it against everything else in its context. Late in a long session, the task in front of Claude gets more attention than a rule it read at the start.

Capitals fail. A rule in capitals with "NEVER" in front works for a few days. Then every rule arrives in capitals and they compete for the same attention.
A hook holds. It runs outside the model, so Claude cannot talk its way past it.
The setup: fire the hook after every file write, run the scorer, and return a block with the exact findings on a fail.
The second hook: on session end, refuse to stop while any file Claude touched still fails.

The checker is code, and code breaks. A dependency updates, Python goes missing on a new machine, or a bad file crashes the scorer. A gate that blocks on its own crash gets commented out on a deadline day, and a silent hook looks the same as a passing one.

A gate blocks a real failure, and lets work through with a warning when the check cannot run.

Rule of thumb. Block when the check ran and failed. Pass with a printed "check skipped" when the check could not run, and fix the scorer that day.

Step 7. Let a Script Print the Pass

At the end of a task, Claude writes "all checks pass" in its summary. That line costs nothing to type, and you cannot tell a true one from a false one.

The false pass: the check never ran, Claude misread the output, or it checked the wrong file.
The pasted pass: asking Claude to paste the output fails too, because it can paste an earlier run.
The fix: a script runs every check and prints the results as a block at the top of the report.

The block comes from the check itself, so it cannot claim a run that never happened. When the block is missing, the script did not run, and you know to look.

Step 8. Keep Every Skill in One Place and in Git

Claude Code reads skills from two folders.

Global skills sit in ~/.claude/skills and load in every project.
Project skills sit in .claude/skills inside one project and load only there.

Keep each skill in exactly one of them. Two copies drift apart, and you end up editing the copy Claude never loads.

Commit every skill folder, your CLAUDE.md files and your hooks to Git. An eval that passed last month and fails today shows up as a diff, and one revert fixes it.

Never commit the files that hold live API keys. Commit a template with the key names and no values. On a new machine, clone the repo and run a short restore script.

The full backup and restore guide lives in Claude Code Memory Architecture. For skills built around one job, see Building An AI Marketing Team with Claude Skills.

FAQ

Do I need the Skill Creator to run evals? No. Three written checks, five inputs from last month and a fresh chat to grade them cover the basics. The Skill Creator runs the same loop in parallel and adds the reports.

How many checks should a skill have? Start with three. A check you cannot grade as Pass or Fail is too vague, so rewrite it until a stranger could grade it.

Why must the grader be a fresh subagent? The agent that wrote the draft grades it too kindly. A fresh context sees only the output and the checks.

What is the difference between a scorer and an eval? An eval runs the skill on a set input and grades the output. A scorer is a script that gives one output a number, so a hook can block on it.

Where should the gate live? In a hook that fires after every file write. A rule in a prompt is a request, and a hook runs whether Claude agrees or not.

Appendix

The folder layout from Step 1:

~/.claude/skills/weekly-report/
├── SKILL.md          # trigger line, steps, output format, three Pass/Fail checks
├── scripts/
│   └── score.py      # exits 0 on pass, 1 on a real failure
└── references/
    ├── good-1.md
    └── good-2.md

The hook from Step 6, registered in .claude/settings.json:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Write|Edit",
        "hooks": [{ "type": "command", "command": "python .claude/hooks/gate.py" }]
      }
    ]
  }
}

The gate script blocks on a real failure and lets the work through when the scorer breaks:

#!/usr/bin/env python3
import json, subprocess, sys

event = json.load(sys.stdin)
path = event.get("tool_input", {}).get("file_path", "")
if not path.endswith(".md"):
    sys.exit(0)

try:
    run = subprocess.run(["python", "scripts/score.py", path],
                         capture_output=True, text=True, timeout=60)
except Exception as err:
    print(f"WARNING: check skipped, scorer did not run: {err}", file=sys.stderr)
    sys.exit(0)

if run.returncode == 1:
    print(json.dumps({"decision": "block", "reason": run.stdout}))
elif run.returncode != 0:
    print(f"WARNING: check skipped, scorer crashed: {run.stderr[-300:]}", file=sys.stderr)
sys.exit(0)

Claude Skill Evals Cut Tokens 17%. Here Is…