Most Claude skills run on vibes, with no Claude skill evals behind them. You write a prompt, run it once, and it works. Two days later a new input breaks it, and you spend 45 minutes tweaking.
Each round proves nothing about the next input. Anthropic now ships evals inside its Skill Creator, and one A/B test cut a skill's token use by 17% with no quality loss.
TLDR
A skill is a folder with three parts:
SKILL.mdfor the prompt,scripts/for code,references/for examples.A good skill asks for every input before it acts.
Three Pass/Fail checks over five real inputs show the failure before anyone else sees it.
A/B tests show which reference files earn their tokens. One test cut 17% with no quality loss.
A scorer in a hook blocks bad output, and a script prints the pass so the model never types it.
Take one skill that drafts my LinkedIn comments.
On vibes: I read each output by eye. A new input breaks it, and the fix proves nothing about the next one.
With evals: three written checks grade every output. A scorer runs on every save.
The result: in one session the scorer stopped 21 comments. None of them posted.
Six terms carry the rest of this post. Each one hangs off the skill.
Step 1. Build the Skill as a Folder With Three Parts
Every skill I run has the same three parts. Only SKILL.md is required.
SKILL.md holds the prompt. Its description line tells Claude when to trigger. Its body sets the steps and the output format.
scripts/ holds code the skill runs. Code gives the same answer on every run, and a prompt does not.
references/ holds examples. Two real outputs you liked teach Claude your register faster than a paragraph of adjectives.
To start one, pick a task where you paste the same instructions every week. Create a folder named after it in ~/.claude/skills/. Write the name, the trigger description, the steps and the output format into SKILL.md.
Tip. Keep the references short. Every reference file costs tokens on every run, so two good examples beat ten average ones.
Step 2. Make the Skill Ask for Its Inputs Before It Acts
A weak skill starts writing the moment it loads. A good skill asks first, and it cannot reach step 2 until step 1 has an answer.
Take a skill that sends a Slack DM to a list of people.
It asks for the message.
It asks for the channel.
It asks for a confirmation.
Only then does it send.
A lazy prompt such as "blast this to the team" still produces a safe result, because the skill fills the gaps before Claude acts. This step turns a 20 prompt job into a 2 prompt job.
My Substack Writing Harness, a skill that drafts and scores my essays, asks seven questions before the first draft. It asks for the hot take, the proof only I have, the opening story, the reader and the goal. When an answer comes back empty, the skill asks again rather than inventing an opinion for me.
Step 3. Write Three Pass/Fail Checks and Grade Five Real Runs
The Skill Creator is the Anthropic skill that builds and tests other skills. It ships three folders for testing.
Eval Viewer: shows the test results.
Agents: subagents that analyze, compare and grade outputs.
Scripts: benchmarks and structured reports.
A subagent is a second Claude instance that starts with an empty context and does one job. Each test run produces a report with every tool call, every step and a Pass/Fail grade.
Simon Willison argues that skills may matter more than MCP.
Good prompts take 5 to 10 rounds of changes. That count has nothing to do with a skill issue, because language models behave that way. The Skill Creator runs those rounds in parallel with subagents, and each variation reports four numbers.
Speed: how long the skill takes to run.
Token usage: whether it costs more than it should.
Output quality: whether it matches your voice, style and format.
Tool call accuracy: whether its MCP tools fire correctly.
Your own version of this takes four moves.
Write three Pass/Fail checks for a good output, such as "under 300 words".
Put the checks at the bottom of
SKILL.md, so Claude reads them on every run.Run the skill on five different inputs from last month.
Grade each output in a fresh chat, or ask the Skill Creator to run evals.
Where a check fails, fix SKILL.md and rerun all five inputs. Style becomes a test too. Style match, word count and tone each become a Pass/Fail condition the grader checks.
Alex Lieberman recorded a 38-minute evals lesson with the person who leads Labs at LangChain.
Step 4. A/B Test Which Reference Files Earn Their Tokens
The A/B test compares models, and its sharper use is context engineering. Context engineering means you decide what goes into the prompt. The test shows whether a reference file improves the output or only makes the prompt bigger.
Version A: 93,000 tokens.
Version B: a leaner prompt at 77,000 tokens, with the same output quality.
The result: a 17% token cut with no quality loss.
Anthropic splits skills into two kinds, and each kind asks its eval a different question. A capability uplift skill helps Claude do something the base model cannot do reliably. An encoded preference skill puts steps Claude can already do into your order.
Evals catch the silent failures in both kinds. Anthropic's own PDF skill used to fail on forms with no fillable fields, because Claude could only guess where each answer belonged.
The fix: evals isolated the failure, and new text now anchors to coordinates pulled from the document.
The tuning: Anthropic ran its description tuning tool across its public document skills.
The result: triggering accuracy improved on five of the six. (Source: Anthropic, "Improving skill-creator: Test, measure, and refine Agent Skills," March 3, 2026.)
Two Anthropic engineers who built Skills explain the design here.
Step 5. Make a Scorer Refuse the Bad Output
A grade you read is a reminder. A grade that stops the save is a gate. Move every check you can from the first kind to the second.
The self-evaluation loop runs in six steps, and the cap stops it from running forever.
Define good and bad output as written checks.
Generate the output.
Run the scorer first, because a script is cheap and repeatable.
Send the output to a fresh subagent that grades it against the checks.
On a fail, regenerate with the failing checks attached.
Stop at 3 to 5 loops and hand the output to a person, with the checks it still fails.
Watch out. The grader in step 4 must be a new subagent with no memory of the draft. An agent rarely rejects an answer it wrote itself.
Anatoli Kopadze shares an Anthropic talk on agents that check each other's work and catch their own mistakes.
Code cannot score tone. For a check like that, run two fresh graders and pass the output only when both agree. Keep the loop cap, because two graders that disagree five times are telling you the check is vague.
The scorer is the part I sell as its own skill. The Deslop Audit scores a draft for machine tells, then reviews it or rewrites it clean. In one session my scorer refused 21 LinkedIn comments, and no prompt reminder would have stopped those 21.
Step 6. Put the Scorer in a Hook the Agent Cannot Skip
A rule in CLAUDE.md or SKILL.md is text, and Claude weighs it against everything else in its context. Late in a long session, the task in front of Claude gets more attention than a rule it read at the start.
Capitals fail. A rule in capitals with "NEVER" in front works for a few days. Then every rule arrives in capitals and they compete for the same attention.
A hook holds. It runs outside the model, so Claude cannot talk its way past it.
The setup: fire the hook after every file write, run the scorer, and return a block with the exact findings on a fail.
The second hook: on session end, refuse to stop while any file Claude touched still fails.
The checker is code, and code breaks. A dependency updates, Python goes missing on a new machine, or a bad file crashes the scorer. A gate that blocks on its own crash gets commented out on a deadline day, and a silent hook looks the same as a passing one.
Rule of thumb. Block when the check ran and failed. Pass with a printed "check skipped" when the check could not run, and fix the scorer that day.
Step 7. Let a Script Print the Pass
At the end of a task, Claude writes "all checks pass" in its summary. That line costs nothing to type, and you cannot tell a true one from a false one.
The false pass: the check never ran, Claude misread the output, or it checked the wrong file.
The pasted pass: asking Claude to paste the output fails too, because it can paste an earlier run.
The fix: a script runs every check and prints the results as a block at the top of the report.
The block comes from the check itself, so it cannot claim a run that never happened. When the block is missing, the script did not run, and you know to look.
Step 8. Keep Every Skill in One Place and in Git
Claude Code reads skills from two folders.
Global skills sit in
~/.claude/skillsand load in every project.Project skills sit in
.claude/skillsinside one project and load only there.
Keep each skill in exactly one of them. Two copies drift apart, and you end up editing the copy Claude never loads.
Commit every skill folder, your CLAUDE.md files and your hooks to Git. An eval that passed last month and fails today shows up as a diff, and one revert fixes it.
Never commit the files that hold live API keys. Commit a template with the key names and no values. On a new machine, clone the repo and run a short restore script.
The full backup and restore guide lives in Claude Code Memory Architecture. For skills built around one job, see Building An AI Marketing Team with Claude Skills.
FAQ
Do I need the Skill Creator to run evals? No. Three written checks, five inputs from last month and a fresh chat to grade them cover the basics. The Skill Creator runs the same loop in parallel and adds the reports.
How many checks should a skill have? Start with three. A check you cannot grade as Pass or Fail is too vague, so rewrite it until a stranger could grade it.
Why must the grader be a fresh subagent? The agent that wrote the draft grades it too kindly. A fresh context sees only the output and the checks.
What is the difference between a scorer and an eval? An eval runs the skill on a set input and grades the output. A scorer is a script that gives one output a number, so a hook can block on it.
Where should the gate live? In a hook that fires after every file write. A rule in a prompt is a request, and a hook runs whether Claude agrees or not.
Appendix
The folder layout from Step 1:
~/.claude/skills/weekly-report/
├── SKILL.md # trigger line, steps, output format, three Pass/Fail checks
├── scripts/
│ └── score.py # exits 0 on pass, 1 on a real failure
└── references/
├── good-1.md
└── good-2.mdThe hook from Step 6, registered in .claude/settings.json:
{
"hooks": {
"PostToolUse": [
{
"matcher": "Write|Edit",
"hooks": [{ "type": "command", "command": "python .claude/hooks/gate.py" }]
}
]
}
}The gate script blocks on a real failure and lets the work through when the scorer breaks:
#!/usr/bin/env python3
import json, subprocess, sys
event = json.load(sys.stdin)
path = event.get("tool_input", {}).get("file_path", "")
if not path.endswith(".md"):
sys.exit(0)
try:
run = subprocess.run(["python", "scripts/score.py", path],
capture_output=True, text=True, timeout=60)
except Exception as err:
print(f"WARNING: check skipped, scorer did not run: {err}", file=sys.stderr)
sys.exit(0)
if run.returncode == 1:
print(json.dumps({"decision": "block", "reason": run.stdout}))
elif run.returncode != 0:
print(f"WARNING: check skipped, scorer crashed: {run.stderr[-300:]}", file=sys.stderr)
sys.exit(0)