I built with Claude for a long time. For most of that time, my workflow looked like this:
Write a prompt. Run it once. It works. Feel great about yourself. Run it again two days later on a slightly different input. It breaks. Spend 45 minutes tweaking. Repeat forever.
Thatâs not engineering. Thatâs hope with extra steps.
Anthropic just shipped Skills 2.0, and it changes how you build. This is not a ânew feature droppedâ moment. It is a âwait, this actually works differentlyâ moment.
đ± Click Follow if you want to stay sharp on Product, AI, Technology, Marketing and content for builders. I publish consistently and keep it practical.
What Changed (The Short Version)
Skills are the core building block on Claude. A skill is a modular, reusable instruction set. It tells the AI how to behave for a specific task. Think of them as the âplaybooksâ for your automations.
Skills 2.0 adds the missing piece: built-in evals.
You can now test your skills in a systematic and automatic way. The Skill Creator holds the grades, the benchmarks, and the side-by-side comparisons.
You need no external testing framework. You drop the guesswork of ârun it and eyeball it.â You get structured, repeatable proof that your skill works.
The Architecture They Built
The Skill Creator now ships with a dedicated folder structure:
Eval Viewer: where test results get visualized
Agents: AI sub-agents that analyze, compare, and grade outputs
Scripts: for benchmarking and generating structured reports
Here is what that means in practice. You no longer build in a black box. Every skill run produces a full transparency report: tool calls, execution steps, formal Pass/Fail grades. You can see exactly what Claude did, why it did it, and where it fell short.
The new architecture: A dedicated environment for testing and grading skills.
The Part That Actually Impressed Me
Nobody likes to admit one thing about prompts. They rarely work perfectly on the first try. Good ones take 5-10 iterations minimum. Thatâs not a skill issue. Thatâs just the nature of language models.
Skills 2.0 automates that iteration loop.
You run multiple test variations in parallel with sub-agents. Each variation tests for:
Speed: how fast is the skill executing?
Token usage: is this costing more than it should?
Output quality: does it actually match your voice/style/format?
Tool call accuracy: if the skill uses MCP integrations, are they firing correctly?
That last one is underrated. Workflows get more complex with more tools, more integrations and more external context. The âplumbingâ then matters as much as the prose.
The Skill MD file acts as the âsource codeâ for your automation.
Running multiple variations simultaneously to test reliability.
Full Transparency on Every Run
One thing gets little attention. Claude generates a structured report after every test run.
Itâs not just a Pass/Fail badge. Itâs a full breakdown: every tool call made, every step taken, every decision in sequence. If something breaks, you can see exactly where and why. Pasted image 20260310233921.png
đž [SCREENSHOT 4, 00:03:40] The âStructured Reportâ document showing tool call steps and execution sequence. Caption: Full Transparency: Claude breaks down every step taken during a test run.
Self-Updating Skills (Yes, Really)
One feature here is Progressive Updates, and it matters more than it sounds.
You give feedback such as âstop using em-dashesâ or âkeep this under 300 words.â The Skill MD file then updates its own rules automatically.
The skill learns from correction. You rewrite nothing by hand.
You spent time on prompt files that drift, and you explained the same preferences again and again. This is the fix. The skill compounds. Your preferences accumulate. It gets sharper over time, and you do no extra work.
Skills that update their own rules based on your feedback.
Precision Benchmarking
Skills 2.0 also lets you define exact criteria for âgoodâ. You set those criteria as measurable benchmarks.
Style match, word count and tone adherence all become benchmarks. They stopped being editorial preferences. They are codified Pass/Fail conditions, and the system tests them on every single run. Pasted image 20260310234344.png
đž [SCREENSHOT 6, 00:10:45] The Fail/Pass criteria table showing âStyle Matchâ and âWord Countâ as graded benchmarks. Caption: Precision Optimization: Setting specific benchmarks for tone, length, and style.
AB Testing: The Use Case I Didnât Expect
This one surprised me.
Skills 2.0 lets you run Version A vs. Version B comparisons. The obvious use is a model test. Compare a new model, such as Opus 4.7, against your current one before you switch.
The smarter use is context engineering. Test whether a specific reference file helps your output or only bloats the prompt.
The demo gave a real example. Version A ran at 93,000 tokens. Version B used a leaner prompt, held the same output quality, and ran at 77,000 tokens. That is a 17% token reduction with zero quality loss. At scale, that is a meaningful cost difference rather than a rounding error.
Where This Actually Gets Used
This isnât theoretical. Here are five places where Skills 2.0 changes the calculus immediately:
1. Content teams running on templates You built a skill for newsletters, social posts, or briefs. Evals prove that the skill holds up across 50 different inputs rather than the three you tested by hand. One version drift caught early saves hours of editorial cleanup later.
2. Sales teams with automated outreach Personalization at scale breaks quietly. A skill that writes great cold emails on Monday might go generic by Friday after a few edge-case inputs. Skills 2.0 catches that regression before it hits your open rates.
3. Agencies managing client voice Every client has a different tone, different rules, different no-go words. Progressive Updates puts a clientâs word bans into the skill permanently. Those banned phrases stay in the skill rather than in a Slack thread.
4. Operators building internal AI tools Your team uses Claude for data summaries, ops reporting, or document processing. AB testing then optimizes the cost of every run. A 17% token reduction looks small until you multiply it by 10,000 monthly runs. Then itâs a budget line.
5. Anyone switching models Anthropic ships a new model often. You now have a structured way to test it against your existing setup. You skip the âlet me just try it and seeâ step. You get real benchmark data that compares the old and the new side by side.
Why This Sets a New Standard
Most AI tooling still operates on faith. You build something, it works in your test cases, you ship it, and you quietly hope it holds up.
Anthropic pushes toward something different: evals as a first-class feature. It is no developer add-on and no third-party integration. It sits inside the tool you already build with.
Businesses use Claude for real workflows: sales automation, content pipelines, ops tooling. Those workflows cannot run on vibes. They need proof. They need repeatability. They need a skill that works on input #1 and still works on input #500.
Skills 2.0 is what makes that possible.
The Bottom Line
The shift here runs from âAI as a chatbotâ to âAI as a platform for reliable, self-improving micro-agents.â
Thatâs not a subtle distinction. One is a tool you use. The other is infrastructure you build on.
Anthropic put evals directly into the UI with no code and no external frameworks. So AI engineers are no longer the only users. It serves anyone who builds anything serious with Claude.
The bar just moved. Pay attention to it.
Read the Claude Blog here: https://claude.com/blog/improving-skill-creator-test-measure-and-refine-agent-skills
More on Skills, context engineering, and building with Claude comes soon. Follow along if this is the kind of thing you want.
Related reading: Building An AI Marketing Team with Claude Skills, Claude Code Memory Architecture, GitHub Copilot Complete Guide
