Testing Agent Skills Properly: Building a Skill That Evaluates Other Skills

“Convinced that linting and testing your skills is essential, and confused as to how there seems to be no standard approach for it.” — a practitioner thread that started this project

Why this exists

A skill is a prompt that ships with files. It looks like documentation, but it behaves like a dependency: it can regress when the model changes, when an API moves, or when someone edits the description and the skill stops triggering. You wouldn’t ship a backend service without a performance test, yet most skills ship on the strength of “it seemed to work once”.

We read the four main sources on the subject and found they agree on far more than the lack of a standard suggests. So we encoded the shared practice into a skill that generates evals for other skills: skill-eval-generator. This post covers what the sources say, what we borrowed, and how the skill works.

The four sources and their core ideas

1. Google: skills are products, so govern them like products

Google’s write-up on building, testing and scaling Google Agent Skills (source) starts from a scaling problem: once many teams contribute skills, a few vague, broken or incomplete ones degrade the experience for everyone. Their answer is layered automation.

Check-in gates: linters for frontmatter, line counts, layout and naming; link checkers to catch 404s and hallucinated URLs; AI-assisted checklists for structural patterns and guardrails.
Continuous evals: on submit (authors must supply prompt suites and scoring rubrics) and weekly across the whole library, because models, harnesses and APIs drift.
With vs without: each skill is compared against no skill on accuracy (quality, completion) and efficiency (tokens, time), repeated across agent frameworks for statistical significance. The resulting 2x2 shows whether the skill gives measurable uplift.
Ownership: maintainers own CI; skill owners own long-term quality. A degraded eval is the owner’s bug.

What we took: lint on every change, scheduled regression runs, and the accuracy-plus-efficiency framing.

2. arXiv: generate the tasks from the skill itself

“A Framework for Evaluating Agentic Skills at Scale” (arXiv 2606.17819) notes that no reusable methodology existed for evaluating an individual skill, and supplies one. The pipeline is: analyze the skill and user intent, engineer the environment, generate tasks, validate them, run, grade.

Several ideas matter here:

Environment engineering first. A skill may need CLIs, MCP servers, network, credentials, a repo, git state, a database, a browser, seeded external state or multi-turn input. If you can’t provide it, the task isn’t executable.
Hidden rubrics, two kinds. Task completion (did it get done) and instruction following (did it do it the way the skill says). Each is scored out of 100.
Validation against leakage. A QA step removes ambiguous tasks and checks that the prompt doesn’t spell out the solution or reveal the rubric.
Skill delta. Score with the skill minus score without it, plus runtime, tokens and cost.
The headline finding: skills add roughly 5 to 22 points, mostly through instruction following. Goal completion saturates near the top for frontier models. In the authors’ words, the 2026 question is not “can the agent solve this task?” but “does the agent solve it the way I want it to?” A small model with a skill often matches a bigger one without.
Diagnosis and deletion. Look at the rubric items where with and without differ most: that’s what the skill actually does. A negligible delta means the model already knows it, and the skill can go.
Honest limits: skill selection isn’t tested (the solver is told the skill is relevant), a single LLM judge is biased on style, and hard environments were excluded.

What we took: the two assertion groups, the leakage check, the environment checklist, and delta-based triage.

3. Anthropic: build the evaluations first

The best-practices guide’s “Evaluation and iteration” section (source) argues for evaluation-driven development: run the model without the skill, write down concrete failures, build at least three scenarios for those gaps, baseline, write the minimal instructions that close them, and iterate. It admits there’s no built-in runner and calls evals “the source of truth”.

Other parts of the same guide feed directly into linting and review:

Name: at most 64 characters, lowercase letters, digits and hyphens, no reserved words. Description: non-empty, at most 1,024 characters, third person, saying what it does and when to use it.
SKILL.md body under 500 lines, references one level deep, no time-sensitive content.
Claude A / Claude B. One instance authors the skill, a fresh instance uses it on real work, and you bring observations back to the author.
Watch navigation. Files read in a surprising order, links never followed, or a section read repeatedly tell you where the structure is wrong.
Test with every model you plan to use. Haiku asks whether there’s enough guidance, Opus whether you over-explained.

What we took: eval-first, the lint rules, multi-model runs, and reading transcripts.

4. Agent Skills docs: the practical loop

“Evaluating skill output quality” (source) is the closest thing to a standard format today. Its pieces:

Test cases live in evals/evals.json: a realistic prompt, an expected output and optional input files. Start with two or three, vary phrasing, include an edge case.
Run every case twice, with and without the skill (or against a snapshot of the old version), in a clean context each time.
Record timing.json immediately. Harnesses don’t persist it.
Write assertions after you see the first outputs. Good ones are objective and observable; “the output is good” and exact-phrase matches are bad. Style goes to human review.
Grade PASS/FAIL with quoted evidence, use scripts for mechanical checks, and review the assertions themselves. Optionally run a blind A/B judge.
Aggregate to benchmark.json with mean, standard deviation and delta.
Triage: drop assertions that pass in both configs, fix those that fail in both, study those that pass only with the skill, tighten instructions where variance is high.
Improve with an LLM that sees failed assertions, human feedback, transcripts and the current SKILL.md. Generalize, keep it lean, explain the why, bundle helper scripts that every run rewrites.

What we took: the file formats and the whole loop. Our format is a superset, so existing tools can still read it.

Where the sources agree

Four ideas appear everywhere: compare against a no-skill baseline, use realistic and varied prompts with edge cases, grade with explicit assertions or rubrics, and keep a human in the loop. Google and the agentskills docs add repeated runs for variance, and Google adds continuous runs for regression.

Where we added something

Trigger tests. A skill that never loads is worthless, and the description decides that. The sources mostly cover output quality, and the arXiv paper lists skill selection as future work. We added evals/triggers.json with should-trigger prompts and near-miss should-not-trigger prompts, reported as precision and recall.
A shared eval format and a thin toolchain. The forwarded complaint was that everyone reimplements this. We fixed the format and wrote two small scripts, leaving the runner to the harness.
Continuous mode. Lint on change, full suite on a schedule and on model upgrades, alert on a pass-rate drop.

How skill-eval-generator works

The skill is itself a skill, so it follows the format it teaches. Its workflow:

Lint. lint_skill.mjs checks frontmatter limits, third-person and when-to-use descriptions, line count, broken file references, reference depth and eval presence. Deterministic and free, so it runs first.
Analyze the skill. What does it promise, what would the model get wrong without it, and what environment does it need?
Trigger tests. Eight to ten positives, eight to ten near-misses.
Task evals. Three to start, each with a realistic prompt, an expected output and assertions split into goal and instruction groups. Scripted checks (file exists, regex, valid JSON) are preferred over judged ones.
Validate the suite. The same lint script with --evals flags prompts that name the skill or leak steps, vague assertions, missing instruction-following checks and duplicate ids.
Run with and without the skill in clean contexts, ideally three runs each, saving outputs, transcript and timing.
Grade scripted checks first, then an LLM judge that must quote evidence.
Aggregate. aggregate_benchmark.mjs writes benchmark.json with mean and standard deviation per configuration, the delta, and an assertion triage: always-pass, always-fail, skill-adds-value and flaky.
Human review, improve, repeat until feedback is empty or gains plateau.
Verdict: ship, iterate or delete, with the cost of the skill stated next to what it buys.

A minimal eval, concretely

{
  "id": "top-months-chart",
  "prompt": "I have a CSV of monthly sales in data/sales_2025.csv. Top 3 months by revenue as a bar chart?",
  "expected_output": "Bar chart of the top 3 months with labeled axes.",
  "assertions": {
    "goal": [
      { "text": "A chart image exists", "check": { "type": "file_exists", "glob": "outputs/*.png" } }
    ],
    "instruction": [
      { "text": "Uses the bundled chart script", "check": { "type": "transcript_regex", "pattern": "scripts/chart\\.py" } }
    ]
  }
}

Note the prompt: casual, realistic, and silent about the skill’s method. The instruction group is where a skill proves itself, because a capable model will usually reach the goal either way.

We ran it on a real skill

We pointed the skill at skill-vetter, an existing security-review skill. Setup: three inert fixture skills (a malicious weather skill that exfiltrates SSH keys, AWS credentials and memory files to a raw IP and runs remote commands via eval; a benign note formatter; a medium-risk git digest that calls the GitHub API). Each eval ran with and without the skill, twice, in clean subagents on the same model, with 7 to 12 assertions per eval split into goal and instruction groups. Mechanical checks were scripted; the rest I judged by reading the answers.

	with skill	without skill	delta
Overall pass rate	100%	49.8%	+50.2 points
Goal completion	100%	100%	0
Instruction following	100%	14.3%	+85.7 points

What the data actually says:

The baseline found everything. Without the skill the model caught the exfiltration, the raw IP, the eval and base64 use, and the access to memory files, and told us not to install it. All 13 substance assertions passed in both configurations. The skill did not improve detection.
The whole delta is form plus calibration. With the skill, the answer came in the required report format with an explicit risk level, verdict and permissions section. Without it, it came as free prose. One case also differed in judgment: for the git digest the baseline said “low risk, safe to install”, while the skill’s rubric produced “medium, install with caution”.
Variance sat entirely in the baseline. With the skill, all runs scored 100% (standard deviation 0). Without it, pass rate was 49.8% ± 10.7 and instruction following 14.3% ± 22.1, because the free-form answers drifted. With only two runs per case, treat those spreads as rough.
The eval was too easy. Our own triage flagged it: 13 of 25 assertions passed everywhere, and none failed everywhere. Per the arXiv paper’s point, frontier models complete the goal either way, and what a skill changes is the “how”.

So is skill-vetter worth keeping? Yes if you want a consistent, auditable report format and conservative risk grading. If you only need the model to spot obvious malware, it already does. That is a more useful answer than “100% vs 50%”.

What the exercise taught us about the skill itself

The lint step caught a real mistake: my first eval prompts used paths containing the skill’s name, which leaks it. I moved the fixtures.
The grader runner was missing, so I added run_checks.mjs. The aggregator also needed to skip ungraded assertions.
A fixture comment (“FIXTURE ONLY”) leaked into two answers. Fixtures must look like the real thing.
We did not capture tokens, time or transcripts, so the efficiency axis was not measured and “did not execute the scripts” could not be graded. The skill now says to capture them from the harness at completion.
Next iteration should add hard cases: subtle exfiltration, prompt injection hidden in a reference file, a benign skill that looks suspicious, and multi-skill bundles. We should also separate format assertions from substance in the score.

Lessons and caveats

The honest result is sometimes “delete it.” If the delta is flat, the skill is costing tokens for nothing.
Assertions rot too. Review them each iteration. Ones that always pass inflate the score without measuring anything.
Judges are biased on style. Keep aesthetics for humans, and use a different model to judge than to solve.
Selection is half the problem. Many eval setups tell the agent to use the skill. Trigger tests are the cheap partial answer.
Don’t test against live accounts. Skills that post, email or publish need stubs or dry-run modes.
Our evidence is thin. One skill, three cases, two runs each, a single model and a mostly self-judged rubric. Treat the numbers as a demonstration of the method, not a verdict on skill-vetter.

Should the standard define this?

Our view: define the on-disk eval format and result schema, and leave runners to the ecosystem. A shared evals/ convention makes skills portable and lets tooling accumulate. Runners depend on harness details such as subagents, sandboxing and token accounting, and will differ.

References

Google Cloud Blog, “Behind the scenes: How we build, test, and scale Google Agent Skills”: https://cloud.google.com/blog/topics/developers-practitioners/behind-the-scenes-how-we-build-test-and-scale-google-agent-skills
“A Framework for Evaluating Agentic Skills at Scale”, arXiv 2606.17819: https://arxiv.org/html/2606.17819
Anthropic, Agent Skills best practices, evaluation and iteration: https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices#evaluation-and-iteration
Agent Skills, “Evaluating skill output quality”: https://agentskills.io/skill-creation/evaluating-skills