Companion to the hands-on tutorial: Check your agent skills before merge with NVIDIA SkillEvaluator on Cui.Guru. The tutorial shows how to use it. This post explains why it is built this way.
NVIDIA/SkillEvaluator (about 3.4k stars per the GitHub API on 2026-11-18, Apache 2.0, version 1.0.1 in pyproject.toml) describes itself as a “multi-tier framework for evaluating AI agent skills and plugins”. I cloned it and read src/skillevaluator/, mostly harness/, tier_workflows.py and tier3/harbor/collector.py. The package is large (about 30k lines in the top-level modules alone), so I followed one thread: how a folder with a SKILL.md becomes a verdict.
The workflow in motion
Architecture
The three tiers map to three package areas. Tier 1 (tier1/, validators/, harness/) is static validation with optional LLM checks. Tier 2 (tier2/, deduplication/, embedding/) looks for overlapping guidance and needs embeddings plus chat. Tier 3 (tier3/) runs a real agent; its harbor/ subpackage wraps the Harbor framework and nemo_gym/ is an experimental alternative backend. validate composes them: Tier 1 gates, Tier 2 gates unless --no-block-on-dedup, Tier 3 only with --block-on-agent-eval.
The harness check is the interesting part architecturally. harness-check is “the Tier 1 harness check on its own”, so one engine serves both the standalone panel and the validate pipeline.
The core design idea
Every agent harness has its own skill loader: different folders, YAML strictness and size limits. Instead of hard-coding those differences in Python, SkillEvaluator makes them data. Each harness has a profile in harness/profiles/ (claude-code, codex, copilot-cli, hermes, openclaw, opencode). A profile records which version it was verified against and how, the skill roots it scans, and a list of rules. Each rule is a boolean expression over facts extracted from your skill, an effect (advisory, degraded or not_loaded), a message and a fix.
The facts are computed once per skill in harness/overlay.py: description length in UTF-16 units, whether the frontmatter has duplicate keys or tab indentation, whether the body uses $ARGUMENTS, the install root, catalog size and more. The Copilot CLI profile, for instance, was verified against Copilot CLI 1.0.105 by “proxy-captured canary probes”, and its rule for the description limit reads as one line of YAML. Adding a new harness, or updating one after a release, is editing a file rather than shipping new logic.
The second half of the idea is restraint. Rule expressions are parsed with Python’s ast and only a whitelist of node types is allowed, so a profile directory (you can point --harness-profiles at your own) cannot smuggle in arbitrary code. And because the engine takes the strictest effect, the verdict is simple to explain: one not_loaded rule means the model never sees your skill.
Tier 3 applies the same “reduce it to one comparable number” thinking to behaviour. The agent runs your reviewed cases twice, with and without the skill, and _compute_lift in tier3/harbor/collector.py subtracts the two on a shared basis: only cases and metrics both arms scored, with the same denominator on each side. If a judge fallback means the two arms of a case were graded by different models, that case is left out of that judge metric and counted in judge_mixed_cases. That is why the docs warn the lift can differ from subtracting the displayed arm scores.
Source excerpts
A profile rule is data. From harness/profiles/copilot-cli.yaml:
- {id: CP-DESC-LIMIT, when: "description_utf16_units > 1024", effect: not_loaded, outcome: rejected,
message: "Skill dropped: description is {description_utf16_units} UTF-16 units, {utf16_over} over the 1,024 limit (an emoji counts as 2).",
fix: "Cut the description by at least {utf16_over} UTF-16 units.", confidence: high}
Facts come from the files. In harness/overlay.py, the facts dictionary includes entries like these:
"description_utf16_units": _utf16_units(description),
"frontmatter_duplicate_keys": bool(_duplicate_frontmatter_keys(fm_yaml)),
"frontmatter_tab_indent": _tab_indent_breaks_yaml(fm_yaml),
A whitelisted expression engine. _Rule.evaluate walks the parsed tree and refuses anything outside a short list of node types (names, constants, comparisons, boolean operators, lists):
tree = ast.parse(expression, mode="eval")
for node in ast.walk(tree):
if not isinstance(node, cls._ALLOWED):
raise ValueError(f"Unsupported rule expression: {expression}")
return bool(cls._eval(tree.body, names))
Strictest effect wins. evaluate_profile ranks effects and derives the status from the worst one that fired:
_EFFECT_RANK = {"advisory": 0, "degraded": 1, "not_loaded": 2}
...
status = "not_loaded" if worst == 2 else "degraded" if worst == 1 else "loads"
Gating is explicit. The doc comment on _compute_lift in tier3/harbor/collector.py states the contract: the overall lift is “the mean over the metrics both arms scored, with the same denominator on each side”.
What it looks like. I installed skillevaluator from PyPI and ran harness-check on a throwaway skill with a duplicated license: key and a $ARGUMENTS placeholder. Claude Code loaded it; Codex, Copilot CLI, OpenClaw and OpenCode dropped it; Hermes reported it degraded. The panel footer read “4 of 6 not loaded”. Running quality-check on the same file scored 79.5/100 (grade C) and still passed: a skill can score acceptably and still be invisible to most harnesses, which is exactly the gap this tool targets.
Takeaways
- Treat harness quirks as versioned data with a recorded verification method, not as scattered
ifstatements. - A tiny whitelisted expression language is enough for rules, and safe enough to let users supply their own profiles.
- Separate “will it load?” (static, free, seconds) from “does it help?” (live, costed, paired) and let CI gate on each independently.
- A paired with/without design, restricted to shared cases and metrics, keeps a single lift number honest.
- The tool labels itself Experimental, and its profiles are tied to specific harness versions, so re-check after upgrades.
Want skill quality gates wired into your own agent pipeline? Get in touch.
Click to load Disqus comments