ai, agents, deep dive,

Deep Dive: How NVIDIA SkillEvaluator Predicts Whether Your Agent Skill Will Even Load

Cui Cui Follow Nov 18, 2026 · 7 mins read
Deep Dive: How NVIDIA SkillEvaluator Predicts Whether Your Agent Skill Will Even Load
Share this

Companion to the hands-on tutorial: Check your agent skills before merge with NVIDIA SkillEvaluator on Cui.Guru. The tutorial shows how to use it. This post explains why it is built this way.

NVIDIA/SkillEvaluator (about 3.4k stars per the GitHub API on 2026-11-18, Apache 2.0, version 1.0.1 in pyproject.toml) describes itself as a “multi-tier framework for evaluating AI agent skills and plugins”. I cloned it and read src/skillevaluator/, mostly harness/, tier_workflows.py and tier3/harbor/collector.py. The package is large (about 30k lines in the top-level modules alone), so I followed one thread: how a folder with a SKILL.md becomes a verdict.

The workflow in motion

Animated workflow: a skill folder is read into static facts, evaluated against six harness profiles, producing a loads, degraded or not_loaded verdict per harness; then Tier 3 runs the agent with and without the skill and computes Skill Lift

Architecture

The three tiers map to three package areas. Tier 1 (tier1/, validators/, harness/) is static validation with optional LLM checks. Tier 2 (tier2/, deduplication/, embedding/) looks for overlapping guidance and needs embeddings plus chat. Tier 3 (tier3/) runs a real agent; its harbor/ subpackage wraps the Harbor framework and nemo_gym/ is an experimental alternative backend. validate composes them: Tier 1 gates, Tier 2 gates unless --no-block-on-dedup, Tier 3 only with --block-on-agent-eval.

The harness check is the interesting part architecturally. harness-check is “the Tier 1 harness check on its own”, so one engine serves both the standalone panel and the validate pipeline.

The core design idea

Every agent harness has its own skill loader: different folders, YAML strictness and size limits. Instead of hard-coding those differences in Python, SkillEvaluator makes them data. Each harness has a profile in harness/profiles/ (claude-code, codex, copilot-cli, hermes, openclaw, opencode). A profile records which version it was verified against and how, the skill roots it scans, and a list of rules. Each rule is a boolean expression over facts extracted from your skill, an effect (advisory, degraded or not_loaded), a message and a fix.

The facts are computed once per skill in harness/overlay.py: description length in UTF-16 units, whether the frontmatter has duplicate keys or tab indentation, whether the body uses $ARGUMENTS, the install root, catalog size and more. The Copilot CLI profile, for instance, was verified against Copilot CLI 1.0.105 by “proxy-captured canary probes”, and its rule for the description limit reads as one line of YAML. Adding a new harness, or updating one after a release, is editing a file rather than shipping new logic.

The second half of the idea is restraint. Rule expressions are parsed with Python’s ast and only a whitelist of node types is allowed, so a profile directory (you can point --harness-profiles at your own) cannot smuggle in arbitrary code. And because the engine takes the strictest effect, the verdict is simple to explain: one not_loaded rule means the model never sees your skill.

Tier 3 applies the same “reduce it to one comparable number” thinking to behaviour. The agent runs your reviewed cases twice, with and without the skill, and _compute_lift in tier3/harbor/collector.py subtracts the two on a shared basis: only cases and metrics both arms scored, with the same denominator on each side. If a judge fallback means the two arms of a case were graded by different models, that case is left out of that judge metric and counted in judge_mixed_cases. That is why the docs warn the lift can differ from subtracting the displayed arm scores.

Source excerpts

A profile rule is data. From harness/profiles/copilot-cli.yaml:

- {id: CP-DESC-LIMIT, when: "description_utf16_units > 1024", effect: not_loaded, outcome: rejected,
   message: "Skill dropped: description is {description_utf16_units} UTF-16 units, {utf16_over} over the 1,024 limit (an emoji counts as 2).",
   fix: "Cut the description by at least {utf16_over} UTF-16 units.", confidence: high}

Facts come from the files. In harness/overlay.py, the facts dictionary includes entries like these:

"description_utf16_units": _utf16_units(description),
"frontmatter_duplicate_keys": bool(_duplicate_frontmatter_keys(fm_yaml)),
"frontmatter_tab_indent": _tab_indent_breaks_yaml(fm_yaml),

A whitelisted expression engine. _Rule.evaluate walks the parsed tree and refuses anything outside a short list of node types (names, constants, comparisons, boolean operators, lists):

tree = ast.parse(expression, mode="eval")
for node in ast.walk(tree):
    if not isinstance(node, cls._ALLOWED):
        raise ValueError(f"Unsupported rule expression: {expression}")
return bool(cls._eval(tree.body, names))

Strictest effect wins. evaluate_profile ranks effects and derives the status from the worst one that fired:

_EFFECT_RANK = {"advisory": 0, "degraded": 1, "not_loaded": 2}
...
status = "not_loaded" if worst == 2 else "degraded" if worst == 1 else "loads"

Gating is explicit. The doc comment on _compute_lift in tier3/harbor/collector.py states the contract: the overall lift is “the mean over the metrics both arms scored, with the same denominator on each side”.

What it looks like. I installed skillevaluator from PyPI and ran harness-check on a throwaway skill with a duplicated license: key and a $ARGUMENTS placeholder. Claude Code loaded it; Codex, Copilot CLI, OpenClaw and OpenCode dropped it; Hermes reported it degraded. The panel footer read “4 of 6 not loaded”. Running quality-check on the same file scored 79.5/100 (grade C) and still passed: a skill can score acceptably and still be invisible to most harnesses, which is exactly the gap this tool targets.

Takeaways

  • Treat harness quirks as versioned data with a recorded verification method, not as scattered if statements.
  • A tiny whitelisted expression language is enough for rules, and safe enough to let users supply their own profiles.
  • Separate “will it load?” (static, free, seconds) from “does it help?” (live, costed, paired) and let CI gate on each independently.
  • A paired with/without design, restricted to shared cases and metrics, keeps a single lift number honest.
  • The tool labels itself Experimental, and its profiles are tied to specific harness versions, so re-check after upgrades.

Want skill quality gates wired into your own agent pipeline? Get in touch.

Join Newsletter
Get the latest news right in your inbox. We never spam!
Cui
Written by Cui Follow
Hi, I am Z, the coder for cuizhanming.com!

Click to load Disqus comments