NVIDIA/SkillEvaluator hit GitHub trending for week 47 of 2026 on November 19, 2026, with roughly 3.8k stars already on the public counter—and it answers a concrete question for teams shipping Claude Code, Codex, and other agent skills: can you gate skill folders in CI the way you gate application code? The project ships a three-tier evaluation pipeline, reusable GitHub and GitLab templates, and Tier 1 security scanning that pulls in SkillSpector plus Gitleaks, Bandit, optional Semgrep, and dependency auditing.
If you run coding agents locally, the practical shift is not star count. It is whether every skill pull request must pass harness compatibility (will Claude Code or Codex actually load the skill?) and static security checks before merge—without writing your own evaluation harness.
NVIDIA's SkillEvaluator documentation positions the tool inside the NVIDIA Verified Skills pipeline and the NVIDIA/skills catalog. Support is labeled experimental with no SLA—treat reports as evidence to review, not as a compliance stamp.
TL;DR
| Question | Answer |
|---|---|
| Am I affected? | Yes if you install or publish third-party skills without CI gates—SkillEvaluator is the stack many teams will copy |
| Is there a fix? | Process: run skillevaluator validate --tiers 1 (or ci init) on skill paths before merge; pin CLI version in CI |
| Does data leave my machine? | Tier 1 keyless checks stay local; Tier 2–3 use your configured LLM provider; scanners may need Docker |
| What should I run first? | pip install skillevaluator then skillevaluator quality-check ./my-skill or skillevaluator harness-check ./my-skill |
| Is this the same as last month's SkillSpector post? | Related but broader—see our SkillSpector explainer for the dedicated scanner; SkillEvaluator orchestrates it |
Why SkillEvaluator is trending now
GitHub trending is a velocity signal, not a security review. Week 47 surfaced SkillEvaluator beside smaller repos because maintainers shipped a documented CI story (skillevaluator ci init) at the same time interest in agent skills spiked across Claude Code, Codex, and OpenCode.
The README's pitch is operational: "Catch skill issues before merge" and connect a repository to automated quality and security gates without building evaluation code. That matches what security teams asked for after incidents involving poisoned skill text and MCP tool descriptions—mechanisms we covered in the malicious AGENTS.md hijack write-up and the MCP security practical guide.
SkillEvaluator also rides the same wave as large harness bundles such as ECC (ecc-universal), where hundreds of skills arrive in one install command. A trending evaluator does not replace reading those files—it gives maintainers a repeatable gate when skill count grows faster than manual review.
The three tiers (what each one answers)
NVIDIA documents three independent tiers in the SkillEvaluator docs and README:
| Tier | Question | Example command | Typical gate? |
|---|---|---|---|
| Tier 1: Validation | Is the skill well-formed, safe enough to try, and harness-compatible? | skillevaluator tier1 ./my-skill --no-llm | Yes — validate blocks on Tier 1 failures |
| Tier 1: Harness check | Will the agent load the skill under harness limits? | skillevaluator harness-check ./my-skill | Standalone diagnostic; included in full Tier 1 |
| Tier 2: Deduplication | Does guidance repeat or overlap other skills? | skillevaluator tier2 ./my-skill | Blocks unless --no-block-on-dedup |
| Tier 3: Live agent evaluation | Does the skill improve task outcomes? | skillevaluator tier3 ./my-skill | Optional; needs --block-on-agent-eval to fail CI |
Tier 1 is what most coding-agent teams should adopt first. It runs without an LLM API key when you pass --no-llm, though the README is explicit that omitting scanners makes coverage INCOMPLETE and can exit nonzero—external scanners are part of "full static coverage," not an optional nice-to-have.
Tier 3 uses Docker sandboxes and supported agent runtimes (Harbor is named in the README as the open-source evaluation framework behind sandboxed runs). That tier measures lift, not just safety—useful for authors, expensive for every PR.
Tier 1 security stack (SkillSpector and friends)
The README lists 11 CI checks backed by 500+ static rules and detection patterns for a full install. Security-relevant pieces include:
- SkillSpector for skill-specific attack patterns (the same family as our SkillSpector deep dive).
- Gitleaks for secret patterns in the skill tree.
- Bandit for risky Python in bundled scripts.
- Semgrep when configured (optional).
- pip-audit for known vulnerable dependencies.
Quality gates alongside security cover schema against the Agent Skills specification, version metadata, PII heuristics, license fields, Unicode and script linting, and the Harness Check profile.
Harness Check: load limits per agent
Harness Check is the piece most laptop operators miss. The README documents example output showing per-harness status: skills that truncate descriptions to fit list limits, paths agents do not scan (for example .agents/skills not loaded on some harnesses), and UTF-16 length caps affecting Copilot CLI.
That matters because a "secure" skill nobody loads is still a deployment failure—and a renamed copy in a searched directory might load instead. The agent skills specification and security review post walked the same trust-boundary problem at the format level; Harness Check automates one slice of it.
Example from the README (illustrative harness names and limits):
skillevaluator harness-check ./my-skill
The tool prints which harnesses load, which are degraded (for example description truncated), and which fail load rules—without running a model.
How to run checks locally
Keyless quality score
For a first pass with no API key, GPU, or Docker:
pip install skillevaluator
skillevaluator quality-check ./my-skill
The README says this prints a score out of 100 with a grade and findings; default passing threshold is 70. Publication evidence stays INCOMPLETE until security and agent-impact stages run—do not treat a high quality score as a security clearance.
Full Tier 1 validation
With scanners installed per the README's scanner setup guide:
skillevaluator validate ./my-skill --tiers 1
Use --no-llm when you want deterministic static gates only. The combined pipeline command is:
skillevaluator doctor --env-mode docker
skillevaluator validate ./my-skill
doctor checks environment readiness; it does not prove inference success on your skill.
Install the skill into your agent (optional)
The README documents distributing SkillEvaluator itself as a skill:
npx skills add NVIDIA/SkillEvaluator
Then you can ask the agent to evaluate another skill—convenient, but remember the evaluated skill's content still flows through your agent's trust boundary. For untrusted folders, prefer CLI runs in a container or dedicated review machine.
Wiring GitHub CI
The maintenance story that drives trending adoption is skillevaluator ci init:
skillevaluator ci init
Run it from a cloned repository. The CLI generates a setup pull request with workflow templates—skill-checks.yml lands under .github/workflows/ for GitHub. NVIDIA documents reusable CI templates in the ci/ directory for GitHub and GitLab, plus optional PR comment workflows.
Default templates emphasize Tier 1 quality and security without a model key. All-tier CI adds deduplication and live agent evaluation—pull that in only when you have provider credentials, Docker, and reviewed datasets.
For comparison, GitHub and Microsoft's MDASH pilot showed how AI-assisted findings still need human-verified gates before merge. SkillEvaluator is the skill-author analogue: static rules first, optional agent eval when you choose.
What this means if you run AI coding agents
Before you npx skills add a trending folder:
- Pin the evaluator version in CI (
pip install skillevaluator==<version>or lockuv.lockfrom the SkillEvaluator release you tested). - Run Tier 1 on the skill directory—even if the publisher is NVIDIA or another brand name.
- Read Harness Check output for your harness (Claude Code vs Codex vs Copilot CLI paths differ).
- Do not merge skills that fail Gitleaks or SkillSpector without a documented exception baseline—SkillEvaluator supports baselines, but exclusions make coverage partial per README rules.
On developer laptops, combine static gates with runtime observability: agents still execute shell commands and read secrets after a clean scan. Emergency patches and supply-chain stories from November 2026 are a reminder that concurrent production change breaks stacks even when intent is defensive—see AI patch rush and outage claims for the organizational side, separate from skill supply chain.
Concrete commands to verify your install:
pip show skillevaluator
skillevaluator harness-check ./path-to-skill
skillevaluator validate ./path-to-skill --tiers 1 --no-llm
Check exit codes in CI—the README notes incomplete scanner runs should fail closed when configured.
How Beam handles this
Beam overlaps SkillEvaluator on pre-run review, not on multi-tier agent benchmarking.
On your machine, beam scan performs an offline heuristic pass on a skill file or MCP JSON—11 patterns plus an MCP version-pin check per the scanning docs—without executing the skill. beam scan --save can persist a report through the local collector when enrolled, but a plain scan stays on-device. Workspace enforce policy can deny specific PreToolUse proposals on Claude Code and Codex when cached policy allows; other harnesses may only show advisory text, and missing cached policy is fail-open.
SkillEvaluator is stronger when you need CI-wide gates, harness-specific load diagnostics, and SkillSpector's broader pattern set in one orchestrated pipeline. Beam is stronger when you want continuous local telemetry of what the agent actually did after the skill loaded—credential exposure, destructive commands, and MCP calls—under workspace policy you control.
Use both when you maintain a skill repo in GitHub and run agents on a laptop with production credentials: SkillEvaluator on PRs, Beam on the workstation doing the merges.
Limits the README states plainly
- Experimental support — no enterprise SLA; file GitHub Issues for defects.
- Heuristic security — SkillSpector and linters catch known classes; they do not prove benign intent.
- Scanner incompleteness — a scanner that cannot run yields INCOMPLETE status; do not silence that in CI without accepting blind spots.
- Tier 3 cost and complexity — live agent eval needs Docker, provider keys, and human review of generated cases; autopilot starters are not a substitute for authored negative tests.
- Third-party downloads — install pulls additional open-source packages; review licenses.
Related reading
- NVIDIA SkillSpector: agent skill scanner deep dive
- Agent skills specification and security review
- ECC trending harness: install paths and supply-chain traps
- Cloudflare security-audit skill: phased review pattern
- What Beam watches on coding agents
Primary sources: NVIDIA SkillEvaluator on GitHub, SkillEvaluator documentation, and SkillSpector.
Frequently asked questions
The FAQ entries in the front matter mirror this section for schema markup.
What is NVIDIA SkillEvaluator?
An open-source, multi-tier framework to validate agent skills for quality, security, harness compatibility, overlap, and optional live agent lift— with CI templates for GitHub and GitLab.
How do I run an agent skill security check without a cloud API key?
Use skillevaluator quality-check or skillevaluator validate --tiers 1 --no-llm after installing scanners for full Tier 1 coverage; harness-check is also keyless.
Is SkillEvaluator an agent skill security scanner?
Tier 1 includes SkillSpector and other security tools, but the project also scores quality and compatibility. For security-only focus, SkillSpector alone may suffice.
What are agent skills security risks SkillEvaluator does not remove?
Runtime behavior after load, social engineering in instructions, novel exfiltration paths, and compromised update channels outside the scanned commit.
Should I use awesome agent skills lists without evaluation?
Lists help discovery; they are not review. Run Tier 1 (or beam scan locally) before enabling a skill with shell access.
Version numbers, scanner lists, and harness profiles reflect NVIDIA's SkillEvaluator README and documentation as of November 19, 2026.

