Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

SkillEvaluator

SkillEvaluator• Pinned

SkillEvaluator: An open-source tool that checks AI agent skills and measures their effect on task results.

What problem it solves

AI agent skills need to be checked for quality, security, duplication, and their actual effect on agent behavior. Static checks alone cannot establish whether guidance improves task results, so evaluation must also support live comparison with and without a skill.

Key features

  • Three-tier evaluation pipeline — Independent tiers validate skill safety and structure, detect overlapping guidance, and evaluate skill impact through live agent runs.

    Source: README

  • CI quality gates — Repository teams can run quality checks, security scanning, and reporting in GitHub or GitLab without importing evaluator code.

    Source: README, ci

  • Semantic overlap detection — The deduplication tier can examine repetition within a skill and compare guidance across a skill catalog.

    Source: README

  • Live agent evaluation — The live tier runs supported agents in a sandbox and compares task outcomes to determine whether a skill helps the agent.

    Source: README

  • Evaluation dataset generation — Broader live coverage can be created with generated evaluation cases, including fuller datasets for teams reviewing their tests.

    Source: README, src

Quick start

Copied verbatim from the project's README.

pip install --upgrade skillevaluator
skillevaluator ci init
uv run --python 3.13 skillevaluator quality-check \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-quality
uv run --python 3.13 --extra tier3 skillevaluator tier3 create-eval-dataset ./my-skill --full

Who it may suit

This may suit teams that maintain AI agent skills and want repository-level quality checks, as well as evaluators studying how skill guidance changes agent behavior.

Topics to discuss

  • Which quality gates should be mandatory in a shared skill repository?

  • How should semantic overlap be measured across related agent skills?

  • What live evaluation cases provide useful coverage without making maintenance excessive?

Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.

Read the original README https://github.com/NVIDIA/SkillEvaluator#readme

ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.

github.comhttps://github.com/NVIDIA/SkillEvaluator

Comments

…