Skip to content
Ironheights
Last checked

Ironheights vs Cisco skill-scanner

Short answer: Cisco's skill-scanner is the more capable detector when you run it with its LLM judge, and it publishes held-out accuracy we cannot match yet. Ironheights is smaller and rules-only, built around OpenClaw, and adds integrity checks for installed skills and agent files.

At a glance

Cisco skill-scanner compared with Ironheights
Cisco skill-scannerIronheights
License and makerOpen source, Apache-2.0, from Cisco AI Defense[1]Open source, Apache-2.0, independent project
InstallPython (CPython 3.11–3.14) via uv, pip, or Homebrew; version 2.2.2 at the time of checking[1][3]Node.js 20 or newer, one npx command (docs)
Skill formatsAgent Skills specification (Codex and Cursor skill formats), plus non-standard formats with --lenient[1]OpenClaw skills, with OpenClaw paths built in and a doctor command for your install
How it detectsYAML and YARA-X patterns, AST and dataflow analysis, an optional LLM judge, and a CEL decision layer[1]35 published rules (text, MCP config, advisory and integrity) whose severities add up to a review or block verdict and an A to F grade (how it works)
Reads intentYes, with the LLM judge, which Cisco runs in every recommended setup. It can use a local model so nothing leaves the machine[2]No. Fixed patterns decide the verdict. An optional model second opinion is off by default, runs against a server you choose, and only adds advisory notes
Published accuracyOn 1,384 held-out records: rules alone catch 8.0% of malicious skills at MEDIUM+ (4.2% false positives). With the judge, 66.7% reach review at a 15.4% false-positive rate[2]A synthetic 20-skill regression check only. No real-world rate yet[4]
CI and automationSARIF, a reusable GitHub Actions workflow, a pre-commit hook, a Python API, and a REST API[1][2]JSON, SARIF, Markdown and HTML reports, --fail-on, --since-baseline, documented exit codes, a GitHub Action and a pre-commit hook
After installNot covered in the documentation we reviewedSigned baselines and verify for installed skills and agent files, quarantine that moves a skill aside instead of deleting it, and an optional guard plugin that logs risky OpenClaw tool calls. The guard is not a sandbox

Where each one is stronger

Where Cisco skill-scanner is stronger
  • The LLM judge reads skills for intent. Cisco's own numbers show the judge is what lifts recall from about 8% to about two thirds[2].
  • It publishes recall, false-positive rate, and F1 on a held-out set of 1,384 records, with reproduction scripts[2]. We have nothing comparable yet.
  • Deeper analysis: AST and dataflow checks on bundled code, YARA rules, and optional VirusTotal and OSV lookups[1][2].
  • Ready-made CI pieces, from a reusable GitHub Actions workflow to a pre-commit hook[1].
Where Ironheights is different
  • Built for OpenClaw: it knows the OpenClaw skill and state paths and checks your install with doctor.
  • It watches what happens after install: signed baselines of skills and agent files such as AGENTS.md, SOUL.md, and openclaw.json, then verify, plus an optional guard plugin for tool calls.
  • No model, key, or Python environment needed. The flip side is that rules alone decide the verdict and miss what only an intent reader would catch.
  • Small enough to read: 35 rules, each documented in plain language.

On our benchmark

On our 20-skill synthetic corpus, Ironheights (version 0.1.0, the version the benchmark ran on) sent 10/10 malicious samples to review and Cisco's rules sent 4/10, with no benign sample flagged by either. That comparison favors us and is not fair to Cisco: we wrote the corpus to match our own rules, and we ran Cisco without the LLM judge it recommends. It is a regression check, not evidence that Ironheights detects more in the real world. The full method and every sample are on the benchmark page[4].

Using both

The two do not conflict. A reasonable setup for a team that installs third-party skills:

  1. Scan new skills with Cisco's scanner and its judge in CI, and send MEDIUM and above to a reviewer, as Cisco recommends.
  2. Scan with Ironheights before installing on an OpenClaw machine, and record a baseline after.
  3. Run verify on a schedule so a changed skill or agent file is noticed.
skill-scanner scan ./downloaded-skill --use-llm --policy balanced --fail-on-severity high
npx ironheights scan ./downloaded-skill
npx ironheights baseline create
npx ironheights verify   # later, after updates

Sources

  1. cisco-ai-defense/skill-scanner README, GitHub.
  2. Recommended Settings, Cisco Skill Scanner documentation.
  3. cisco-ai-skill-scanner 2.2.2, PyPI.
  4. Ironheights benchmark #1, Ironheights, 9 October 2026.

All sources last checked on 9 October 2026.

See the benchmark All comparisonsIronheights limitations