MCPInstall

ai-scanner-garak

AI model safety scanner built on NVIDIA garak for testing LLMs against 179 security probes across 35 vulnerability families

Installs
878
Last 24h
—
Trending rank
—
Security
Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/reason-machines/trending-skills --skill ai-scanner-garak
  1. 01

    Install

    Run the command in your project. The CLI asks which agents to install for.

  2. 02

    Describe the task

    Ask your agent for something this skill covers. It loads ai-scanner-garak automatically when relevant.

  3. 03

    Review the result

    The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

AI Scanner (0din-ai/ai-scanner)

Skill by ara.so — Daily 2026 Skills collection.

AI Scanner is an open-source Ruby on Rails web application for AI model security assessments, wrapping NVIDIA garak with a multi-tenant UI, scheduling, PDF reports, and SIEM integration. It runs 179 community probes across 35 vulnerability families aligned with the OWASP LLM Top 10.

Installation

Quick Install (Docker)

curl -sL https://raw.githubusercontent.com/0din-ai/ai-scanner/main/scripts/install.sh | bash

Manual Install

curl -O https://raw.githubusercontent.com/0din-ai/ai-scanner/main/dist/docker-compose.yml
curl -O https://raw.githubusercontent.com/0din-ai/ai-scanner/main/.env.example
cp .env.example .env

Edit .env with required values:

# Generate a secure key
openssl rand -hex 64

# .env minimum required values
SECRET_KEY_BASE=<output_of_above_command>
POSTGRES_PASSWORD=<your_secure_db_password>
docker compose up -d

Access at http://localhost — default credentials: admin@example.com / password.
Change the default password immediately after first login.

Configuration (.env)

# Required
SECRET_KEY_BASE=<64-byte-hex>
POSTGRES_PASSWORD=<strong-password>

# Optional: custom port
PORT=8080

# Optional: SIEM integration
SPLUNK_HEC_URL=https://splunk.example.com:8088/services/collector
SPLUNK_HEC_TOKEN=$SPLUNK_HEC_TOKEN
RSYSLOG_HOST=syslog.example.com
RSYSLOG_PORT=514

# Optional: email
SMTP_HOST=smtp.example.com
SMTP_PORT=587
SMTP_USERNAME=$SMTP_USERNAME
SMTP_PASSWORD=$SMTP_PASSWORD

Core Concepts

ConceptDescription
TargetAn AI system to test — API-based LLM or browser-based chat UI
ProbeA single attack test (e.g., prompt injection, data leakage)
ScanA run of selected probes against a target
ASRAttack Success Rate — percentage of probes that succeeded
OrganizationTenant boundary; users and scans are scoped per org

Setting Up a Target

Targets define what you're scanning. Two types:

API-based LLM Target (e.g., OpenAI-compatible endpoint):

# In Rails console or via UI — representative model
target = Target.create!(
  name: "Production GPT-4",
  target_type: "api",
  api_endpoint: "https://api.openai.com/v1/chat/completions",
  api_key: ENV["OPENAI_API_KEY"],
  model_name: "gpt-4",
  organization: current_organization
)

Browser-based Chat UI Target:

target = Target.create!(
  name: "Internal Chatbot UI",
  target_type: "browser",
  url: "https://chatbot.internal.example.com",
  organization: current_organization
)

Running a Scan

Via UI

  1. Navigate to Targets → select your target
  2. Click New Scan
  3. Select probe families or individual probes
  4. Click Run Scan

Via Rails Console

# On-demand scan with specific probe families
scan = Scan.create!(
  target: target,
  probe_families: ["prompt_injection", "data_leakage", "insecure_output"],
  organization: current_organization
)
ScanJob.perform_later(scan.id)

Scheduled Recurring Scan

# Weekly scan every Monday at 2am
scheduled_scan = ScheduledScan.create!(
  target: target,
  probe_families: ["prompt_injection", "jailbreak"],
  cron_expression: "0 2 * * 1",
  organization: current_organization
)

Probe Families (35 total, aligned to OWASP LLM Top 10)

Key probe families available:

# List all available probe families
Garak::ProbeRegistry.families
# => ["prompt_injection", "jailbreak", "data_leakage", "insecure_output",
#     "supply_chain", "sensitive_info", "excessive_agency", "overreliance",
#     "model_theft", "malicious_plugins", ...]

# Get probes within a family
Garak::ProbeRegistry.probes_for("prompt_injection")
# => 179 total probes across all families

Viewing Results

Attack Success Rate (ASR)

scan = Scan.find(scan_id)

puts scan.asr_score          # => 0.23 (23% attack success rate)
puts scan.status             # => "completed"
puts scan.probe_results.count # => 47

# Per-probe breakdown
scan.probe_results.each do |result|
  puts "#{result.probe_name}: #{result.passed? ? 'SAFE' : 'VULNERABLE'}"
  puts "  Attempts: #{result.attempt_count}"
  puts "  ASR: #{result.asr_score}"
end

Trend Tracking

# Compare ASR across scan runs for a target
target.scans.completed.order(:created_at).map do |scan|
  { date: scan.created_at, asr: scan.asr_score }
end

PDF Report Export

# Generate PDF report for a scan
scan = Scan.find(scan_id)
pdf_path = ReportExporter.export_pdf(scan)
# Includes: executive summary, per-probe results, per-attempt drill-down

Via UI: Navigate to a completed scan → Export PDF.

SIEM Integration

Splunk

# config/initializers/siem.rb
SiemIntegration.configure do |config|
  config.provider = :splunk
  config.splunk_hec_url = ENV["SPLUNK_HEC_URL"]
  config.splunk_hec_token = ENV["SPLUNK_HEC_TOKEN"]
  config.forward_on_completion = true
end

Rsyslog

SiemIntegration.configure do |config|
  config.provider = :rsyslog
  config.rsyslog_host = ENV["RSYSLOG_HOST"]
  config.rsyslog_port = ENV["RSYSLOG_PORT"].to_i
end

Multi-Tenant Organization Management

# Create a new organization
org = Organization.create!(name: "Security Team Alpha")

# Invite a user
user = User.invite!(
  email: "analyst@example.com",
  organization: org,
  role: "analyst"   # roles: "admin", "analyst", "viewer"
)

# Data is encrypted at rest per organization
org.encryption_key  # => managed automatically

Development Setup

git clone https://github.com/0din-ai/ai-scanner.git
cd ai-scanner
cp .env.example .env.development

# Install dependencies
bundle install

# Database setup
rails db:create db:migrate db:seed

# Install garak (Python dependency)
pip install garak

# Start development server
bin/dev

Running Tests

# Full test suite
bundle exec rspec

# Specific area
bundle exec rspec spec/models/scan_spec.rb
bundle exec rspec spec/jobs/scan_job_spec.rb

# Lint
bundle exec rubocop

Common Patterns

Testing a New LLM Before Deployment

# Comprehensive pre-deployment scan
target = Target.create!(
  name: "New Model v2 - Pre-deploy",
  target_type: "api",
  api_endpoint: ENV["NEW_MODEL_ENDPOINT"],
  api_key: ENV["NEW_MODEL_API_KEY"],
  model_name: "new-model-v2",
  organization: current_organization
)

# Run all 35 probe families
scan = Scan.create!(
  target: target,
  probe_families: Garak::ProbeRegistry.families,
  organization: current_organization
)
ScanJob.perform_now(scan.id)

if scan.reload.asr_score > 0.15
  puts "WARNING: ASR #{scan.asr_score} exceeds threshold. Review before deploying."
else
  puts "PASS: Model meets security threshold."
end

Using the Mock LLM for Testing Scanner Setup

The built-in Mock LLM lets you validate your scanner configuration without hitting real APIs:

target = Target.create!(
  name: "Mock LLM",
  target_type: "mock",
  organization: current_organization
)
# Run a quick scan to verify everything works end-to-end

Webhook on Scan Completion

# config/initializers/scan_hooks.rb
ActiveSupport::Notifications.subscribe("scan.completed") do |_, _, _, _, payload|
  scan = Scan.find(payload[:scan_id])
  if scan.asr_score > 0.20
    SlackNotifier.alert(
      channel: "#security-alerts",
      message: "High ASR detected: #{scan.asr_score} on #{scan.target.name}"
    )
  end
end

Docker Compose Production Tips

# docker-compose.override.yml — production additions
services:
  web:
    environment:
      RAILS_ENV: production
      FORCE_SSL: "true"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.scanner.rule=Host(`scanner.example.com`)"
      - "traefik.http.routers.scanner.tls.certresolver=letsencrypt"
# Upgrade
docker compose pull
docker compose up -d
docker compose exec web rails db:migrate

Troubleshooting

ProblemSolution
Scan stuck in "running"Check docker compose logs worker — garak Python process may have crashed
SECRET_KEY_BASE error on startRun openssl rand -hex 64 and set in .env
Can't connect to target APIVerify API key env var is set; check firewall allows outbound from container
Browser target scan failsEnsure Playwright/Chrome is available in the worker container
PDF export blankCheck wkhtmltopdf is installed in the web container
SIEM not receiving eventsVerify SPLUNK_HEC_URL includes full path /services/collector
# View all service logs
docker compose logs -f

# Check worker specifically (runs garak)
docker compose logs -f worker

# Rails console for debugging
docker compose exec web rails console

# Check garak is working
docker compose exec worker python -c "import garak; print(garak.__version__)"

Key Files (for Contributors)

app/
  models/
    scan.rb          # Core scan model, ASR calculation
    target.rb        # Target types and validation
    probe_result.rb  # Per-probe result storage
  jobs/
    scan_job.rb      # Async job that invokes garak
  services/
    garak_runner.rb  # Ruby wrapper around garak CLI
    report_exporter.rb
    siem_integration.rb
lib/
  garak/
    probe_registry.rb  # 179 probes, 35 families
dist/
  docker-compose.yml   # Production compose file
scripts/
  install.sh           # One-line installer

Resources

FAQ

What does the ai-scanner-garak skill do?

AI model safety scanner built on NVIDIA garak for testing LLMs against 179 security probes across 35 vulnerability families

How do I install ai-scanner-garak?

Run npx skills add https://github.com/reason-machines/trending-skills --skill ai-scanner-garak in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/ai-scanner-garak/ (personal) or .claude/skills/ai-scanner-garak/ (project).

Which AI agents can use ai-scanner-garak?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is ai-scanner-garak safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains ai-scanner-garak?

It is published by reason-machines in reason-machines/trending-skills. It has 878 recorded installs.

More from reason-machines/trending-skills

View all →

openclaw-control-center

reason-machines/trending-skills

Local-first, security-first control center for OpenClaw agents — visibility dashboard with readonly defaults, token attribution, collaboration tracing, and safe write operations.

Data & Databases4.4K installs

inkos-multi-agent-novel-writing

reason-machines/trending-skills

Multi-agent CLI system for autonomous novel writing, auditing, and revision with human review gates

Docs & Writing2.8K installs

claude-hud-statusline

reason-machines/trending-skills

A Claude Code plugin that displays a real-time HUD showing context usage, active tools, running agents, and todo progress in your terminal statusline.

understand-anything-knowledge-graph

reason-machines/trending-skills

Turn any codebase into an interactive knowledge graph using Claude Code skills — explore, search, and ask questions about any project visually.

agency-agents-ai-specialists

reason-machines/trending-skills

A collection of specialized AI agent personalities for Claude Code, Cursor, Aider, Windsurf, and other AI coding tools — covering engineering, design, marketing, sales, and more.

AI & Agents2.1K installs

agent-browser-automation

reason-machines/trending-skills

Headless browser automation CLI for AI agents using native Rust binary with Chrome DevTools Protocol

Popular security skills

View category →

azure-compliance

microsoft/azure-skills

Run Azure compliance and security audits with azqr plus Key Vault expiration checks. Covers best-practice assessment, resource review, policy/compliance validation, and security posture checks. WHEN: compliance scan, security audit, BEFORE running azqr (compliance cli tool), Azure best practices, Key Vault expiration check, expired certificates, expiring secrets, orphaned resources, compliance assessment.

OfficialSecurity567K installs

minimal-run-and-audit

lllllllama/rigorpilot-skills

Rigor Run skill for README-first deep learning repo reproduction. Use when the task is specifically to capture or normalize evidence from the selected smoke test or documented inference or evaluation command and write standardized reprooutputs/ files, including patch notes when repository files changed. Do not use for training execution, initial repo intake, generic environment setup, paper lookup, target selection, hidden scientific-meaning changes, or end-to-end orchestration by itself.

Security477.4K installs

seo-audit

coreyhaines31/marketingskills

When the user wants to audit, review, or diagnose SEO issues on their site. Also use when the user mentions "SEO audit," "technical SEO," "why am I not ranking," "SEO issues," "on-page SEO," "meta tags review," "SEO health check," "my traffic dropped," "lost rankings," "not showing up in Google," "site isn't ranking," "Google update hit me," "page speed," "core web vitals," "crawl errors," "indexing issues," "stuck on page 2," "striking distance keywords," "title tag rewrite," "low CTR," "local SEO," "Google Business Profile," or "map pack." Use this even if the user just says something vague like "my SEO is bad" or "help with SEO" — start with an audit. For building pages at scale to target keywords, see programmatic-seo. For adding structured data, see schema. For AI search optimization, see ai-seo.

Security238.7K installs

firebase-security-rules-auditor

firebase/agent-skills

Audits Firebase (Firestore, Cloud Storage) security rules for vulnerabilities, privilege escalation, role bypasses, create vs update inconsistencies, resource exhaustion, type safety, size limits, and hasOnly ownership checks. Use when auditing/reviewing rules, running red-team rule assessments, or scoring against auditor checklists. Don't use for Firebase CLI (login, deploy), Auth, Crashlytics, Remote Config, or database queries.

OfficialSecurity139.1K installs

audit

pbakaus/impeccable

Run technical quality checks across accessibility, performance, theming, responsive design, and anti-patterns. Generates a scored report with P0-P3 severity ratings and actionable plan. Use when the user wants an accessibility check, performance audit, or technical quality review.

Security81.9K installs

competitors

coreyhaines31/marketingskills

When the user wants to create competitor comparison or alternative pages for SEO and buyer-facing use. Also use when the user mentions 'alternative page,' 'vs page,' 'competitor comparison,' 'comparison page,' '[Product] vs [Product],' '[Product] alternative,' 'competitive landing pages,' 'how do we compare to X,' 'competitor teardown,' 'audit our competitor pages,' 'are our comparison pages out of date,' or 'competitive asset audit.' Use this for any content that positions your product against competitors. Covers four formats: singular alternative, plural alternatives, you vs competitor, and competitor vs competitor. For auditing existing claims (not researching competitors from scratch, which is competitor-profiling; not technical SEO on these pages, which is seo-audit), use the asset audit here. For internal battle cards and sales-specific competitor docs, see sales-enablement.

Security79.3K installs