
ai-scanner-garak
AI model safety scanner built on NVIDIA garak for testing LLMs against 179 security probes across 35 vulnerability families
- Installs
- 878
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/reason-machines/trending-skills --skill ai-scanner-garak- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads ai-scanner-garak automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
AI Scanner (0din-ai/ai-scanner)
Skill by ara.so — Daily 2026 Skills collection.
AI Scanner is an open-source Ruby on Rails web application for AI model security assessments, wrapping NVIDIA garak with a multi-tenant UI, scheduling, PDF reports, and SIEM integration. It runs 179 community probes across 35 vulnerability families aligned with the OWASP LLM Top 10.
Installation
Quick Install (Docker)
curl -sL https://raw.githubusercontent.com/0din-ai/ai-scanner/main/scripts/install.sh | bash
Manual Install
curl -O https://raw.githubusercontent.com/0din-ai/ai-scanner/main/dist/docker-compose.yml
curl -O https://raw.githubusercontent.com/0din-ai/ai-scanner/main/.env.example
cp .env.example .env
Edit .env with required values:
# Generate a secure key
openssl rand -hex 64
# .env minimum required values
SECRET_KEY_BASE=<output_of_above_command>
POSTGRES_PASSWORD=<your_secure_db_password>
docker compose up -d
Access at http://localhost — default credentials: admin@example.com / password.
Change the default password immediately after first login.
Configuration (.env)
# Required
SECRET_KEY_BASE=<64-byte-hex>
POSTGRES_PASSWORD=<strong-password>
# Optional: custom port
PORT=8080
# Optional: SIEM integration
SPLUNK_HEC_URL=https://splunk.example.com:8088/services/collector
SPLUNK_HEC_TOKEN=$SPLUNK_HEC_TOKEN
RSYSLOG_HOST=syslog.example.com
RSYSLOG_PORT=514
# Optional: email
SMTP_HOST=smtp.example.com
SMTP_PORT=587
SMTP_USERNAME=$SMTP_USERNAME
SMTP_PASSWORD=$SMTP_PASSWORD
Core Concepts
| Concept | Description |
|---|---|
| Target | An AI system to test — API-based LLM or browser-based chat UI |
| Probe | A single attack test (e.g., prompt injection, data leakage) |
| Scan | A run of selected probes against a target |
| ASR | Attack Success Rate — percentage of probes that succeeded |
| Organization | Tenant boundary; users and scans are scoped per org |
Setting Up a Target
Targets define what you're scanning. Two types:
API-based LLM Target (e.g., OpenAI-compatible endpoint):
# In Rails console or via UI — representative model
target = Target.create!(
name: "Production GPT-4",
target_type: "api",
api_endpoint: "https://api.openai.com/v1/chat/completions",
api_key: ENV["OPENAI_API_KEY"],
model_name: "gpt-4",
organization: current_organization
)
Browser-based Chat UI Target:
target = Target.create!(
name: "Internal Chatbot UI",
target_type: "browser",
url: "https://chatbot.internal.example.com",
organization: current_organization
)
Running a Scan
Via UI
- Navigate to Targets → select your target
- Click New Scan
- Select probe families or individual probes
- Click Run Scan
Via Rails Console
# On-demand scan with specific probe families
scan = Scan.create!(
target: target,
probe_families: ["prompt_injection", "data_leakage", "insecure_output"],
organization: current_organization
)
ScanJob.perform_later(scan.id)
Scheduled Recurring Scan
# Weekly scan every Monday at 2am
scheduled_scan = ScheduledScan.create!(
target: target,
probe_families: ["prompt_injection", "jailbreak"],
cron_expression: "0 2 * * 1",
organization: current_organization
)
Probe Families (35 total, aligned to OWASP LLM Top 10)
Key probe families available:
# List all available probe families
Garak::ProbeRegistry.families
# => ["prompt_injection", "jailbreak", "data_leakage", "insecure_output",
# "supply_chain", "sensitive_info", "excessive_agency", "overreliance",
# "model_theft", "malicious_plugins", ...]
# Get probes within a family
Garak::ProbeRegistry.probes_for("prompt_injection")
# => 179 total probes across all families
Viewing Results
Attack Success Rate (ASR)
scan = Scan.find(scan_id)
puts scan.asr_score # => 0.23 (23% attack success rate)
puts scan.status # => "completed"
puts scan.probe_results.count # => 47
# Per-probe breakdown
scan.probe_results.each do |result|
puts "#{result.probe_name}: #{result.passed? ? 'SAFE' : 'VULNERABLE'}"
puts " Attempts: #{result.attempt_count}"
puts " ASR: #{result.asr_score}"
end
Trend Tracking
# Compare ASR across scan runs for a target
target.scans.completed.order(:created_at).map do |scan|
{ date: scan.created_at, asr: scan.asr_score }
end
PDF Report Export
# Generate PDF report for a scan
scan = Scan.find(scan_id)
pdf_path = ReportExporter.export_pdf(scan)
# Includes: executive summary, per-probe results, per-attempt drill-down
Via UI: Navigate to a completed scan → Export PDF.
SIEM Integration
Splunk
# config/initializers/siem.rb
SiemIntegration.configure do |config|
config.provider = :splunk
config.splunk_hec_url = ENV["SPLUNK_HEC_URL"]
config.splunk_hec_token = ENV["SPLUNK_HEC_TOKEN"]
config.forward_on_completion = true
end
Rsyslog
SiemIntegration.configure do |config|
config.provider = :rsyslog
config.rsyslog_host = ENV["RSYSLOG_HOST"]
config.rsyslog_port = ENV["RSYSLOG_PORT"].to_i
end
Multi-Tenant Organization Management
# Create a new organization
org = Organization.create!(name: "Security Team Alpha")
# Invite a user
user = User.invite!(
email: "analyst@example.com",
organization: org,
role: "analyst" # roles: "admin", "analyst", "viewer"
)
# Data is encrypted at rest per organization
org.encryption_key # => managed automatically
Development Setup
git clone https://github.com/0din-ai/ai-scanner.git
cd ai-scanner
cp .env.example .env.development
# Install dependencies
bundle install
# Database setup
rails db:create db:migrate db:seed
# Install garak (Python dependency)
pip install garak
# Start development server
bin/dev
Running Tests
# Full test suite
bundle exec rspec
# Specific area
bundle exec rspec spec/models/scan_spec.rb
bundle exec rspec spec/jobs/scan_job_spec.rb
# Lint
bundle exec rubocop
Common Patterns
Testing a New LLM Before Deployment
# Comprehensive pre-deployment scan
target = Target.create!(
name: "New Model v2 - Pre-deploy",
target_type: "api",
api_endpoint: ENV["NEW_MODEL_ENDPOINT"],
api_key: ENV["NEW_MODEL_API_KEY"],
model_name: "new-model-v2",
organization: current_organization
)
# Run all 35 probe families
scan = Scan.create!(
target: target,
probe_families: Garak::ProbeRegistry.families,
organization: current_organization
)
ScanJob.perform_now(scan.id)
if scan.reload.asr_score > 0.15
puts "WARNING: ASR #{scan.asr_score} exceeds threshold. Review before deploying."
else
puts "PASS: Model meets security threshold."
end
Using the Mock LLM for Testing Scanner Setup
The built-in Mock LLM lets you validate your scanner configuration without hitting real APIs:
target = Target.create!(
name: "Mock LLM",
target_type: "mock",
organization: current_organization
)
# Run a quick scan to verify everything works end-to-end
Webhook on Scan Completion
# config/initializers/scan_hooks.rb
ActiveSupport::Notifications.subscribe("scan.completed") do |_, _, _, _, payload|
scan = Scan.find(payload[:scan_id])
if scan.asr_score > 0.20
SlackNotifier.alert(
channel: "#security-alerts",
message: "High ASR detected: #{scan.asr_score} on #{scan.target.name}"
)
end
end
Docker Compose Production Tips
# docker-compose.override.yml — production additions
services:
web:
environment:
RAILS_ENV: production
FORCE_SSL: "true"
labels:
- "traefik.enable=true"
- "traefik.http.routers.scanner.rule=Host(`scanner.example.com`)"
- "traefik.http.routers.scanner.tls.certresolver=letsencrypt"
# Upgrade
docker compose pull
docker compose up -d
docker compose exec web rails db:migrate
Troubleshooting
| Problem | Solution |
|---|---|
| Scan stuck in "running" | Check docker compose logs worker — garak Python process may have crashed |
SECRET_KEY_BASE error on start | Run openssl rand -hex 64 and set in .env |
| Can't connect to target API | Verify API key env var is set; check firewall allows outbound from container |
| Browser target scan fails | Ensure Playwright/Chrome is available in the worker container |
| PDF export blank | Check wkhtmltopdf is installed in the web container |
| SIEM not receiving events | Verify SPLUNK_HEC_URL includes full path /services/collector |
# View all service logs
docker compose logs -f
# Check worker specifically (runs garak)
docker compose logs -f worker
# Rails console for debugging
docker compose exec web rails console
# Check garak is working
docker compose exec worker python -c "import garak; print(garak.__version__)"
Key Files (for Contributors)
app/
models/
scan.rb # Core scan model, ASR calculation
target.rb # Target types and validation
probe_result.rb # Per-probe result storage
jobs/
scan_job.rb # Async job that invokes garak
services/
garak_runner.rb # Ruby wrapper around garak CLI
report_exporter.rb
siem_integration.rb
lib/
garak/
probe_registry.rb # 179 probes, 35 families
dist/
docker-compose.yml # Production compose file
scripts/
install.sh # One-line installer
Resources
FAQ
What does the ai-scanner-garak skill do?
AI model safety scanner built on NVIDIA garak for testing LLMs against 179 security probes across 35 vulnerability families
How do I install ai-scanner-garak?
Run npx skills add https://github.com/reason-machines/trending-skills --skill ai-scanner-garak in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/ai-scanner-garak/ (personal) or .claude/skills/ai-scanner-garak/ (project).
Which AI agents can use ai-scanner-garak?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is ai-scanner-garak safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains ai-scanner-garak?
It is published by reason-machines in reason-machines/trending-skills. It has 878 recorded installs.
More from reason-machines/trending-skills
openclaw-control-center
reason-machines/trending-skills
Local-first, security-first control center for OpenClaw agents — visibility dashboard with readonly defaults, token attribution, collaboration tracing, and safe write operations.
inkos-multi-agent-novel-writing
reason-machines/trending-skills
Multi-agent CLI system for autonomous novel writing, auditing, and revision with human review gates
claude-hud-statusline
reason-machines/trending-skills
A Claude Code plugin that displays a real-time HUD showing context usage, active tools, running agents, and todo progress in your terminal statusline.
understand-anything-knowledge-graph
reason-machines/trending-skills
Turn any codebase into an interactive knowledge graph using Claude Code skills — explore, search, and ask questions about any project visually.
agency-agents-ai-specialists
reason-machines/trending-skills
A collection of specialized AI agent personalities for Claude Code, Cursor, Aider, Windsurf, and other AI coding tools — covering engineering, design, marketing, sales, and more.
agent-browser-automation
reason-machines/trending-skills
Headless browser automation CLI for AI agents using native Rust binary with Chrome DevTools Protocol
Popular security skills
azure-compliance
microsoft/azure-skills
Run Azure compliance and security audits with azqr plus Key Vault expiration checks. Covers best-practice assessment, resource review, policy/compliance validation, and security posture checks. WHEN: compliance scan, security audit, BEFORE running azqr (compliance cli tool), Azure best practices, Key Vault expiration check, expired certificates, expiring secrets, orphaned resources, compliance assessment.
minimal-run-and-audit
lllllllama/rigorpilot-skills
Rigor Run skill for README-first deep learning repo reproduction. Use when the task is specifically to capture or normalize evidence from the selected smoke test or documented inference or evaluation command and write standardized reprooutputs/ files, including patch notes when repository files changed. Do not use for training execution, initial repo intake, generic environment setup, paper lookup, target selection, hidden scientific-meaning changes, or end-to-end orchestration by itself.
seo-audit
coreyhaines31/marketingskills
When the user wants to audit, review, or diagnose SEO issues on their site. Also use when the user mentions "SEO audit," "technical SEO," "why am I not ranking," "SEO issues," "on-page SEO," "meta tags review," "SEO health check," "my traffic dropped," "lost rankings," "not showing up in Google," "site isn't ranking," "Google update hit me," "page speed," "core web vitals," "crawl errors," "indexing issues," "stuck on page 2," "striking distance keywords," "title tag rewrite," "low CTR," "local SEO," "Google Business Profile," or "map pack." Use this even if the user just says something vague like "my SEO is bad" or "help with SEO" — start with an audit. For building pages at scale to target keywords, see programmatic-seo. For adding structured data, see schema. For AI search optimization, see ai-seo.
firebase-security-rules-auditor
firebase/agent-skills
Audits Firebase (Firestore, Cloud Storage) security rules for vulnerabilities, privilege escalation, role bypasses, create vs update inconsistencies, resource exhaustion, type safety, size limits, and hasOnly ownership checks. Use when auditing/reviewing rules, running red-team rule assessments, or scoring against auditor checklists. Don't use for Firebase CLI (login, deploy), Auth, Crashlytics, Remote Config, or database queries.
audit
pbakaus/impeccable
Run technical quality checks across accessibility, performance, theming, responsive design, and anti-patterns. Generates a scored report with P0-P3 severity ratings and actionable plan. Use when the user wants an accessibility check, performance audit, or technical quality review.
competitors
coreyhaines31/marketingskills
When the user wants to create competitor comparison or alternative pages for SEO and buyer-facing use. Also use when the user mentions 'alternative page,' 'vs page,' 'competitor comparison,' 'comparison page,' '[Product] vs [Product],' '[Product] alternative,' 'competitive landing pages,' 'how do we compare to X,' 'competitor teardown,' 'audit our competitor pages,' 'are our comparison pages out of date,' or 'competitive asset audit.' Use this for any content that positions your product against competitors. Covers four formats: singular alternative, plural alternatives, you vs competitor, and competitor vs competitor. For auditing existing claims (not researching competitors from scratch, which is competitor-profiling; not technical SEO on these pages, which is seo-audit), use the asset audit here. For internal battle cards and sales-specific competitor docs, see sales-enablement.