ai-threat-testing
Offensive AI security testing and exploitation framework. Systematically tests LLM applications for OWASP Top 10 vulnerabilities including prompt injection, model extraction, data poisoning, and supply chain attacks. Integrates with pentest workflows to discover and exploit AI-specific threats.
- Installs
- 195
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/transilienceai/communitytools --skill ai-threat-testing- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads ai-threat-testing automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
AI Threat Testing
Test LLM applications for OWASP LLM Top 10 vulnerabilities using 10 specialized agents. Use for authorized AI security assessments.
Quick Start
1. Specify target (LLM app URL, API endpoint, or local model)
2. Select scope: Full OWASP Top 10 | Specific vulnerability | Supply chain
3. Agents deploy, test, capture evidence
4. Professional report with PoCs generated
Coverage — OWASP LLM Top 10, 2025 edition
Which file addresses which category is decided by
reference/catalog/llm-top10-2025.json, not by the
filename. The llmNN- prefixes on disk predate the 2025 renumbering and no longer match; the
content is correct, the labels were not. Cite an id only with its edition (LLM06:2025), because a
bare LLM06 means two different categories depending on which edition the reader assumes.
| Category | Attack surface |
|---|---|
LLM01:2025 Prompt Injection | Direct and indirect injection, instruction override, filter evasion |
LLM02:2025 Sensitive Information Disclosure | Training-data and cross-tenant RAG leakage, canary verification |
LLM03:2025 Supply Chain | Dependency CVEs, model provenance, malicious serialized models |
LLM04:2025 Data and Model Poisoning | Backdoor triggers, membership inference, behavioural anomalies |
LLM05:2025 Improper Output Handling | Code/XSS injection downstream, unsafe deserialization |
LLM06:2025 Excessive Agency | Tool/plugin abuse, privilege escalation, unauthorised actions — the category that matters for agents rather than chatbots |
LLM07:2025 System Prompt Leakage | Gap — no playbook yet. See the catalogue: what the prompt contains is a separate finding from whether it can be extracted |
LLM08:2025 Vector and Embedding Weaknesses | RAG injection, retrieval manipulation, embedding inversion |
LLM09:2025 Misinformation | Hallucination and confidence manipulation where output is relied upon |
LLM10:2025 Unbounded Consumption | Token flooding, cost impact, and model extraction/theft (2025 treats extraction-by-query as a consumption problem) |
Two classes are testable but are not OWASP categories, so they carry local TX- ids rather than
an invented LLMnn: monitoring evasion / forensic gaps, and adversarial perturbation of non-text
input. tools/test_llm_numbering.py enforces that separation.
Workflows
Full Assessment (4-8 hours):
- [ ] Reconnaissance
- [ ] Deploy all 10 agents
- [ ] Execute exploits
- [ ] Capture evidence
- [ ] Generate report
Focused Testing (1-3 hours):
- [ ] Select a category from the catalogue (LLM01:2025 .. LLM10:2025, or a TX- local class)
- [ ] Deploy agent
- [ ] Execute techniques
- [ ] Document findings
Supply Chain Audit (2-4 hours):
- [ ] Inventory dependencies
- [ ] Scan CVEs
- [ ] Test plugins/APIs
- [ ] Verify model provenance
Integration
Enhances /pentest with AI-specific testing:
- Traditional pentesting + AI threat testing = complete security assessment
- Chain vulnerabilities across traditional and AI vectors
- Unified reporting with CVSS scores
Key Techniques
Prompt Injection: Instruction override, system prompt extraction, filter evasion
Model Extraction: Query sampling, token analysis, membership inference
Data Poisoning: Behavioral anomalies, backdoor triggers, bias analysis
DoS: Token flooding, recursive expansion, context exhaustion
Supply Chain: CVE scanning, plugin audit, model verification
MCP Tool Abuse: MCP server inspectors/debuggers often expose /api/mcp/connect or similar endpoints that accept serverConfig with arbitrary command parameters — unauthenticated RCE. Check for MCP Inspector, MCP Playground, or any MCP debugging UI on non-standard ports (6274, 3000, etc.).
Evidence Capture
All agents collect: screenshots, network logs, API responses, errors, console output, execution metrics.
Reporting
Automated reports include: executive summary, detailed findings (CVSS scores), PoC scripts, evidence, remediation guidance.
Critical Rules
- Written authorization REQUIRED before testing
- Never exceed defined scope
- Test in isolated environments when possible
- Document all findings with reproducible PoCs
- Follow responsible disclosure practices
Integration
- Integrates with
/pentestskill for comprehensive security testing - AI-specific vulnerability knowledge in
/AGENTS.md - Attack playbooks in
reference/llm0X-*.md
FAQ
What does the ai-threat-testing skill do?
Offensive AI security testing and exploitation framework. Systematically tests LLM applications for OWASP Top 10 vulnerabilities including prompt injection, model extraction, data poisoning, and supply chain attacks. Integrates with pentest workflows to discover and exploit AI-specific threats.
How do I install ai-threat-testing?
Run npx skills add https://github.com/transilienceai/communitytools --skill ai-threat-testing in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/ai-threat-testing/ (personal) or .claude/skills/ai-threat-testing/ (project).
Which AI agents can use ai-threat-testing?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is ai-threat-testing safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains ai-threat-testing?
It is published by transilienceai in transilienceai/communitytools. It has 195 recorded installs.
More from transilienceai/communitytools
attack-path-stitcher
transilienceai/communitytools
Stitches confirmed single-asset findings into multi-hop attack paths across the organization. Builds a graph where nodes are assets and edges are confirmed exploit hops citing the findings that enable them.
blockchain-security
transilienceai/communitytools
Smart contract security testing and blockchain CTF exploitation. Covers Solidity vulnerability analysis, EVM storage manipulation, delegatecall attacks, CREATE/CREATE2 address prediction, and common DeFi exploit patterns. Use when analyzing Solidity contracts, solving blockchain challenges, or testing smart contract security.
hackerone
transilienceai/communitytools
HackerOne bug bounty automation - parses scope CSVs, deploys parallel pentesting agents per asset, validates PoCs, and generates platform-ready submission reports.
osint
transilienceai/communitytools
Open-source intelligence gathering - company repository enumeration, secret scanning, git history analysis, employee footprint, and code exposure discovery.
reconnaissance
transilienceai/communitytools
Domain assessment and web application mapping - subdomain discovery, port scanning, endpoint enumeration, API discovery, and attack surface analysis.
cve-poc-generator
transilienceai/communitytools
Open the skill to read its SKILL.md instructions.
Popular security skills
azure-compliance
microsoft/azure-skills
Run Azure compliance and security audits with azqr plus Key Vault expiration checks. Covers best-practice assessment, resource review, policy/compliance validation, and security posture checks. WHEN: compliance scan, security audit, BEFORE running azqr (compliance cli tool), Azure best practices, Key Vault expiration check, expired certificates, expiring secrets, orphaned resources, compliance assessment.
minimal-run-and-audit
lllllllama/rigorpilot-skills
Rigor Run skill for README-first deep learning repo reproduction. Use when the task is specifically to capture or normalize evidence from the selected smoke test or documented inference or evaluation command and write standardized reprooutputs/ files, including patch notes when repository files changed. Do not use for training execution, initial repo intake, generic environment setup, paper lookup, target selection, hidden scientific-meaning changes, or end-to-end orchestration by itself.
seo-audit
coreyhaines31/marketingskills
When the user wants to audit, review, or diagnose SEO issues on their site. Also use when the user mentions "SEO audit," "technical SEO," "why am I not ranking," "SEO issues," "on-page SEO," "meta tags review," "SEO health check," "my traffic dropped," "lost rankings," "not showing up in Google," "site isn't ranking," "Google update hit me," "page speed," "core web vitals," "crawl errors," "indexing issues," "stuck on page 2," "striking distance keywords," "title tag rewrite," "low CTR," "local SEO," "Google Business Profile," or "map pack." Use this even if the user just says something vague like "my SEO is bad" or "help with SEO" — start with an audit. For building pages at scale to target keywords, see programmatic-seo. For adding structured data, see schema. For AI search optimization, see ai-seo.
firebase-security-rules-auditor
firebase/agent-skills
Audits Firebase (Firestore, Cloud Storage) security rules for vulnerabilities, privilege escalation, role bypasses, create vs update inconsistencies, resource exhaustion, type safety, size limits, and hasOnly ownership checks. Use when auditing/reviewing rules, running red-team rule assessments, or scoring against auditor checklists. Don't use for Firebase CLI (login, deploy), Auth, Crashlytics, Remote Config, or database queries.
audit
pbakaus/impeccable
Run technical quality checks across accessibility, performance, theming, responsive design, and anti-patterns. Generates a scored report with P0-P3 severity ratings and actionable plan. Use when the user wants an accessibility check, performance audit, or technical quality review.
competitors
coreyhaines31/marketingskills
When the user wants to create competitor comparison or alternative pages for SEO and buyer-facing use. Also use when the user mentions 'alternative page,' 'vs page,' 'competitor comparison,' 'comparison page,' '[Product] vs [Product],' '[Product] alternative,' 'competitive landing pages,' 'how do we compare to X,' 'competitor teardown,' 'audit our competitor pages,' 'are our comparison pages out of date,' or 'competitive asset audit.' Use this for any content that positions your product against competitors. Covers four formats: singular alternative, plural alternatives, you vs competitor, and competitor vs competitor. For auditing existing claims (not researching competitors from scratch, which is competitor-profiling; not technical SEO on these pages, which is seo-audit), use the asset audit here. For internal battle cards and sales-specific competitor docs, see sales-enablement.