AI Vulnerability Scanner: 8 Tools for Scanning AI and LLM Applications

TL;DR: AI vulnerability scanners test for prompt injection, jailbreaks, tool abuse, and other AI-specific vulnerabilities that traditional scanners miss entirely. This guide compares eight tools — open-source and commercial — across coverage, ease of use, and what they actually find. If you want to start with automated reconnaissance, ARTEMIS Recon maps your AI agent's attack surface in minutes, free, running inside your own Claude Code.

Why traditional vulnerability scanners do not work on AI applications#

Burp Suite does not find prompt injection. Nessus does not detect jailbreaks. OWASP ZAP does not test whether your AI agent can be manipulated into calling tools with attacker-controlled parameters.

This is not a gap in those tools — they were built for a fundamentally different attack surface. Traditional scanners test code: endpoints, parameters, headers, authentication flows. AI vulnerabilities live in the model layer: the interaction between the model, its instructions, its tools, and its context. The attack surface is natural language, not HTTP requests.

An AI application that passes every traditional security scan can still be completely vulnerable to:

• Prompt injection — overriding the system prompt through user input or retrieved content
• System prompt extraction — leaking the application's internal instructions and business logic
• Tool abuse — manipulating the AI into calling APIs with attacker-controlled parameters
• Guardrail bypass — circumventing content filters and safety boundaries through encoding, language switching, or multi-turn escalation
• Data exfiltration — extracting training data, user data, or retrieval-source data through the model's responses

You need both: traditional scanners for the infrastructure and web layers, AI-specific scanners for the model layer. This guide covers the AI-specific side.

What to look for in an AI vulnerability scanner#

Not all AI scanning tools test the same things. Before comparing tools, here is what matters:

Coverage breadth — does it test across all OWASP LLM Top 10 categories, or only prompt injection? Tools that only test prompt injection miss system prompt extraction, tool abuse, data exfiltration, and output integrity issues.

Agent awareness — can it test agentic systems with tool access, multi-step workflows, and memory? A tool built for chatbot testing will miss the highest-severity vulnerabilities in AI agents: tool abuse, privilege escalation, and cross-tool attack chains.

Attack sophistication — does it use static payloads (the same jailbreak template every time) or adaptive attacks (adjusting the approach based on the target's responses)? Static payloads catch the low-hanging fruit. Adaptive attacks find what a real attacker would find.

Multi-turn capability — can it run multi-turn conversations that gradually escalate, or only single-shot prompts? Many real-world attacks require several turns of conversation to set up.

Integration — can it run in CI/CD? Does it produce machine-readable output? Can you integrate it into your existing security workflow?

The 8 AI vulnerability scanners compared#

1. ARTEMIS Recon (Repello AI) — autonomous reconnaissance for AI agents#

ARTEMIS Recon takes a different approach from traditional scanners. Instead of running a fixed list of payloads, it uses an autonomous AI agent to map the target's attack surface — tools, APIs, prompt surface, trust boundaries, input modalities, and escalation paths.

What it does: Automated reconnaissance and attack-surface mapping for AI applications. It identifies what is testable, what tools are accessible, where untrusted input enters the system, and what the blast radius looks like if something is exploited.

How it works: You install the ARTEMIS Recon plugin in your Claude Code environment, point it at an AI target (web or API), and it runs an autonomous reconnaissance engagement. The output is a structured attack-surface map — not just "this is vulnerable" but "here is the complete topology of what an attacker can reach."

Best for: Security teams that need to understand their AI agent's attack surface before deciding what to test deeper. The free tier covers reconnaissance and attack-surface mapping; exploitation and end-to-end red teaming are in the ARTEMIS platform.

Pricing: Free (3 runs, work email signup). ARTEMIS platform for full red teaming is commercial.

2. Garak (NVIDIA) — open-source LLM vulnerability scanner#

Garak is the most widely used open-source AI vulnerability scanner. It works by sending a large library of adversarial prompts to the target model and analyzing the responses for policy violations.

What it does: Broad payload-based scanning across prompt injection, encoding bypasses, toxicity, hallucination, data leakage, and known jailbreak patterns. Ships with hundreds of pre-built "probes" (attack payloads) organized by vulnerability category.

How it works: Configure the target model endpoint, select which probes to run, execute. Garak sends each probe, analyzes the response using "detectors" (pattern matchers and classifiers), and produces a report with pass/fail per probe.

Best for: Teams with AI security expertise who want a comprehensive, customizable scanning tool. Excellent for catching known vulnerability patterns at scale. Less effective for novel attacks or multi-step exploitation chains.

Limitations: Single-shot prompts only — no multi-turn attack capability. Primarily targets model endpoints, not agentic systems with tool access. Requires significant configuration and interpretation expertise.

Pricing: Free, open-source (Apache 2.0).

3. pyRIT (Microsoft) — red teaming orchestration framework#

pyRIT (Python Risk Identification Toolkit) is Microsoft's framework for orchestrating multi-turn AI red teaming attacks. It is less a scanner and more a framework for building automated red team campaigns.

What it does: Multi-turn, multi-step attack orchestration. pyRIT uses an "attacker LLM" to generate adversarial prompts, sends them to the target, evaluates responses, and adjusts the attack strategy across turns. Supports crescendo attacks (gradually escalating), PAIR (Prompt Automatic Iterative Refinement), and custom attack strategies.

How it works: Define an attack objective, configure the target and attacker models, select an orchestration strategy, run. pyRIT manages the conversation loop: the attacker LLM generates a prompt → the target responds → a scorer evaluates the response → the attacker LLM adjusts for the next turn.

Best for: Security researchers and red teams who want to automate sophisticated multi-turn attacks. The orchestration framework is powerful for reproducing real-world attack patterns that single-shot scanners miss.

Limitations: Framework, not a product — requires Python expertise and significant setup. No built-in support for testing tool-calling agents. The attacker LLM adds cost (you are paying for two models per engagement).

Pricing: Free, open-source (MIT).

4. promptfoo — LLM evaluation with security presets#

promptfoo is primarily an LLM evaluation framework, but ships with security-focused test suites that make it effective as a lightweight AI vulnerability scanner.

What it does: Runs structured test suites against LLM endpoints. The built-in security presets cover prompt injection, jailbreaks, PII leakage, competitive information disclosure, and policy violations. Also supports custom test cases.

How it works: Define test cases in YAML (or use built-in presets), configure the target, run. promptfoo sends each test case, evaluates the response against expected behavior, and produces a detailed report with pass/fail and the actual responses.

Best for: Teams already using promptfoo for LLM evaluation who want to add security testing to the same pipeline. The CI/CD integration is mature, and the YAML-based test format is easy to extend.

Limitations: Single-shot test cases — no multi-turn attack orchestration. Security presets are a starting point, not a comprehensive vulnerability assessment. No agent or tool-calling awareness.

Pricing: Free, open-source (MIT). Cloud version available.

5. Counterfit (Microsoft) — adversarial ML attacks#

Counterfit is Microsoft's tool for adversarial machine learning attacks. It predates the current LLM security landscape and focuses on traditional ML attack vectors.

What it does: Evasion attacks, model inversion, membership inference, and data poisoning against ML models. Supports multiple ML frameworks (TensorFlow, PyTorch, scikit-learn).

How it works: Load a target model, select an attack method, configure parameters, run. Counterfit generates adversarial inputs designed to fool the model and reports success rates.

Best for: Teams testing traditional ML models (classifiers, object detection, recommendation systems) rather than LLMs. Relevant when your AI application includes non-LLM ML components.

Limitations: Not designed for LLM or agentic AI testing. No prompt injection, jailbreak, or tool abuse testing. Limited maintenance activity.

Pricing: Free, open-source (MIT).

6. ARTEMIS Platform (Repello AI) — autonomous end-to-end AI red teaming#

The ARTEMIS platform extends beyond the free Recon tier into full autonomous AI red teaming — not just mapping the attack surface but actively exploiting it.

What it does: End-to-end AI red teaming: reconnaissance, threat modeling, attack execution, and exploitation. ARTEMIS plans and executes multi-step attack chains autonomously, using a chain library built from real engagement experience.

How it works: Fully managed cloud platform. Configure the target, start the engagement, ARTEMIS handles the rest — reconnaissance, attack planning, execution, validation, and reporting. The output is a full penetration test report with severity ratings, reproduction steps, and remediation guidance.

Best for: Organizations that need comprehensive AI security testing without building an in-house red team. Particularly effective for agentic systems with complex tool configurations.

Limitations: Commercial product — not free beyond the Recon tier. Engagement results depend on target complexity and accessibility.

Pricing: Commercial. Start with ARTEMIS Recon free →

7. HiddenLayer — AI security platform#

HiddenLayer provides an enterprise AI security platform covering model scanning, runtime protection, and adversarial testing.

What it does: Model scanning for known vulnerabilities, supply chain security (detecting tampered or poisoned models), and runtime anomaly detection. Covers traditional ML models and increasingly LLMs.

How it works: Enterprise platform with API and dashboard. Integrates with model registries and deployment pipelines. Scans models for known vulnerability patterns, monitors inference traffic for anomalies.

Best for: Enterprise teams that need model supply chain security (scanning models from Hugging Face, internal registries) alongside runtime protection. Strongest in the traditional ML space.

Limitations: More focused on model-level security (supply chain, weights, inference monitoring) than application-level AI testing (prompt injection, tool abuse, guardrail bypass in deployed applications).

Pricing: Enterprise (contact sales).

8. Mindgard — AI threat intelligence and testing#

Mindgard offers AI-specific threat detection and automated security testing, with a focus on continuous monitoring.

What it does: Automated AI security testing across prompt injection, model evasion, and data extraction. Continuous monitoring for AI-specific threats in production.

How it works: SaaS platform. Configure the target AI application, Mindgard runs automated test campaigns and monitors production traffic for anomalies.

Best for: Organizations that need continuous AI security monitoring alongside periodic testing. The combination of testing and runtime monitoring covers both proactive and reactive security.

Limitations: Enterprise-focused — not suited for individual developers or small teams.

Pricing: Enterprise (contact sales).

Comparison matrix#

How to choose#

If you are just starting: Begin with ARTEMIS Recon to map your attack surface (free, takes minutes), then use Garak for broad payload scanning against the areas ARTEMIS identified as exposed.

If you have AI security expertise: Use pyRIT for sophisticated multi-turn attacks on your highest-risk endpoints, Garak for breadth, and promptfoo for continuous CI/CD scanning.

If you are testing AI agents (not just chatbots): ARTEMIS is currently the only tool built specifically for agentic AI with tool access, multi-agent topologies, and complex trust boundaries. Open-source tools test the model endpoint but miss the tool-calling, memory, and multi-step workflow attack surface.

If you need enterprise compliance: Look at ARTEMIS Platform, HiddenLayer, or Mindgard for managed platforms with reporting, audit trails, and integration support.

For a broader view of the AI security tooling landscape, see our guide to the best AI security tools and how these scanners fit into a complete AI security testing program.

Frequently asked questions#

What is an AI vulnerability scanner?

An AI vulnerability scanner is a tool that automatically tests AI applications — LLMs, chatbots, AI agents, and RAG pipelines — for security vulnerabilities specific to AI systems. Unlike traditional web vulnerability scanners that look for SQL injection or XSS, AI vulnerability scanners test for prompt injection, jailbreaks, system prompt extraction, guardrail bypasses, tool abuse, and data exfiltration through the model layer.

Can traditional vulnerability scanners find AI-specific vulnerabilities?

No. Traditional scanners like Burp Suite, Nessus, or OWASP ZAP test the web application and infrastructure layers — they cannot find prompt injection, jailbreak vulnerabilities, system prompt leakage, or tool abuse in AI agents. These vulnerabilities exist in the model layer and require AI-specific testing approaches. You still need traditional scanners for the infrastructure, but they are blind to the AI-specific attack surface.

What is the difference between an AI vulnerability scanner and AI red teaming?

AI vulnerability scanning is automated, broad, and fast — it tests known vulnerability patterns (prompt injection templates, jailbreak payloads, encoding bypasses) across the attack surface. AI red teaming is manual or semi-automated, deep, and creative — it discovers novel attack chains, multi-step exploits, and business-logic vulnerabilities that scanners miss. Most organizations need both: scanners for continuous coverage, red teaming for depth on high-value targets.

Are open-source AI vulnerability scanners good enough?

Open-source scanners like Garak and pyRIT are excellent for specific use cases — Garak for broad payload-based scanning, pyRIT for multi-turn attack orchestration. They require significant setup, maintenance, and expertise to run effectively. Commercial tools like ARTEMIS add autonomous attack planning, AI-agent-specific testing, managed infrastructure, and reporting. For teams without dedicated AI security expertise, commercial tools save weeks of setup time.

How often should you scan AI applications for vulnerabilities?

Every time the system prompt, tool configuration, retrieval sources, or model version changes. In practice, this means scanning after every deployment that touches the AI layer. Many teams run automated scans in CI/CD on prompt or config changes, plus a deeper manual red team assessment quarterly. Model provider updates (new Claude or GPT version) can change vulnerability behavior without any code change on your side.

Start with the attack surface#

The most common mistake in AI vulnerability scanning is running a tool before understanding what you are scanning. AI applications are not like web applications with well-defined endpoints — the attack surface includes implicit tool access, hidden retrieval sources, and emergent behaviors that change with every prompt.

ARTEMIS Recon maps that surface automatically. Point it at your AI application, get a structured attack-surface map in minutes. Then you know exactly where to focus your scanning — whether you use open-source tools, commercial platforms, or manual testing.

Three free runs. Work email signup. Runs inside your own Claude Code. Map your AI's attack surface →