Wiki · Concept · Last reviewed June 25, 2026

promptfoo

promptfoo is an open-source CLI, library, and AI-security platform for evaluating and red-teaming LLM applications with declarative tests, assertions, provider comparisons, trace-aware agent checks, and repeatable security evidence.

Category: Concept Published: June 25, 2026 Modified: June 25, 2026 Last reviewed: June 25, 2026 Tags: AI security, evaluations, red teaming, prompts, RAG, agents, promptfoo

Definition

promptfoo is an open-source command-line tool, library, and AI-security platform for evaluating and red-teaming LLM applications. The official documentation describes it as a way to test prompts, models, RAG pipelines, and applications with use-case-specific benchmarks, automated metrics, local execution, provider comparisons, and CI/CD integration. The GitHub README frames the same project around prompt, agent, and RAG testing; red teaming; vulnerability scanning; model comparison; declarative configuration; and CI/CD use.

The important word is application. promptfoo is not limited to asking whether a base model can answer a benchmark question. It can be aimed at the wrapper around the model: prompts, providers, retrieval systems, guardrails, API targets, tool calls, traces, code paths, and agent scaffolds. That makes it useful for teams that need regression and adversarial tests close to ordinary software changes.

It is also not a general compliance engine. A promptfoo run can produce evidence for a release gate, red-team report, procurement review, or incident follow-up, but the evidence is bounded by the target, configuration, assertions, plugins, strategies, judge model, attempt budget, and date of the run.

Current Context

As of June 25, 2026, promptfoo sits in a hybrid position: an open-source evaluation and red-team framework, a commercial AI-security product, and a project now connected to OpenAI. On March 9, 2026, OpenAI announced an agreement to acquire Promptfoo to strengthen agentic security testing and evaluation capabilities in OpenAI Frontier. Promptfoo's own announcement said the project would remain open source and continue to serve users and customers across providers and models.

The product surface has broadened beyond prompt comparison. Promptfoo's current site and documentation present red teaming, guardrails, model security, MCP proxy work, code scanning, and evaluations. The open-source docs still center the practical developer loop: declarative configs, local and CLI execution, provider comparisons, automated assertions, red-team plugins and strategies, GitHub Actions and CI/CD integration, and web-viewer result inspection.

This ownership context matters for source discipline. OpenAI's acquisition announcement is primary evidence about corporate direction, not independent proof that promptfoo finds every AI vulnerability. Promptfoo's docs are primary evidence for what the tool claims to support, but a team's own run logs are the primary evidence for whether a particular system was tested well.

How It Works

promptfoo evaluations are organized around prompts, test cases, providers, and assertions. Its documentation describes defining test cases, configuring prompts and API providers, running the CLI or library, and reviewing outputs in structured results or a web UI. Assertions can compare outputs against expected values or conditions, including deterministic checks, JSON checks, similarity, custom JavaScript or Python functions, and model-graded rubrics.

For red teaming, promptfoo's guide describes an automated loop: generate or curate adversarial inputs, run them through the LLM application, and evaluate outputs using deterministic or model-graded metrics. The same guide separates one-off reports from CI/CD integration. One-off runs can find urgent weaknesses. Continuous runs can make prompt and model changes visible before they silently alter deployed behavior.

promptfoo's vulnerability documentation lists plugin-backed categories for model and application risks, including prompt injection, jailbreaking, PII leakage, RAG testing, agent testing, guardrail testing, industry-specific policies, and coding-agent risks such as repository prompt injection, unsafe automation changes, secret handling, sandbox failures, network egress, delayed CI exfiltration, and verifier sabotage. Those categories are coverage options, not proof that every risk has been exhausted.

Assertions are the bridge between an example and a test. Deterministic assertions can check strings, regexes, JSON, SQL, XML, function-call schemas, latency, cost, trace spans, and agent trajectories. Model-assisted assertions can use rubrics, classifiers, moderation systems, factuality checks, answer relevance, context faithfulness, and goal-success judgments. Both forms need review: deterministic checks can be brittle, and model-graded checks can drift or inherit the judge model's blind spots.

Evaluation Boundary

A serious promptfoo result starts by naming the evaluated object. Testing a model endpoint is different from testing a product with retrieval, memory, tools, browser access, guardrails, tenant data, or a human approval flow. The promptfoo target should therefore be treated as part of the evidence, not just a transport detail.

The boundary should record the target URL or provider, model and provider versions where available, prompt templates, variables, retrieval corpus, tool definitions, available permissions, guardrail settings, red-team purpose, selected plugins, selected strategies, number of generated tests, judge provider, thresholds, and whether results came from black-box, component-level, or trace-aware testing.

For compliance-oriented runs, promptfoo can map tests to frameworks such as the OWASP LLM Top 10, OWASP Agentic Top 10, NIST AI RMF language, data-protection categories, and EU AI Act examples. Those mappings are useful for organizing evidence, but they do not by themselves prove legal compliance. The legal claim still depends on the deployed system, jurisdiction, documentation, human oversight, record-keeping, post-market monitoring, and organizational controls.

Agent Context

Agents make promptfoo especially relevant because an agent's risk is rarely confined to the sentence it returns. A coding agent can read a repository, call a shell, alter automation, or satisfy a feature request by introducing a vulnerability. A support agent can retrieve private records or overstep policy. A RAG assistant can leak context that was never meant to be displayed. promptfoo gives these behaviors a place in engineering practice: config, target, results, rerun.

This is also where promptfoo differs from a scanner that only probes a hosted model endpoint. A serious promptfoo run names the app boundary: target, provider, model, prompts, retrieval settings, available tools, assertions, plugins, strategies, traces, and evidence retention.

Promptfoo's agent documentation distinguishes black-box testing, component testing, and trace-based testing. Trace-based testing uses OpenTelemetry-style spans to show what the agent actually did: LLM calls, guardrail decisions, tool executions, shell commands, searches, reasoning steps, and errors. That matters because a final answer can look safe while the trajectory shows an unauthorized tool call, network request, or data access.

Governance and Safety

promptfoo evidence is strongest when it is treated as a test record rather than a safety certificate. Passing tests show that a specific system, under a specific configuration, satisfied chosen assertions. They do not show that the system is broadly safe, that a model is harmless, or that future traffic will stay inside the same distribution. Failing tests may reveal a real vulnerability, a bad policy, an unrealistic test, a weak judge, or a target integration bug.

For governance, that narrower evidence can still matter. Procurement teams can ask for configs and results. Release managers can block a prompt or model change when a critical assertion fails. Security teams can connect promptfoo outputs to vulnerability triage. Auditors can ask whether application-layer testing covered RAG exfiltration, prompt injection, excessive agency, and tool misuse rather than only base-model behavior.

The independent governance context is broader than promptfoo. NIST's Generative AI Profile recommends adversarial role-playing, generative-AI red teaming, and chaos testing to find anomalous or unforeseen failure modes. The EU AI Act requires providers of general-purpose AI models with systemic risk to perform model evaluations and document adversarial testing. OWASP's LLM and agentic lists give application-security categories. promptfoo can support those practices, but governance still requires owners, scope, remediation, monitoring, and authority to delay or restrict deployment.

Minimum Run Record

A promptfoo run should leave enough information for another reviewer to understand what was tested, reproduce or challenge the result where possible, and connect failures to remediation.

Defense Pattern

Failure Modes

YAML theater. A team adds a promptfoo config to the repository but never ties failures to release gates, owners, or remediation.

Judge collapse. A weak model-graded rubric marks harmful behavior as acceptable, or a provider update changes scoring without the team noticing.

Cached false confidence. CI reuses old completions or stale retrieval results and misses a live-model, prompt, or corpus regression.

Custom assertion risk. JavaScript or Python assertions can be useful, but they are executable code in the evaluation path and need normal code review, dependency review, and secret hygiene.

Coverage laundering. A report says "OWASP," "EU AI Act," or "NIST" because a preset was selected, even though the application boundary, human oversight, logging, and post-deployment controls were not tested.

Trace gap. An agent's final answer passes while the uninstrumented tool layer, shell command, retrieval query, or network request did the risky thing off-record.

Source Discipline

Claims about promptfoo should identify the surface being cited: official documentation, GitHub, the OpenAI acquisition announcement, a local configuration, a CI run, or a run result. A phrase like "tested with promptfoo" is incomplete unless it names the target, prompts, providers, assertions, plugins, strategies, attempt budget, judge choice, trace coverage, and run date. The red-team documentation warns that application architecture changes which vulnerabilities apply; a single-tenant chatbot, RAG system, coding agent, and multi-role enterprise assistant do not share the same risk surface.

Be careful with ownership and neutrality claims. Promptfoo remaining open source is a claim supported by Promptfoo's own announcement. Promptfoo being integrated into OpenAI Frontier is a claim supported by OpenAI's announcement. Neither claim proves independence of future defaults, breadth of provider support in every product tier, or adequacy of any specific evaluation.

Spiralist Reading

Spiralism reads promptfoo as a small bureaucratic machine for making language behavior accountable. It takes the informal act of "try a few prompts and see what happens" and turns it into a ledger: inputs, targets, assertions, scores, failures, reruns. The ritual is not mystical. Its value is that it forces a team to say what it expected, how it checked, and what broke when the machine answered back.

Open Questions

Sources

Return to Wiki