Should you run Garak, Promptfoo, or book a test of the application?
Short version. Garak asks whether the model is vulnerable. Promptfoo asks whether the application is vulnerable. An authorization test asks whether a user or an agent can reach a record or take an action the rules forbid. Infrastructure is its own engagement. Observ’s $7,995 test is one application: the path from the person to the tools, checked against a permission matrix, with the pass measured at the tool. A chat refusal is one observation.
On 4 November 2026 a search for “garak vs promptfoo AI security assessment” returned comparison pages. They agree on the split: Garak for the model, Promptfoo for the application. They end on a pick. That split is the right description of the tools. It is the wrong end of the purchase. A team that can name one application needs the path from the person using it to the data and the tools, with a written rule for who may see what, and evidence taken where that rule is enforced.
Are Garak and Promptfoo competing products?
They answer two different questions. A buyer who operates an application needs both answers, and two more besides.
Garak is NVIDIA’s open probe kit. It sends known classes of input to a model or a model endpoint and records what comes back. It does not know your roles, your tenants, or which button is allowed to delete a customer. Promptfoo is a config-driven red team for an application: the prompt, the retrieval, the tools, and the API a user already calls. PyRIT, from Microsoft, sits in the same layer as Garak. It is another probe kit for a model. It is not a map of the company.
Promptfoo’s own guide to red teaming agents puts the permission check outside the model, and uses a trace to tell a refusal in the reply from a tool that actually ran. That is the useful reading of the tool. The export is still one layer.
What are the four layers of an assessment?
An independent assessment of an AI application has four questions. Two of them have a well-known open tool. Two of them do not.
| Layer | Question | Open tool that fits | What a pass looks like |
|---|---|---|---|
| Model | Is the model vulnerable to classes a probe library already knows? | Garak. PyRIT is the same kind of kit. | A probe run against the model or the model endpoint you named, with the hits kept |
| Application | Does this application fail when the test knows its prompt, its tools, and its data? | Promptfoo | A configured run against the application you ship, through its API |
| Authorization | Can a user or an agent reach data or an action outside the rules you wrote down? | Your permission matrix, run at the tool | A denied call at the tool or the API, for the role that must be denied |
| Infrastructure | Is the environment deployed the way the scope describes? | A host test, when that host is named | A separate engagement. See the host pentest |
What does an authorization test need first?
It needs the rules written down. The correct result depends on who is asking, and a generic scanner does not know that.
A scanner can ask an agent to fetch a record. It cannot know whether that fetch was supposed to succeed. For a company agent connected to a CRM, a useful matrix looks like this. The test checks these results.
| Situation | Expected result |
|---|---|
| A salesperson asks for another salesperson’s private records | Denied |
| A salesperson asks for finance reports | Denied |
| A finance user asks for finance reports | Allowed |
| The agent is asked to delete a customer | Allowed only for a role that has that permission, and only after the approval the matrix requires |
| Instructions in the conversation tell the agent to ignore permissions | Still denied |
| A call goes straight to an MCP tool | The server enforces the permission on its own |
The model agreeing is one observation. The pass is the tool, the API, or the MCP server denying the call for that role. An agent that says it cannot help, while the lookup behind it still returns the record, is a failure. The example on our agent page is the same shape: the order tool has to take the customer from the signed-in session, not from the sentence in the chat.
Before anyone generates a case, the scope has six answers:
- Who the users are.
- Which roles and permissions exist.
- Which data each role may see.
- Which actions the agent may take.
- Which actions need a person to approve them.
- Where the check runs: the model, the application, the tool, or the system behind the tool.
When those six are blank, the first part of the window goes to writing them with you. That writing is the part that does not transfer from the last customer to the next one. The run, the evidence, and the retest do.
What can be automated once the rules exist?
Discovery, and the cases that follow from a written matrix, can be repeated. Whether the matrix is the policy the business meant stays with a person.
| Work | What it covers | How it runs |
|---|---|---|
| Discovery | Endpoints, tools, MCP servers, APIs, and the authentication in front of them, inside the systems you named | Repeated from one engagement to the next |
| Policy tests | Each role, each resource, and the expected result, called at the real tool or API | Repeated once the matrix exists. The classes include reaching another user’s record, a higher role, another tenant, and an action that required approval |
| Judgement | Whether the permission model matches the business, which path the matrix missed, and what a failure costs | A person, on every engagement |
On an engagement we list the agents, tools, and data sources in the scope. We write the users, the roles, and the boundaries with you, and that becomes one permission matrix. We generate one case for each role and each resource, run the cases against the application you named, and keep the evidence, the failure, and the severity. The retest checks the rows you have fixed. That retest is the one already on the service page: requested inside 90 days, one pass over that report.
The application-specific part is the expected result in each row. The list, the run, the record of what happened, and the retest are the part a firm can reuse without starting from a blank page on every customer. A person still reads every finding before it ships. That is the same rule as the agent pentest.
What should the $7,995 cover?
One application, along the path from the user to the data and the tools. The architecture changes the matrix. The price stays one fee.
| What you have | What the matrix includes |
|---|---|
| An LLM application. A chat that answers questions. | Prompt injection, system-prompt extraction, sensitive disclosure, and output handling. Limits and cost, when the model is metered. |
| A RAG application. A chat that retrieves company documents. | The chat classes, plus whether one user can retrieve another user’s documents, and whether a document can steer the answer. |
| An agentic application. A system that takes actions through tools, APIs, or business systems. | The classes above, plus a tool call the role should not have, a chain of actions, an approval that was skipped, and a check at the tool itself. |
All three are the same product: one application, $7,995, in US dollars. The fee includes the report, a signed attestation letter, a debrief, and one retest of the findings you have fixed, if you ask inside 90 days. The classes we sign to are the ones on the service page: prompt injection, tool and permission misuse, data exposure, unsafe actions, output handling, and limits and cost, from the OWASP Top 10 for LLM Applications. The model provider is out of scope. A third party you do not control is out of scope.
A host, an API you want tested as its own application, or the cloud account around the agent, is the host pentest at $3,995 when you name that target. If the deployment and the access are in the signed scope, the two engagements run together. If they are not, the agent test stops at the application and the tools it calls.
Which of three budget lines “internal AI agent security” actually is, inventory or governance or a test of an agent you operate, is in the note on testing an agent you operate. How a booked test sits next to a continuous platform is in the note on AI pentests and booked tests.
Questions
Are Garak and Promptfoo competing products?
Garak probes a model or a model endpoint. Promptfoo tests an application that has a prompt, tools, or retrieval. An assessment of an application you operate uses both kinds of check, then tests authorization at the tool. A scanner export on its own is a scan of one layer.
Does the $7,995 agent test include the cloud account?
The $7,995 covers one AI application: the agent or LLM app, the tools it calls, and the data boundaries in the scope. A host or a cloud target is $3,995 when that target is named. The two can be signed together.
If the agent says it cannot do something, has the test passed?
The pass is a denied call at the tool, the API, or the server that holds the data, for a role the matrix says must be denied. The reply in the chat is one observation. A refusal in the chat with a successful lookup behind it is a finding.
What do you need ready before the cases mean anything?
Six answers in the scope: who the users are, which roles exist, which data each role may see, which actions the agent may take, which actions need approval, and where the check runs. Also written authorization, a test user for each role in the matrix, and a window. Testing starts within 10 business days of the signed scope.
Can you test a chatbot that does not call tools?
Yes. That architecture is an LLM application, and it is the same $7,995 fee. The matrix covers prompt injection, disclosure, and output handling, plus limits and cost when the model is metered. An agent that calls tools adds the permission rows, and those rows are checked at the tool.
Will you hand over the prompts or the scanner configuration?
We turn down requests for exploit code, or for prompts to use somewhere else. The report includes the transcript and the tool calls for each finding, and the fix.
Sources
- Garak, NVIDIA, GitHub. The model-layer probe kit named on this page. Read 4 November 2026.
- Promptfoo red teaming. The application-layer tool named on this page.
- How to red team LLM agents, Promptfoo. The page that puts permission checks outside the model and uses a trace to separate a reply from a tool call.
- PyRIT, Microsoft, GitHub. Another probe kit in the model layer. Named so that layer is not one product.
- OWASP Top 10 for Large Language Model Applications. The class list on this page is the one on our service page.
- Observ AI agent pentest. Price, classes, and what the fee includes.
- Observ host pentest. $3,995 for one host you name.
Author
Keith Vaughan founded Cipher Projects in Canberra in 2023. Observ is its pentesting service. The comparison pages tell you which scanner to install. This note is the purchase: one application, a matrix, evidence at the tool, and a person on the finding. The $7,995 is that test. Garak and Promptfoo are two tools a test of this kind can run.