We Audit AI Agents. Can Someone Hack Our Verifiers?

Over the past month, we've audited 25+ AI agent frameworks — from Coinbase AgentKit to browser-use to Microsoft UFO. We've found real CVEs: prompt injection leading to remote code execution, wallet drains, unauthorized tool access.

Then we asked the uncomfortable question: Can someone do the same thing to us?

ThoughtProof verifies AI outputs using multiple independent models. A claim goes through generators, a critic, and a synthesizer — each running on different LLMs. The theory is that injecting one model doesn't help when three others are checking it.

But theory isn't proof. So we attacked our own pipeline.

The Attack Vector

The theoretical attack is straightforward: embed prompt injection in the content being verified. If a user submits a claim like:

In a single-agent system, this is game over. The model reads the injected instruction, follows it, and outputs a compromised result. We've seen this exact pattern in real-world agent exploits.

Why Multi-Model Makes This Harder

In ThoughtProof's pipeline, this injection has to fool every model independently:

That's 5 independent models across 4 different providers. Each has different system prompts, different training data, different injection resistance. A payload that works on GPT-4o might fail on DeepSeek. One that works on DeepSeek might get flagged by the Anthropic critic.

Our Dissent Preservation Ratio (DPR) metric catches exactly this: if generators suspiciously agree when they shouldn't, the score drops and a false-consensus flag fires.

But "Harder" Isn't "Impossible"

We don't claim invulnerability. Honest assessment:

What an attacker CAN'T do: Execute code, access tools, drain wallets, or pivot to other systems. Our verifiers have zero tool access. The worst case is a wrong confidence score — bad, but not catastrophic.

What an attacker COULD theoretically do: Craft a model-universal injection that subtly biases all verifiers toward agreement. This is ~10x harder than single-agent injection, but not theoretically impossible. Especially with universal jailbreaks that work across model families.

The Fix: Anthropic's Sectioning Pattern

Anthropic's "Building Effective Agents" guide recommends a specific pattern for this:

"Implementing guardrails where one model instance processes user queries while another screens them for inappropriate content. This tends to perform better than having the same LLM call handle both."

The insight: don't ask the verifier to detect injections. Use a separate model whose only job is injection detection. This is the Sectioning variant of their Parallelization pattern.

We implemented this as the Injection Guard — a pre-verification scanner that runs before the pipeline:

Key design decisions:

Separate model. The guard runs on a different model than the verifiers. It can't be influenced by the same payload that targets the pipeline.

Never blocks. The guard flags and caps confidence — it doesn't reject. Verification still runs for transparency. You can see what the pipeline would have said.

Cheap and fast. Uses the smallest available model (Grok, DeepSeek, GPT-4o-mini). Adds ~1-2 seconds, costs fractions of a cent.

Layered with static scanning. We already had regex-based pattern detection (security.ts) for classic injection phrases. The LLM guard catches the subtle stuff that regex misses — paraphrased instructions, encoded payloads, social engineering.

Defense in Depth

ThoughtProof now has three layers of injection defense:

Each layer has a different failure mode. Static scan misses paraphrased attacks. LLM guard might miss novel techniques. Multi-model diversity might fail against universal jailbreaks. But the probability of all three failing simultaneously is vanishingly small.

What We Learned

1. Audit yourself first. It's easy to find vulnerabilities in other people's code. The real test is turning that lens inward. We found a gap — and we closed it.

2. Separation of concerns isn't just architecture — it's security. The guard model and the verifier model have different jobs, different prompts, different attack surfaces. That's the whole point.

3. Multi-model verification isn't just about accuracy — it's about resilience. This was Anthropic's insight with the Parallelization pattern, and it's what we've been building for weeks. Nice to see it validated.

4. Ship the fix with the disclosure. We found the gap, built the guard, and shipped pot-sdk@1.0.0 — all in the same session. That's how it should work.

Try It