The coming AI security crisis (and what to do about it) | Sander Schulhoff

I’ve uncovered some big problems in AI security, starting with a hard truth: guardrails don’t actually stop determined attackers. If someone really wants to jailbreak a top model, they’ll slip past those checks.

Today’s guest is Sander Schulhof, a leading voice on adversarial robustness, and this conversation is serious but overdue. The core idea is that today’s AI systems can be tricked, the risk rises fast with agents and robots, and the goal here is clearer understanding, not fear.

Sander, set the stage: what are we talking about, and why now?

We’re digging into AI security: prompt injection, jailbreaks, indirect prompt injection, AI red teaming, and some uncomfortable realities in the AI security industry.

Give folks your quick background and why your lens on this is unique.

I’m an AI researcher focused on prompting and red teaming; I published the first major guide on prompting and ran the first—and now largest—genAI red teaming competition. We open‑sourced a massive prompt‑injection dataset used across frontier labs and big enterprises, and through years of testing I’ve seen guardrails fail repeatedly.

Define jailbreaks versus prompt injection, and ground it with a simple example.

Jailbreaking is you versus the model, where you trick it into doing something it shouldn’t. Prompt injection targets an app’s system prompt, like telling a story bot to ignore its instructions and output bomb‑making steps instead.

A fresh example: a researcher showed how ServiceNow’s agent could recruit more powerful internal agents to read and modify a database and email out sensitive data. It turned a benign workflow into a coordinated attack.

That might be the first case I’ve seen with clear real‑world damage from an agentic chain, which is telling.

Share a few more examples so people grasp how this shows up in the wild.

A company’s Twitter bot was coaxed into posting threats after users told it to ignore its policy, and a math site that executed model‑written code had its secrets exfiltrated because the code ran on the same server. Both were prompt injections against the app’s instructions.

On the jailbreak side, a bombing plot was reportedly planned with a chat model, and a recent code‑assistant attack bypassed protections by splitting a malicious request into smaller, innocent‑looking steps.

The stakes rise with agents and robots, where AI has real power. Walk through that risk.

Chatbot harm is mostly reputational, but agents can leak data, move money, or act in the world, and vision‑language robots can be manipulated with spoken or visual prompts. We’ve already seen robotic systems get jailbroken.

A lot of companies are trying to help with this; paint the industry map and what actually exists.

Frontier labs do deep research, while vendors sell monitoring, compliance, automated red teaming, and guardrails. The first two help; the last two are where I see the biggest gaps.

How do automated red teaming and guardrails work in practice?

Automated red teaming uses models to generate attacks that elicit harmful outputs. Guardrails are classifiers placed before and after your model to block suspicious inputs and outputs.

That sounds sensible on paper. What’s the fundamental problem?

Automated red teaming always finds something because these model families share weaknesses, so you mostly learn what everyone already knows. The bigger issue is that guardrails don’t hold up against skilled, adaptive attackers.

The attack space is effectively infinite, so claims like catching ninety‑nine percent are based on tiny, unrepresentative samples. When we ran adaptive evaluations with top labs, humans broke every defense in a handful of tries, while automated attackers needed more attempts but still succeeded widely.

Guardrails don’t meaningfully deter determined actors; if you can beat the base model, you can route around the wrapper. Some vendors oversell results, miss non‑English attacks, or publish stats that don’t stand up.

If frontier labs haven’t solved adversarial robustness after years of work, it’s unrealistic to expect an enterprise tool to crack it. And prompt‑based defenses inside the system prompt are even weaker.

Given that, why aren’t labs pouring more into security right now?

Incentives favor capability, and many agent tasks still outstrip current models’ competence, so companies push for smarter systems first. You can patch a bug, but you can’t patch a brain, which makes reliability a different beast.

Alright, a CISO hears this and panics. What should they actually do?

If you’re running a simple FAQ or read‑only bot, your risk is mostly brand harm, and a guardrail won’t change that. Don’t spend on illusions of safety.

Treat any action the agent can take and any data it can see as potentially exposed because a user can coerce the model. Constrain permissions tightly and scope access so the model can’t harm anything you care about.

This is where AI security meets classical security; you need people fluent in both. For example, a code‑generation service should execute model‑written code in a container and sanitize outputs before they touch your app.

This feels like alignment work for security teams: keep the ‘god’ in the box. Does that map to research?

Yes, the control subfield asks whether we can harness a hostile model without letting it cause harm, which connects to the whole probability‑of‑doom discussion.

Is there value in layering lots of defenses just to add friction?

Stacking guardrails is operationally heavy and doesn’t buy real security; log inputs and outputs for observability, but invest your effort in system design and permissioning.

What about agents that read and act on data, like email or the web?

Agents are easier to mislead through indirect prompt injection, whether in your inbox or on a webpage, and we’ve already seen a browser agent leak account data by following crafted instructions. This is where harm shows up fast.

A promising approach is CAMEL, which assigns only the permissions a task truly needs, like read‑only for summarizing or write‑only for sending a drafted message. It limits damage but can’t help when a single task legitimately needs both read and write.

So CAMEL is a framework you implement, not a product off the shelf. Sounds like a practical guardrail on permissions rather than content.

Exactly, and depending on the application it can be a solid fit.

Beyond that, what else matters?

Education and hiring matter a lot; teams need to understand how these failures occur and design around them. Bring in someone who truly understands both AI behavior and system security.

You’re not selling software here, which I appreciate.

We actually want to discourage buying false comfort; better to build correctly than to bolt on marketing claims.

For foundation model teams listening, what would actually move the needle?

Progress on robustness has been minimal; static benchmarks are misleading, and adaptive human evaluations tell the real story. Explore adversarial training earlier in the training pipeline and new architectures, and invest more in real‑world, adaptive testing.

It’s easier to forbid dangerous knowledge than to police nuanced actions, which is why indirect prompt injection against agents remains unsolved. Capability gains alone haven’t delivered robustness.

Anyone doing this especially well you want to recognize?

Security teams at frontier labs are pushing hard, but this area needs more resources.

Looking ahead a year or so, what do you expect to happen?

I expect a market correction as buyers realize guardrails and automated red teaming don’t deliver durable safety, especially with strong open source alternatives around. Meanwhile, as agents gain real power, we’ll start seeing concrete harms show up in production.

Anything crucial we missed before we wrap?

Stop publishing new jailbreaks; we already know models can be broken, and it arms attackers more than it helps defenders. Big takeaway: guardrails don’t work, agents raise the stakes, and you need hybrid expertise and education to deploy safely.

Sander, thank you for going deep on a touchy topic; I think people will start seeing why this matters. Where can folks find you, and how can listeners be helpful?

I’m on Twitter at Sander Schulhof, and my ask is simple: think carefully before you ship, apply permissioning patterns like CAMEL when appropriate, and if you can’t bound the blast radius, don’t deploy. I’ll share a list of trustworthy AI security resources too.

Thanks again, Sander. And for listeners, please subscribe on your favorite app and check out past episodes at Lennyspodcast.com. See you next time.