How agents are stress-tested

A safety eval tests how a model or agent behaves under adversarial pressure, rather than how capable it is at a task. It measures resistance to jailbreaks and prompt injection, refusal of harmful requests, and safe use of tools. The tests come in two broad shapes: fixed benchmarks and live red-team exercises.

Red teaming is people, or other models, actively trying to break the system: jailbreak it, make it leak data, or turn a tool against its operator. It finds novel failures a fixed test set would never contain. Benchmarks give you a score you can compare; red teaming gives you coverage of the attacks nobody wrote a test for.

You feed it inputs crafted to override its instructions and measure how often the attack lands. The part that matters most is indirect injection, where the instruction hides in a web page, email or document the agent retrieves rather than in the user's message. A low attack-success rate is evidence of resistance, not proof of safety. OWASP ranks prompt injection as the top risk for LLM applications (OWASP).

No. It means the agent resisted the specific attacks in that test set, on the day it ran. New attacks, new tools and a different deployment context can all defeat it. Evals make regressions visible and narrow the risk; they do not remove it.

A mix of parties. Consortia such as MLCommons publish cross-model safety benchmarks, the frontier labs run their own internal red-team and safety evaluations, and academic and open-source groups build the adversarial frameworks. Who ran a test matters: a vendor grading its own model is weaker evidence than an independent one.