What is AI red teaming?

AI red teaming is the practice of deliberately attacking your own AI system — with adversarial prompts, jailbreaks and edge-case inputs — to find where it breaks before someone else does. It borrows the name and the mindset from security red teaming, applied to model behaviour rather than a network perimeter.

What a red team looks for

• Jailbreaks — getting the model past its safety training (the "DAN" family and successors)
• Prompt injection — direct and indirect
• Data leakage — extracting training data or the system prompt
• Harmful output — toxicity, dangerous instructions, misinformation stated confidently
• Excessive agency — an agent taking an action it should have refused

Manual vs automated

Manual red teaming — a human probing creatively — finds novel failure modes. Automated scanners run a large fixed battery of known attacks on every model change, which is what makes them useful in CI. NVIDIA Garak is the common open-source automated scanner; frameworks like Promptfoo do similar work. Most serious deployments use both.

Why it is now standard

NIST's AI Risk Management Framework and the OWASP LLM Top 10 both treat adversarial testing as expected practice, not optional. If a model reaches production without it, the first red team is your users.