# What is AI red teaming? Definition and business implications

> Red teaming, borrowed from cybersecurity, is the practice of testing an AI system by simulating adverse usage attempts: rule bypassing, sensitive-data extraction, forbidden-content generation. It aims to identify vulnerabilities before a malicious actor exploits them in production.

## Definition
AI red teaming combines three approaches. Manual red teaming: experts (often from cybersecurity, linguistics, law backgrounds) try to derail the model with elaborate prompts, role-plays, and guardrail bypasses. Automated red teaming: attack models generate thousands of adversarial prompts to explore the vulnerability space. Bug bounty programmes: external researchers, paid in proportion to flaws found, test the system continuously. Since 2024, red teaming has become a documentary standard for leading laboratories. Anthropic publishes its Responsible Scaling Policy, rewritten in February 2026, and for each new model a system card detailing internal and external adversarial testing. Public AI evaluation institutes (UK AI Security Institute, US Center for AI Standards and Innovation) perform external red teaming on frontier models before deployment. The European AI Act requires documented adversarial testing for general-purpose AI models with systemic risk (article 55), an obligation that has applied since August 2025 and that the Commission's AI Office has been able to enforce with fines since 2 August 2026.

## Example
In May 2025, Anthropic activated its ASL-3 protections for the launch of Claude Opus 4: classifiers that block a narrow class of content related to chemical, biological, radiological and nuclear (CBRN) weapons, a bug bounty programme dedicated to bypassing them, and jailbreak monitoring. In September 2025, the company published the results of adversarial testing of these safeguards by the US Center for AI Standards and Innovation (CAISI) and the UK AI Security Institute, on Claude Opus 4 and 4.1 before deployment. The government teams found several vulnerabilities, including prompt injections and attacks using ciphers or character substitution. Anthropic patched them and restructured its safeguards. These models are now retired from Anthropic's API, but the practice has continued: before Claude Fable 5.1 launched in September 2026, Trajectory Labs spent about 74 hours and over 6,500 requests trying to make it produce exploits, without finding a universal jailbreak.

## For a leader
Four clauses to require in any contract with an AI vendor for stakes-bearing systems (legal, financial, medical, HR). First, the obligation to provide a pre-deployment red-teaming report for each major version of the model, with an anonymised list of identified and corrected vulnerabilities. Second, the minimum frequency of continuous red teaming during the contract: quarterly for critical cases, semi-annually for others. Third, transparency on the third parties involved (academic laboratories, public AI evaluation institutes, specialised firms): purely internal red teaming is insufficient. Fourth, the right to a counter-evaluation at your expense, by a third party of your choice, on your specific use cases. These clauses are not bespoke high-end add-ons: they are today accepted by serious market suppliers.

## FAQ

**What is AI red teaming?**
Borrowed from cybersecurity, it is the practice of testing an AI system by simulating adverse usage: rule bypassing, sensitive-data extraction, forbidden-content generation. The aim is to identify vulnerabilities before a malicious actor exploits them in production.

**What are the three red-teaming approaches?**
Manual red teaming, where experts (cybersecurity, linguistics, law) try to derail the model. Automated red teaming, where attack models generate thousands of adversarial prompts. Bug bounty programmes, where external researchers, paid in proportion to the flaws found, test the system continuously.

**Is red teaming a regulatory obligation?**
For general-purpose AI models with systemic risk, the European AI Act requires documented adversarial testing (article 55), an obligation that has applied since August 2025. Since 2024, red teaming has become a documentary standard for leading laboratories, and public AI evaluation institutes (UK AI Security Institute, US Center for AI Standards and Innovation) perform it on frontier models before deployment.

**Which clauses should you require from an AI vendor?**
Four, for stakes-bearing systems. A pre-deployment red-teaming report for each major version, with an anonymised list of corrected vulnerabilities. A minimum frequency of continuous red teaming (quarterly for critical cases, semi-annually otherwise). Transparency on the third parties involved. The right to a counter-evaluation at your expense, by a third party of your choice, on your own use cases.

**Is internal red teaming enough?**
No. Purely internal red teaming is insufficient. Transparency on the third parties involved (academic laboratories, public AI evaluation institutes, specialised firms) is a clause to require. These requirements are today accepted by serious market suppliers.

## See also

- [Guardrails](https://wyp.agency/en/glossary/guardrails/)
- [Alignment](https://wyp.agency/en/glossary/alignment/)
- [Algorithmic bias](https://wyp.agency/en/glossary/algorithmic-bias/)
- [Hallucination](https://wyp.agency/en/glossary/hallucination/)
- [AI Act](https://wyp.agency/en/glossary/ai-act/)
- [LLM evaluation (evals, LLM as a judge)](https://wyp.agency/en/glossary/llm-evaluation/)
- [Prompt injection](https://wyp.agency/en/glossary/prompt-injection/)

## Sources

- Strengthening our safeguards through collaboration with US CAISI and UK AISI, Anthropic, September 2025 (https://www.anthropic.com/news/strengthening-our-safeguards-through-collaboration-with-us-caisi-and-uk-aisi)
- AI Safety Level 3 Deployment Safeguards Report, Anthropic, May 2025 (https://www.anthropic.com/asl3-deployment-safeguards)
- Activating AI Safety Level 3 protections, Anthropic, 22 May 2025 (https://www.anthropic.com/news/activating-asl3-protections)
- Regulation (EU) 2024/1689 on artificial intelligence (AI Act), article 55 (https://eur-lex.europa.eu/eli/reg/2024/1689/oj)
- System Card: Claude Fable 5.1 & Claude Mythos 5.1, Anthropic, 1 September 2026 (https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude%20Fable%205.1%20&%20Claude%20Mythos%205.1%20System%20Card.pdf)
- Responsible Scaling Policy, Anthropic (version 3.4, 8 July 2026) (https://www.anthropic.com/responsible-scaling-policy)

---
Entry from the WYP.agency AI glossary · https://wyp.agency/en/glossary/red-teaming/ · Last updated : 2026-10-14