Last reviewed:
What is explainable AI (XAI)? Definition, methods and legal obligations
Explainable AI (XAI) covers the methods that make it possible to understand why an AI system produced a given result, and to explain it to the person concerned. In Europe it is also a legal requirement: since the CJEU judgment of 27 February 2025, a person subject to an automated decision can demand an explanation of the procedure and principles actually applied.
Two related notions are often confused. Interpretability is the ability to understand how a model works internally: a decision tree or a scorecard can be read directly. Explainability is the ability to justify a specific result in understandable terms, including for an opaque model such as a neural network. For black-box models, two methods dominate. LIME (Ribeiro, Singh and Guestrin, 2016) locally approximates the model with a simple one around a decision. SHAP (Lundberg and Lee, 2017) assigns each variable its contribution to the result, for example “length of employment weighed -12 points”. For large language models the topic is still a research field: in March 2025 Anthropic published “circuit tracing” work that follows some of a model's internal steps. The same team showed in April 2025 that the reasoning displayed by a reasoning model is not a reliable explanation: Claude 3.7 Sonnet mentioned a hint it had actually used only 25% of the time on average. On the legal side, Article 22 of the GDPR governs fully automated decisions with legal effects. In the Dun & Bradstreet Austria judgment (C-203/22, 27 February 2025), the CJEU specified that the person may demand a “concise, transparent, intelligible” explanation of the procedure and principles applied. Merely handing over an algorithm is not enough. The AI Act adds Article 13 (transparency of high-risk systems towards deployers) and Article 86 (right to an explanation of an individual decision), applicable to Annex III systems from 2 December 2027 following the Omnibus.
Concrete example
Real case, at the origin of the 2025 ruling: in Austria, a mobile operator refused a customer a 10-euro monthly contract on the basis of a creditworthiness score computed automatically by Dun & Bradstreet Austria. The customer asked to understand. The company invoked trade secrets. The CJEU ruled that the company must explain which data were used and how, in an understandable way, and that if it considers this reveals a trade secret, it must send the information to the supervisory authority or the court, which strikes the balance.
Illustrative case: a French consumer credit company uses a scoring model. For each refusal, the advisor sees the three factors that weighed most (computed with SHAP) and a standard sentence: “debt ratio above 35%, less than one year in current job”. The customer can contest and an analyst reviews the file. The company thus meets Article 22 of the GDPR and prepares for Article 86 of the AI Act.
Comparison
| Interpretability | Explainability | |
|---|---|---|
| Question asked | How does the model work? | Why this result for this case? |
| Scope | The whole model | One specific decision |
| Example methods | Scorecard, decision tree, linear regression; circuit tracing for LLMs (research) | SHAP, LIME, counterfactuals (“had your income been X, the answer would have changed”) |
| Audience | Data scientists, auditors | Person concerned, advisor, regulator |
| Reference text | AI Act, Article 13 (transparency towards the deployer) | GDPR, Articles 15 and 22; AI Act, Article 86 |
FAQ
What is explainable AI?
It is the set of methods that make it possible to understand and justify the output of an AI system: which factors weighed, in which direction, and how much. The goal is that a human (user, customer, regulator) can check a decision and challenge it.
What is the difference between explainability and interpretability?
Interpretability concerns the model itself: you understand how it works by reading it (scorecard, decision tree). Explainability concerns a result: you justify a specific decision, even when the model is opaque, for example with SHAP or LIME.
What are the main explainable AI methods?
The most widely used are SHAP, which assigns each variable its contribution to the result, and LIME, which approximates the model with a simple one around a decision. For large language models, interpretability methods (circuit tracing) are still research.
Is AI explainability mandatory?
Yes, in several cases. The GDPR (Articles 15 and 22) requires explaining a fully automated decision with significant effects, as the CJEU clarified on 27 February 2025. The AI Act requires transparency of high-risk systems (Article 13) and a right to explanation (Article 86), from 2 December 2027 for Annex III.
Do you have to disclose the algorithm to explain a decision?
No. According to the CJEU (Case C-203/22), merely communicating a formula or an algorithm is not a sufficient explanation. You must describe the procedure and principles actually applied, in an understandable way. A trade secret can be invoked, but the authority or the court then strikes the balance.
Is a reasoning model that shows its thinking explainable?
Not really. Anthropic showed in April 2025 that reasoning models often leave out elements they used: Claude 3.7 Sonnet mentioned a hint it was given 25% of the time on average, DeepSeek R1 39%. Displayed reasoning is not proof of the actual reasoning.
See also
Further reading
Reasoning models don't always say what they think, Anthropic, 3 April 2025
Sources
- Judgment of the Court of Justice of the European Union (First Chamber) of 27 February 2025, CK v Magistrat der Stadt Wien, Dun & Bradstreet Austria GmbH, Case C-203/22. https://curia.europa.eu/juris/liste.jsf?num=C-203/22
- Regulation (EU) 2024/1689 (AI Act), Articles 13 and 86, and Regulation (EU) 2026/1744 (Digital Omnibus on AI) for the timeline, EUR-Lex. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- Reasoning models don't always say what they think, Anthropic Alignment Science, 3 April 2025. https://www.anthropic.com/research/reasoning-models-dont-say-think
- A Unified Approach to Interpreting Model Predictions (SHAP), Lundberg and Lee, NeurIPS 2017, arXiv:1705.07874. https://arxiv.org/abs/1705.07874