C6.3 Explainability

PMI / Cox [12]; risk-engineering.org [26]

The third component requires that every finding, rating, ranking or alert record the rule that produced it, the data it was computed from, the threshold it was compared with and the version of each, and that every simulation record its seed. The requirement comes from the critique of risk matrices, which showed that a ranking can be structurally wrong and insisted that it be reproducible so that it can be challenged [12], and from the engineering course's reproducibility habit [26]. It is the component that makes the other eleven auditable, and it is the one most often omitted because it is invisible when everything is going well.

The component reconciles a judgment with a computation. Most risk ratings are presented as judgments and are in fact computations whose rule nobody wrote down: a risk is amber because an analyst looked at a dashboard and felt that it was. An explainability note turns the feeling into a sentence an auditor can re-run: amber because this indicator breached this threshold on these dates, from this extract, under this version of the scale. Where an AI system contributed to a finding, the note names the system, its version and the test level at which it was last measured, which is how C4.7 and C6.3 meet. The note is short, and its length is not the point; its existence is.