The instrument problem: measuring an agent you intend to let act
Two SOC agents can have the same accuracy and make very different mistakes when allowed to act. A short guide to measuring how often they act and how often those actions are wrong.
By the SIRP team
In Accuracy is not calibration, Faiz Shuja compares two agents that are each right on 900 of 1,000 alerts. One gives every alert the same confidence score. The other separates alerts into higher- and lower-confidence groups. At a 95% threshold, the first acts on none. The second acts on 750, gets 30 of those actions wrong, and defers the other 250. These are illustrative numbers from the paper.
Both systems can be well calibrated because their confidence scores match how often they are right in each group. The difference is resolution, the ability to separate easier cases from harder ones. That separation gives an analyst something to work with when deciding which cases to automate.
The paper proposes measuring two quantities together. Coverage is the share of cases on which the agent acts. Risk is the share of those actions that are wrong. Reporting both shows what happens as the threshold changes. The paper also walks through how evaluation can mislead teams and how to test a proposed change in shadow mode before using it to make decisions.
Download the full paper by Faiz Shuja (PDF)