Accuracy is not calibration: what changes when the SOC agent acts
Two SOC agents can be equally accurate and very different in how many wrong actions they take. What separates them is whether their confidence tells you when to stop.
By Faiz Shuja, Founder & CEO, SIRP
Elastic's Security Labs team recently published two pieces that are worth your time. The first, "Alert triage in the agentic SOC: self-correcting agents", describes how they took alert-triage accuracy in their own SOC from 60% to 92%, not by changing models, but by restructuring what the agent knows before it reasons. The second, "Agentic SOC: token budget architecture", is a token-budget analysis. Across 36,822 production conversations (about 8 billion tokens), Elastic reported that specialized agents cost 3.0 to 5.7 times less than a deliberative single agent.
Both findings match what we have seen building OmniSense, and one of them is genuinely counter-intuitive. The claim that the number of LLM calls, not the size of the system prompt, dominates token cost runs against most people's instincts, including ours when we started. A longer, more specific prompt that removes a round of deliberation is cheaper than a short one that causes three. That is a good result, honestly measured, and published with the methodology attached. There is not enough of that in this industry.
I want to build on it rather than argue with it, because there is something both articles share that is easy to miss.
Both articles measure investigation
In Elastic's architecture, the system investigates and recommends. An analyst reads the verdict and closes the case. That is a well-designed assistive system, and for a great many SOCs it is exactly the right thing to build.
But nothing in it acts. And the moment you let an agent act, a property that neither accuracy nor token budget captures becomes the one that matters most.
The blast radius is not symmetric
A recommendation that is wrong costs an analyst a few minutes of reading. They disagree, they close it differently, the system learns.
An action that is wrong costs something else. A contained host that was clean. An executive blocked from their own mailbox during a board meeting. A forensic artifact wiped by an over-eager remediation before anyone imaged it.
Same error rate. Entirely different consequence. This is why "how accurate is it" stops being the interesting question the moment execution enters the picture, and why an accuracy number on its own cannot tell you whether a system is safe to automate.
Accuracy and calibration are different properties
Accuracy is how often the system is right. Calibration is whether its stated confidence tracks its actual correctness: whether the things it says it is sure about are, in fact, the things it gets right.
Only the second one tells you when to stop.
Consider an illustrative example: two systems, both 90% accurate over 1,000 alerts. Both are wrong 100 times.
System A reports roughly 90% confidence on everything, whether it is right or wrong. Its confidence carries no information about its correctness. Set an automation threshold anywhere below 90 and it acts on all 1,000 alerts, including all 100 it got wrong.
System B reports high confidence on 750 alerts, where it is right 96% of the time, and low confidence on the remaining 250, where it is right 72% of the time. Same 900 correct, same 100 wrong. But set the threshold above its low band and it acts on 750 alerts, is wrong on 30 of them, and hands the other 250 to a human.
Identical accuracy. Thirty wrong actions instead of a hundred, and 250 cases that reached a person because the system knew it was on thin ice. The difference between those two systems is not intelligence. It is self-knowledge, and it is the only thing standing between an accuracy number and an autonomy decision.
This also explains something that sounds paradoxical: as accuracy improves, calibration matters more, not less. A system that is right 99% of the time will be trusted with more consequential actions. The 1% it is wrong about is now attached to decisions that hurt.
What we learned trying to measure it
We spent a long time measuring quality the obvious way: ask a model to score each response on an absolute scale, then track the score.
It did not work, and the way it failed is worth sharing. Our absolute-scale reviewer ended up flagging the large majority of responses. A detector that fires on most cases has no discriminative power left. At that point you are measuring the scale, not the system underneath it. We could not tell an improvement from a regression, because almost everything was flagged either way.
This is not a fact about our pipeline. It is a fact about using language models as absolute judges. They are unreliable at "rate this 1 to 10" in a way that is easy to miss, because the numbers look like data. Any team building this will hit the same wall.
What worked was going relative. Instead of scoring one response absolutely, on a sample of conversations we regenerate the same request through a second path and have a judge pick a winner between the two, with the positions randomized, because judges have a position bias that will otherwise quietly become your result. Ties are allowed. "Both unacceptable" is recorded as its own outcome, which turns out to be one of the more useful signals we collect. Win rates over time are a far better input to routing decisions than absolute scores ever were.
If you are standing up quality measurement for an agentic pipeline, that is the lesson I would most want to hand over: relative comparison is a sharper instrument than absolute scoring, and confidence numbers deserve the same skepticism you would apply to any other model output. They are a claim the system is making, not a measurement it is reporting.
What follows architecturally
If calibration is what governs safe action, then autonomy cannot sensibly be a global switch. A system is not "autonomous". It is autonomous for particular things it has demonstrated it handles well, and not for others.
That is the principle we design to. Each customer's policy tier sets which actions need a person's approval, and by default SARA's response actions wait for that approval. SARA shows its own confidence estimate on the incident, and the case reaches the analyst with the reasoning and the proposed action attached, so the person decides on evidence rather than on an alert. Letting any class of action run without a person is something a system should earn, case by case, on evidence like the above.
How you would measure any of this, including ours
None of the above is checkable from a vendor's marketing page, which is a problem for buyers and ultimately for vendors too. So we publish the SOC Autonomy Framework: an open, vendor-neutral model for classifying how autonomous a security system actually is, with a classifier and a benchmark suite anyone can run.
We publish it because "autonomous" has become a word every vendor uses, including us. A buyer deserves a definition they can apply themselves, to us as rigorously as to anyone else. If you run it against OmniSense and find our level lower than you expected, that result is more useful to both of us than a number on a slide.
The point
Elastic's team improved their triage accuracy substantially and told everyone how. That is a real contribution, and the industry gets better when engineering teams publish their methods instead of their adjectives.
The next question, for anyone whose system does not stop at a recommendation, is not how often it is right. It is whether it knows which cases it is likely to be wrong about, and whether it behaves differently when it does.
A clean handover to a person is not a failure. It is the safety feature. And it is the one that gets more important, not less, as everything else improves.
SIRP builds OmniSense, a Self-Driving SOC. The SOC Autonomy Framework, its classifier and its benchmark suite are published openly at github.com/SIRP-Labs.