Can an AI SOC analyst hallucinate? Yes. But not the way most people picture it. The picture in most heads is a model inventing indicators out of thin air. The failure that should actually worry you is quieter, and worse. The model does exactly what it was built to do. It completes a pattern with half the picture missing.

Put precisely: the inherent risk is that the model performs probabilistic pattern completion over an incomplete representation of the event. It is not that an LLM randomly turns a true event into a false one. I want to walk through the mechanics here, no vendor claims attached. Once you see how the math behaves, you will never read an accuracy slide the same way again.

How does an LLM actually triage a security alert?

Forget the product UI for a minute. Under the hood, every LLM investigation runs the same loop. The alert, the tool outputs, everything that has happened so far, it all gets encoded as tokens. Billions of learned weights get applied through matrix multiplication. Out comes a probability distribution over what comes next. The model picks the likely continuation, appends it, and goes again. That is it. That is the machine.

Notice what is not in there. No truth module. No little database of what is actually happening in your environment. The model has one skill, producing whatever usually follows from contexts that look like this one, and most of the time that is exactly what you want, because most alerts really do look like the ten thousand alerts before them.

But there is a structural constraint hiding in that loop, and it is the one that matters for security. Attention can only distribute weight across what is in the context. Evidence that was never retrieved does not get a low weight. It gets no weight. It is not in the calculation at all. The blind spots of an investigation are, mathematically, invisible to the very machinery rendering the verdict.

And the prose will not warn you. A model writes with the same easy confidence whether its investigation was exhaustive or half-blind. Fluency comes from the generation process, not from the evidence underneath it.

The completeness blind spot: why doesn’t the model know what it didn’t check?

Sit next to a good analyst for a shift and you will notice something they do without thinking about it. They keep a running tally of what they have not checked. Couldn’t reach the DC logs. Never got to email. That tally nags at them, and it should.

A raw LLM keeps no such tally. Unless the surrounding system spells it out, the model may not know:

which systems were queried,
which queries succeeded,
which queries failed,
whether the responses were complete,
whether expected data sources were unavailable,
whether a null value means “not found” or “not checked.”

So an empty result set arrives in the context looking the same whether the environment is clean or the query never ran. Which means the model can blur two statements that should send an investigation in opposite directions:

“No evidence of compromise”

with

“No accessible evidence of compromise”

The second one should make confidence drop through the floor. There is a hole in the investigation, and the hole is exactly where an attacker would be standing. The first can reasonably close an alert. Hand a probability engine a context that encodes both the same way, and it has no idea which one it is looking at.

What is the central guardrail for AI triage?

Everything in this piece compresses down to one rule:

Missing evidence must increase uncertainty — not increase confidence in benignity.

Here is what makes the rule hard, and interesting: the model cannot enforce it on itself. Its attention has no access to what was never retrieved. That is the whole problem. The only component that knows a query was attempted and failed is the orchestration layer around the model. So when an orchestration layer represents failed retrieval as absent evidence, the model does what its arithmetic requires. It calculates that benign is the most probable explanation of a context with no threat indicators in it, and it closes a true positive as false. Confidently.

This is also why two products running the same foundation model can hand you opposite verdicts on the same alert. The model is not the differentiator. The ledger is.

What does this failure look like in a real investigation?

Picture an alert any SOC has seen: remote-code-execution activity on a server. The model does what pattern completion does. It reaches for the most common explanation of that alert class. Say, a famous CVE. It goes hunting for indicators of that theory and comes up empty. Some of those empty results are genuinely clean lookups. A couple are queries that quietly failed. In the context window, they look identical.

Now the failure compounds. No evidence for the hypothesis gets read as no attack. The verdict practically writes itself: false positive, low risk, case closed. And the actual attack keeps running through a layer the model never successfully examined, while the alert that would have caught it sits in the closed pile under a confident paragraph of explanation.

Nothing malfunctioned. Every step was the probability machine behaving exactly as designed. That is the part that should bother you. Without the guardrail, the confident wrong verdict is on the schedule.

Why doesn’t a 99% accuracy score protect you?

Raise any of this with a vendor and you will get an accuracy number back. Fine. Let us do the arithmetic the slide skips. Ninety-nine percent sounds excellent. At 10,000 alerts a day, it means 100 wrong verdicts a day. Call it 36,500 a year. And the half that should keep you up at night are the real attacks filed as benign, because those do not sit in anyone’s queue waiting for a second look. They just vanish.

The accuracy number is not dishonest. It is answering the wrong question. The question worth asking is what an error turns into. In a fail-closed design, an error becomes a silently closed attack. In a fail-open design, it becomes a flagged hand-off to a human with the evidence attached. Same error rate, completely different bad day.

What does the AI SOC business case leave out?

Let us be honest about why these tools are getting bought right now. It is cost. The pitch that wins budget meetings is standing down a five-analyst Tier 1 bench and running the same queue on software priced below one of those analysts, fully loaded. And the savings are real. They land this quarter, in numbers a CFO can verify.

What never shows up in the spreadsheet is the thing you gave up: that running tally of unchecked corners the humans were keeping for free. There is no line item for it. Auto-closed alerts do not file tickets. The cost stays invisible until a breach review works its way back to the first indicator and finds it in the closed pile, dispositioned benign by the model, and someone asks how it got missed.

The honest answer: nobody was there to catch it. Triage belonged to a model that was right 99% of the time, and this alert lived in the other 1%.

D3 Security

Message History

Delivery option(s)