ctianalyst.com

Interview question

Walk me through how you would use Analysis of Competing Hypotheses on an intrusion where the evidence is ambiguous. What makes ACH different from just weighing the evidence?

ACH questions separate people who can name the technique from people who understand its central counterintuitive claim: supporting evidence is nearly worthless.

What a strong answer covers

Expert answer

The thing that makes ACH different is counterintuitive: evidence that supports your favoured hypothesis is nearly worthless, because it usually supports several of them equally well. What carries information is evidence that is inconsistent with a hypothesis. So ACH works by elimination rather than accumulation, which is the opposite of how people naturally reason.

The process. First, enumerate hypotheses — before looking hard at the evidence, and deliberately including ones I think are unlikely. Getting a second person to add to the list matters here, because the hypotheses I omit are the ones my priors already excluded.

Second, list the evidence, including absences. "No exfiltration observed despite six days of functioning C2" is evidence, and it is often more diagnostic than anything present.

Third, build the matrix: hypotheses across, evidence down, each cell marked consistent, inconsistent, or not applicable.

Fourth, refine. Any row that is consistent with everything has no diagnostic value and should be struck out rather than counted as support for the leading hypothesis. This step is where most of the value is.

Fifth, rank by fewest inconsistencies. Sixth, sensitivity-test: which single item, if it turned out to be wrong or planted, collapses the conclusion? That item deserves the most scrutiny, and in cyber it is very often the one that was easiest to observe.

Finally, report the alternatives alongside the lead, and define the indicators that would change the ranking so collection can go after them.

Concretely, on an ambiguous ransomware intrusion I might carry four hypotheses: an opportunistic financially motivated affiliate; a targeted intrusion by an actor with prior access; destructive activity disguised as ransomware; and an insider. "A known ransomware family was deployed" is consistent with all four, so it gets struck. "No exfiltration despite a working channel" is inconsistent with the usual double-extortion pattern and starts to discriminate. "Initial access used credentials valid before the compromise window" bears on the insider hypothesis. That is the matrix doing work.

In practice nobody builds a full matrix during a live incident. What survives into daily work is the habit: generate alternatives unprompted, look for what would disconfirm, and strike the evidence that does not discriminate.

Mistakes that cost candidates points

Try answering it

You have read the answer, which is the easy part. Answer it in your own words and get graded against this same rubric, with the follow-up probe an interviewer would ask next.

Go deeper

Related questions