Source grading for analysts
A reliable source can report something false. An unreliable source can be right. Grading them on one axis loses the information that matters.
Source evaluation is the least glamorous tradecraft skill and one of the most frequently probed, because it is easy to check whether someone actually does it.
The Admiralty scale
Also called the NATO system, it grades every piece of reporting twice — once for the source, once for the information. A rating looks like B2.
| Source reliability | Information credibility |
|---|---|
| A — Completely reliable | 1 — Confirmed by other sources |
| B — Usually reliable | 2 — Probably true |
| C — Fairly reliable | 3 — Possibly true |
| D — Not usually reliable | 4 — Doubtful |
| E — Unreliable | 5 — Improbable |
| F — Reliability cannot be judged | 6 — Truth cannot be judged |
Reliability is a property of the source, built from its track record, methodology, access and incentives. Credibility is a property of this specific claim, judged mainly on whether independent reporting corroborates it and whether it is internally consistent.
The pairing is what carries information. An A6 — a highly reliable source making a claim nobody can corroborate — is a genuinely common and genuinely awkward situation, and the notation forces you to notice it.
The trap
The two axes are meant to be independent, and in practice analysts let reliability leak into credibility: a claim gets graded as more credible because a trusted vendor made it, rather than because anything corroborates it. Interviewers ask about this directly. Name the failure mode.
Circular reporting
The most common way corroboration is faked. Four articles reference a claim; three of them cite the fourth. You have one source, not four, and your confidence should reflect that.
How to catch it:
- Trace every claim to the earliest publication that presents original evidence, not the most recent one.
- Check whether "independent" sources share collection — two vendors using the same telemetry partner, or the same sinkhole, are not independent.
- Watch for identical phrasing, identical indicator lists, or the same typo propagating.
- Distinguish a source that observed something from one that agrees with someone who observed it.
Grading the sources CTI actually uses
| Source | What to weigh |
|---|---|
| Your own telemetry | Highest reliability available, but scoped to your visibility. Absence of evidence is not evidence of absence — know your blind spots. |
| Vendor research blogs | Often excellent primary evidence, with a commercial incentive to publish and to emphasise novelty. Grade the evidence shown, not the brand. Reports that show their working deserve higher credibility than ones that assert conclusions. |
| Government advisories | Usually high reliability and frequently based on non-public collection you cannot verify. Note that you are trusting rather than corroborating. |
| Sharing communities (ISACs, trust groups) | High relevance to your sector. Reliability varies by contributor, and handling restrictions apply — see TLP. |
| Commercial feeds | Ask how it was collected before asking how big it is. Volume is not quality, and unexplained provenance is a low grade regardless of price. |
| Criminal forums and leak sites | Actors lie, exaggerate and re-list old data. Treat claims as claims; corroborate against victim-side evidence before reporting a breach as fact. |
| Social media research | Fast and occasionally first. Frequently wrong in the first hours of an incident, and heavily prone to circular amplification. |
Source grades are inputs to the confidence half of your assessment, not decoration in an appendix. If your judgement rests on C3 material, your confidence cannot be high, and your product should say why.
Run a mock interview on this material. You get the question, a graded response against a vetted rubric, and the probing follow-up an interviewer would ask next.
Related reading
- Attribution and its limits Attribution is the fastest way to fail a scenario interview. The skill is not naming the actor — it is knowing what your evidence can and cannot support, and saying so.
- Analytic confidence and estimative language These are two different axes. You can be highly confident that something is unlikely. If you cannot say that sentence and explain it, you will lose points in every scenario round.
- Structured analytic techniques ACH is not a spreadsheet ritual. It is a discipline for finding the evidence that would prove you wrong, which is the only kind that carries much information.
- Writing intelligence reports You will be judged on your writing more than your tooling. The bar is a judgement up front, evidence behind it, and an explicit "so what" for someone you have named.