How to Avoid Intelligence Analysis Errors

Reprinted from Britten Coyne Partners Strategic Risk Blog

This week, Mike Morrell had former senior CIA executive Martin Peterson on his Intelligence Matters podcast, which is well worth listening to.

Petersen succinctly summarized the lessons he’d learned over a career (and taught to new analysts) about the root causes of many intelligence analysis errors. They apply equally well to many situations outside the world of intelligence, where analysts and decisions makers must make sense of highly uncertain situations.

Petersen highlighted three classic types of error, and the questions to ask to avoid them.

(1) You don’t have a good understanding of the organization you’re trying to analyze.

• How do you get to the top in this organization?
• What is the organization’s preferred method of exercising power and making decisions?
• What are acceptable and unacceptable uses of power in this organization?

(2) You don’t have a good understanding of the individuals making decisions.

• How do they assess the current situation?
• How do they see their options?
• What is their tolerance for risk, under the current circumstances?
• What do they believe about your capabilities, intentions, and will?
• What is their definition of an acceptable outcome?

(3) You don’t understand your own analysis.

• Rather than asking someone how confident they are in their analysis, ask them where their analysis is most vulnerable to error. This is the same approach as the one we use, which is identifying the most uncertain assumptions in an analysis, and the implications of different outcomes for them. The underlying issues are also surfaced by use of pre-mortems, which we also recommend.
• What are you not seeing that you should be seeing if your hypothesis/theory/line of analysis is correct? As Sherlock Holmes (and Thomas Bayes) both teach us, sometimes the dog that doesn’t bark provides the most important evidence.
• Petersen emphasized that if you ever catch yourself saying, ‘It makes no sense for them to do that’, it is a clear warning sign that you either don’t understand either the organization and/or the decision makers that are the target of your analysis.

All important lessons to keep in mind as you try to make sense of the complex, highly uncertain, and fast changing situations that abound in the world we face today.