Jump to:Page Content
Good intelligence can often prevent bad things from happening, but only if the dots are connected in the right ways. A new Harvard Kennedy School Faculty Working Paper analyzes the ways in which "estimative intelligence" – that used to help policymakers deal with situations that are uncertain and complex – can be produced more accurately by assessing uncertainty rather than seeking to eliminate it altogether.
“Assessing Uncertainty in Intelligence” is co-authored by Richard Zeckhauser, Frank Plumpton Ramsey Professor of Political Economy, and Jeffrey Friedman, doctoral candidate in public policy.
The authors argue that trying to eliminate uncertainty altogether fosters two significant problems which do not occur when simply attempting to assess uncertainty.
Consequence neglect occurs when collectors, analysts and consumers of intelligence focus too much on the probability of each possible scenario and too little on the magnitude of each scenario’s potential consequences.
The authors cite the example of the Japanese attack at Pearl Harbor. U.S. Intelligence had encountered an explicit warning that Japan planned to attack Hawaii in 1941. Upon further investigation, the Navy determined that the Peruvian ambassador had originally received the information from his chef.
“There is no doubt, even in hindsight, that the U.S. Intelligence community should not have grounded strategic warning on the basis of a report from a Peruvian chef. However, it is hardly clear that the right move was to ‘discard and forget’ the information,” the authors write. “The report had a low degree of reliability but its potential consequences were enormous.”
Probability Neglect is the reverse problem, the authors explain, arising when intelligence focuses predominantly on the potential consequences of various possibilities while giving less attention to their respective likelihoods.
“When an estimate simply says some event is ‘unlikely,’ it is difficult for readers to weigh the prediction properly. Conversely, policymakers often have trouble thinking about what small probabilities mean, and sometimes effectively treat them as if they were zero,” write Zeckhauser and Friedman. “When likelihoods and consequences are not identified separately and then considered together, estimative intelligence will be incomplete, unclear and subject to misinterpretation.”
It is important to separate likelihood and confidence and to be explicit about each, the authors explain, because the analysis provides information about how those predictions might change if new information emerged.
In the case of Pearl Harbor, had there been several independent reports bearing similar information, Zeckhauser and Friedman argue, there would have been substantial grounds for taking the threat seriously.
“Estimative intelligence will always be as much art as science, and the result will always be imperfect. Yet uncertainty is bound to persist on critical issues, and when it does, the ultimate goal should be to assess this uncertainty in a clear and accurate manner,” the authors conclude.
Richard Zeckhauser’s policy investigations explore ways to promote the health of human beings, to help markets work more effectively, and to foster informed and appropriate choices by individuals and government agencies. His latest book, with Jack Donahue, is Collaborative Governance: Private Roles for Public Goals in Turbulent Times. His previous book, with Jonathan Nelson, was The Patron's Payoff: Conspicuous Commissions in Renaissance Italy.
Jeffrey A. Friedman is a graduate student associate at the Weatherhead Center for International Affairs and a doctoral candidate in public policy at Harvard Kennedy School. His research focuses on the theoretical and empirical foundations of military doctrine and strategy.
Richard Zeckhauser, Frank Plumpton Ramsey Professor of Political Economy
“Estimative intelligence will always be as much art as science, and the result will always be imperfect," write the authors.