Risk Analysis 101
As we all know, or rather should know, Risk Analysis is a risky business!
It is therefore surprising, not to say disappointing, that so many risk analysts are so relaxed about this risky business. In particular, one would have thought that peer-reviewed journals that are dedicated to risk analysis would be immune to obvious pitfalls in this business.
Furthermore, I would have expected such peer-reviewed journals to be well informed about the state of the art in areas of expertise that are closely affiliated with risk analysis.
It turns out that my expectations had been too high, hence the Risk Analysis 101 project.
In brief
The objective of this project is to provide readers of Risk Analysis An International Journal (RISA), and the public at large, a second opinion on some of the articles that were published in this Journal. The primary goal of these second opinions is to advise the readership of the Journal of obvious --- and not so obvious --- erroneous/misleading claims, assertions, statements and so on, that managed to get through the peer-review process of the Journal.
Let me be more specific about the background of this project, its scope and objectives.
Since the end of 2003, I have been in contact with many risk analysts, mostly academics, dealing with non-probabilistic approaches to decision-making under severe uncertainty. Four facts, on which we had interesting exchanges drew my attention to peer-reviewed articles concerned with this topic:
- Rhetoric plays a central role in this business.
- Disrespect for the state of the art in relevant "sister" disciplines is rampant.
- A lack of appreciation of the difference between local and global risk analysis, e.g. local and global robustness, is quite common.
- A lack of appreciation of the difference between worst-case analysis of performance levels and worst-case analysis of performance constraints, is quite common.
- A serious confusion between a generic model and its instances --- e.g. a generic maximin model and some of its numerous instances --- can easily lead to the re-invention of the wheel, and a square one at that!
I should point out that these remarks refer particularly to the literature on info-gap decision theory, a supposedly new, revolutionary, non-probabilistic theory for decision-making under uncertainty, which is in fact a classic example of a voodoo decision theory, that is based on a reinvented model of local robustness that is universally known as Radius of Stability model (circa 1960).
For your convenience, here is info-gap's robustness model:
and here is a generic radius of stability model:
The details, including a proof that info-gap's robustness model is a radius of stability model, can be found here.
In view of this, at the end of 2006 I launched a campaigned to contain the spread of info-gap decision theory in Australia. An important part of this campaign was the publication of articles that explained the danger in using simplistic approaches for the management of severe non-probabilistic uncertainty (see here).
I also posted on my website reviews of publications, mostly peer-reviewed articles, on info-gap decision theory, where I explain in detail the problematic issues with this theory.
Why RISA?
One peer reviewed journal that drew my attention was Risk Analysis (RISA), an official publication of the Society for Risk Analysis. A number of my reviews were on articles published in this journal.
According to the publisher of RISA (Read on July 4, 2012):
Risk Analysis is ranked among the top 10 journals in the ISI Journal Citation Reports under the social sciences, mathematical methods category, and provides a focal point for new developments in the field of risk analysis. This international peer-reviewed journal is committed to publishing critical empirical research and commentaries dealing with risk issues.
So I asked myself:
How is it that a respectable peer-reviewed journal such as RISA continues to publish articles that are so obviously flawed, hence misleading?!
Remark: I have to explain the continue in the phrasing of this question. And the explanation is that on a number of occasions, this issue was pointed out to RISA. In fact, RISA accepted for publication the following article that is dedicated to this very issue:
Sniedovich, M. (2012) Fooled by local robustness. Risk Analysis (Early View)
The abstract of this article reads as follows:
One would have expected the considerable public debate created by Nassim Taleb's two best selling books on uncertainty, Fooled by Randomness and The Black Swan, to inspire greater caution to the fundamental difficulties posed by severe uncertainty. Yet, methodologies exhibiting an incautious approach to uncertainty have been proposed recently in a range of publications. So, the objective of this short note is to call attention to a prime example of an incautious approach to severe uncertainty that is manifested in the proposition to use the concept radius of stability as a measure of robustness against severe uncertainty. The central proposition of this approach, which is exemplified in info-gap decision theory, is this: use a simple radius of stability model to analyze and manage a severe uncertainty that is characterized by a vast uncertainty space, a poor point estimate, and a likelihood-free quantification of uncertainty. This short discussion serves then as a reminder that the generic radius of stability model is a model of local robustness. It is, therefore, utterly unsuitable for the treatment of severe uncertainty when the latter is characterized by a poor estimate of the parameter of interest, a vast uncertainty space, and a likelihood-free quantification of uncertainty.
Yet, RISA continues to provide a platform for a grossly misleading rhetoric on robust decision-making under severe uncertainty.
And so, as indicated above, the objective of the Risk Analysis 101 project is to provide readers of RISA, and the public at large, a second opinion on some of the articles published in this journal.
How does it work?
I created a website entitled RISK ANALYSIS 101 that is dedicated to the Risk Analysis 101 project, where I plan to post comments and second opinions on articles published in RISA. The URL is
The second opinions can be classified into two categories:
- Second opinions on specific articles published in RISA.
- Second opinion on specific topics discussed in articles published in RISA.
These comments and second opinions are intended primarily for readers of RISA, but also for the authors of the articles and the referees and the editors who accepted them for publication in RISA.
Second opinions
The following second opinions are dedicated to specific articles published recently in RISA:
- Robust-Optimization: the elephant in the robust-satisficing room
A second opinion on:
Ben-Haim, Y. (2012) Doing Our Best: Optimization and the Management of Risk.
Risk Analysis (Early View)
- Fooled by local robustness ... again!
A second opinion on:
Hall, J., Lempert, R.J., Keller, K., Hackbarth, C., Mijere, A., and McInerney, D.J. (2012)
Robust climate policies under uncertainty: A comparison of Info-Gap and RDM methods.
Risk Analysis, 32(10), 1657-1672.
The following second opinions are dedicated to specific topics discussed recently in RISA:
The following article was motivated by 5 papers that had been published in Risk Analysis:
- Sniedovich M. (2011). Fooled by robustness: a radius of stability perspective.
Working paper No. MS-1-11, Department of Mathematics and Statistics, The University of Melbourne, Australia.
Available here (PDF file).
A short version of the above article was published recently in Risk Analysis:
- Sniedovich M. (2012). Fooled by local robustness. Risk Analysis, 32(10), 1630-1637.
Latest news
The following articles discuss the latest news from RISA (April 26, 2013).
The following are relatively old reviews of info-gap publications. The reviews of RISA articles are shown in bold face.
- Ben-Haim (2001, 2006): Info-Gap Decision Theory: decisions under severe uncertainty.
- Regan et al (2005): Robust decision-making under severe uncertainty for conservation management.
- Moilanen et al (2006): Planning for robust reserve networks using uncertainty analysis.
- Burgman (2008): Shakespeare, Wald and decision making under severe uncertainty.
- Ben-Haim and Demertzis (2008): Confidence in monetary policy.
- Hall and Harvey (2009): Decision making under severe uncertainty for flood risk management: a case study of info-gap robustness analysis.
- Ben-Haim (2009): Info-gap forecasting and the advantage of sub-optimal models.
- Yokomizo et al (2009): Managing the impact of invasive species: the value of knowing the density-impact curve.
- Davidovitch et al (2009): Info-gap theory and robust design of surveillance for invasive species: The case study of Barrow Island.
- Ben-Haim et al (2009): Do we know how to set decision thresholds for diabetes?
- Beresford and Thompson (2009): An info-gap approach to managing portfolios of assets with uncertain returns.
- Ben-Haim, Dacso, Carrasco, and Rajan (2009): Heterogeneous uncertainties in cholesterol management.
- Rout, Thompson, and McCarthy (2009): Robust decisions for declaring eradication of invasive species.
- Ben-Haim (2010): Info-Gap Economics: An Operational Introduction.
- Hine and Hall (2010): Information gap analysis of flood model uncertainties and regional frequency analysis.
- Ben-Haim (2010): Interpreting Null Results from Measurements with Uncertain Correlations: An Info-Gap Approach
- Wintle et al. (2010): Allocating monitoring effort in the face of unknown unknowns
- Moffitt et al. (2010): Securing the Border from Invasives: Robust Inspections under Severe Uncertainty
- Yemshanov et al. (2010): Robustness of Risk Maps and Survey Networks to Knowledge Gaps About a New Invasive Pest
- Davidovitch and Ben-Haim (2010): Robust satisficing voting: why are uncertain voters biased towards sincerity?
- Schwartz et al. (2010): What Makes a Good Decision? Robust Satisficing as a Normative Standard of Rational Decision Making
- Arkadeb Ghosal et al. (2010): Computing Robustness of FlexRay Schedules to Uncertainties in Design Parameters
- Hemez et al. (2002): Info-gap robustness for the correlation of tests and simulations of a non-linear transient
- Hemez et al. (2003): Applying information-gap reasoning to the predictive accuracy assessment of transient dynamics simulations
- Hemez, F.M. and Ben-Haim, Y. (2004): Info-gap robustness for the correlation of tests and simulations of a non-linear transient
- Ben-Haim, Y. (2007): Frequently asked questions about info-gap decision theory
- Sprenger, J. (2011): The Precautionary Approach and the Role of Scientists in Environmental Decision-Making
- Sprenger, J. (2011): Precaution with the Precautionary Principle: How does it help in making decisions
- Hall et al. (2011) : Robust climate policies under uncertainty: A comparison of Info-Gap and RDM methods
- Ben-Haim and Cogan (2011) : Linear bounds on an uncertain non-linear oscillator: an info-gap approach
- Van der Burg and Tyre (2011) : Integrating info-gap decision theory with robust population management: a case study using the Mountain Plover
- Hildebrandt and Knoke (2011) : Investment decisions under uncertainty --- A methodological review on forest science studies.
- Wintle et al. (2011) : Ecological-economic optimization of biodiversity conservation under climate change.
- Ranger et al. (2010) : Adaptation in the UK: a decision-making process.
 |  | |
| www.RiskAnalysis101.wordpress.com | ||
