Not Normal: The Uncertainty of Scientific Measurements

Abstract:  When scientists report uncertainties, what do they tell us? How excited should we get when a 3 or 4 sigma result suggests new physics? Researchers in fields from Medicine to Physics do a pretty good job of estimating the chance of small errors, but the frequency of big disagreements for even the best-made measurements is orders of magnitude greater than naively expected. Unknown problems appear to have power-law distributions consistent with how complex systems fail and how systematic errors are constrained. Occasional outliers are unavoidable at the research frontier.