"The Seven Pillars of Statistical Wisdom" by Stephen M. Stigler

A review of: "The Seven Pillars of Statistical Wisdom" by Stephen M. Stigler.

Every student of an abstract subject like maths, physics or even philosophy is familiar with this: you are introduced to a foundational concept yet it seems pretty counterintuitive, and you can think of a number of reasons why the said concept ought to be considered problematic. Yet somehow, the textbooks are less than sympathetic.

My advice is to check the history of the under-motivated concept. The original formulations were often so much more compelling, especially when you realise precisely what problem their authors were trying to solve. Your own misgivings may well be represented in critiques by the innovator’s contemporaries.

It was the very success of later generations which led to the wholesale reconceptualisation of their subject’s foundations.

And so it is with statistics, a subject where deep ideas are often obscured by a focus on technique, and where it sometimes seems that little distinguishes a correct line of argument from an equally plausible, but fallacious, alternative.

Professor Stephen Stigler, in this determinedly historical book, starts with a concept as apparently trivial as the mean, or average, of a sequence of observations. Even this is counterintuitive as it requires discarding information, the individuality of the observations. By what right are ‘bad’ measurements to be treated in the same way as ones we think, or know, to be of higher quality? It took quite a few years for the idea to catch on.

Stigler’s second pillar, information measurement, looks at the processing of large data sets. Opinion polls have made us somewhat aware that the accuracy of the proposed mean is proportional to the square root of the number of observations, not the absolute number.

Sampling was applied to the Royal Mint in Isaac Newton’s time, to ensure that the coins they produced used the right amount of gold. In the absence of a correct theory of standard deviation, the tolerance boundaries were set way too wide. Stigler dryly notes that Newton was warden, then master of the Royal Mint from 1696 to 1727 and that on his death in that year left a sizeable fortune. “But evidently his wealth can be attributed to investments, and there is no reason to cast suspicion that he had seen the flaw in the Mint’s procedures and exploited it for personal gain.”

Later chapters deal with hypothesis testing (pillar 3); statistical processing within the dataset itself, without reference to population norms – as in Student’s t-test (pillar 4); regression to the mean - a concept which proved very hard to pin down (pillar 5); experimental design, particularly when varying multiple qualities at the same time (pillar 6); and finally pillar 7, the notion that a complicated phenomenon may be simplified by subtracting the effect of known causes, leaving a residual phenomenon to which attention may now be focused.

If you are both interested and well-versed in statistics, you will find this book illuminating and witty. The converse also applies.