Showing posts with label statistics. Show all posts
Showing posts with label statistics. Show all posts

Saturday, November 11, 2017

"Why Are Handsome Men Such Jerks?"



Jordan Ellenberg writes:
"You may have noticed that, among the men in your dating pool, the handsome ones tend not to be nice, and the nice ones tend not to be handsome.

"Is that because having a symmetrical face makes you cruel? Does it mean that being nice to people makes you ugly? Well, it could be. But it doesn’t have to be. ...

"The handsomest men in your triangle, over on the far right [the green line], run the gamut of personalities, from kindest to (almost) cruelest. On average, they are about as nice as the average person in the whole population, which, let’s face it, is not that nice. ..

The ugly guys you like, though—they make up a tiny corner of the triangle [the red line], and they are pretty darn nice. They have to be, or they wouldn’t be visible to you at all.

The negative correlation between looks and personality in your dating pool is absolutely real. But the relation isn’t causal. "
You can see that the "niceness" distribution of the "uglies" is totally bunched up at nice while the "niceness" distribution of the "handsomes" is not dissimilar to the overall population norm.

It follows that the average of the most handsome people [centre of green line] is going to be meaner than the average of the ugly ones [centre of red line].

Niceness suddenly negatively correlates with handsomeness, whereas in the overall population - by hypothesis - there is no correlation at all.

Notice you could equally ask, "Why are the nicest ones so ugly?" - and then the green and red lines are interchanged and horizontal.

This is called Berkson's paradox and is a general feature of populations where a selection is made jointly on two weakly-correlated or uncorrelated variables. The selection process itself can induce correlations where none existed previously, or even reverse previous correlations.

The effect is particularly pernicious in academic admissions (and also medical trials).

---

[h/t: Razib Khan].

Monday, April 11, 2016

"The Seven Pillars of Statistical Wisdom" by Stephen M. Stigler

A review of: "The Seven Pillars of Statistical Wisdom" by Stephen M. Stigler.



Every student of an abstract subject like maths, physics or even philosophy is familiar with this: you are introduced to a foundational concept yet it seems pretty counterintuitive, and you can think of a number of reasons why the said concept ought to be considered problematic. Yet somehow, the textbooks are less than sympathetic.

My advice is to check the history of the under-motivated concept. The original formulations were often so much more compelling, especially when you realise precisely what problem their authors were trying to solve. Your own misgivings may well be represented in critiques by the innovator’s contemporaries.

It was the very success of later generations which led to the wholesale reconceptualisation of their subject’s foundations.

And so it is with statistics, a subject where deep ideas are often obscured by a focus on technique, and where it sometimes seems that little distinguishes a correct line of argument from an equally plausible, but fallacious, alternative.

Professor Stephen Stigler, in this determinedly historical book, starts with a concept as apparently trivial as the mean, or average, of a sequence of observations. Even this is counterintuitive as it requires discarding information, the individuality of the observations. By what right are ‘bad’ measurements to be treated in the same way as ones we think, or know, to be of higher quality? It took quite a few years for the idea to catch on.

Stigler’s second pillar, information measurement, looks at the processing of large data sets. Opinion polls have made us somewhat aware that the accuracy of the proposed mean is proportional to the square root of the number of observations, not the absolute number.

Sampling was applied to the Royal Mint in Isaac Newton’s time, to ensure that the coins they produced used the right amount of gold. In the absence of a correct theory of standard deviation, the tolerance boundaries were set way too wide. Stigler dryly notes that Newton was warden, then master of the Royal Mint from 1696 to 1727 and that on his death in that year left a sizeable fortune. “But evidently his wealth can be attributed to investments, and there is no reason to cast suspicion that he had seen the flaw in the Mint’s procedures and exploited it for personal gain.”

Later chapters deal with hypothesis testing (pillar 3); statistical processing within the dataset itself, without reference to population norms – as in Student’s t-test (pillar 4); regression to the mean - a concept which proved very hard to pin down (pillar 5); experimental design, particularly when varying multiple qualities at the same time (pillar 6); and finally pillar 7, the notion that a complicated phenomenon may be simplified by subtracting the effect of known causes, leaving a residual phenomenon to which attention may now be focused.

If you are both interested and well-versed in statistics, you will find this book illuminating and witty. The converse also applies.

Wednesday, February 10, 2016

"My simulated sister is smarter than me"

Apologies that this is a bit techie - and it won't make sense without reading the previous post.

Yesterday I did some simple stats to show that my sister is most likely 8 IQ points smarter than me (to be fair and by symmetry, the converse could also be true).

Health warnings:
  1. expected value only when averaged over large numbers of copies of my sister and myself;
  2. equally true for my brother and myself - the stats are gender-blind.
How far is one sibling likely to be from the parental midpoint average?

Intuitively, you wouldn't expect every sibling to be exactly the average (they're not clones) but over a large family the pluses and minus would sort of average out to the mid-parental mean. But what about if we're just considering the deviation from average, without caring about the sign?

We seem to have a choice: halve the expected difference between two siblings, or find the average (absolute) deviation from the mean. As we saw yesterday, these give different answers.

I therefore decided to run an Excel simulation using the built-in RAND() function. Here's the four coin-set (taking values from 0 to 4):
IF(RAND()>0.5,1,0) + IF(RAND()>0.5,1,0) + IF(RAND()>0.5,1,0) + IF(RAND()>0.5,1,0)
and here is the last part of the spreadsheet model showing 100 tosses of two 4-coin sets (random variables X and Y) showing the number of heads.

If you like, you can consider this a simple four gene model for intelligence, with each gene presenting as two alleles, each of which code up or down for IQ by 7.5 points.



I ran each 100 toss simulation ten times and noted the results in the table on the right.
  • The heading "Mean-IQ" refers to ten runs of the IQ (7.5) value in the "abs(X-Y)" column on the left, showing the mean difference in IQ between the two siblings; 

  • the heading "Dec-IQ" refers to ten runs of the IQ (7.5) value in the "abs(X-2)" column on the left, showing the average deviation (+ and -) of a single sibling's IQ from the parental-midpoint mean.
From yesterday's post the computed values are respectively 8.2 and 5.625.

If we go back to selecting embryos for implantation, which is the right statistic to use to measure our likely IQ gain over the biological default of just taking what comes?

The leftmost statistic, 5.625 IQ points above the mean, would be sort of accurate if we were conceptually considering two embryos, one randomly varying and the other always exactly on the parental midpoint mean. But it wouldn't work, not least because the random embryo might well be below the mean but we're counting all variation as positive. So it's not realistic.

The statistic we get by halving the expected inter-sibling gap of 8.2 IQ points is better as we always select the smarter of the two embryos. However, since both X and Y are varying freely on the range {0,1,2,3,4} it's a bit difficult to correlate the abs(X-Y)/2 gap with the range-midpoint (mean) of 2. At this point we handwave and mutter about symmetry.

And what do you do when the presented embryos are all below the expected average?*

---

* Which with two embryos will occur 25% of the time. I feel like spending some more money and genotyping a few more ...