Showing posts with label statistics. Show all posts
Showing posts with label statistics. Show all posts

Monday, March 03, 2008

The Conehead Economy

Economic growth is a good thing. (Well, there's some big questions over both the possibility and desirability of perpetual growth, but let's ignore them for now.) On average, economic growth means that more people can afford more things, which means they can live happier and healthier lives and not need to worry about starving to death.

Ah, but there's a trap hidden in that statement: on average. Average growth is a very different thing than real growth, especially if you measure average with the "arithmetic mean" that you probably learnt about in school. A toy example will show what I mean: suppose our toy economy consists of five workers: Homer, Moe, Apu, Barney and Mr Burns. In 2006, Homer, Moe, Apu and Barney make $30,000 a piece, and Mr Burns makes $3,000,000. In 2007, Mr Burns' income has increased by $1,000,000, while the others earn exactly the same amount. The result is that the average income increases from $624,000 to $824,000. Lo and behold, Mayor Quimby can crow that his Millionaire Friendly Policies has led to the average worker getting a 32% increase in income in just one year! Marvellous! Obviously a rising tide raises all boats.

Drawn out in detail like that, it seems so obvious that nobody could possibly be fooled by it. Lies, damned lies and statistics. But in fact, that is precisely what the miracle of American economic growth since the late 1970s is made up of. Only the numbers are different, the principle is the same: the vast number of Americans have seen virtually no economic growth, or even a loss of income, while the overall average is inflated by enormous gains at the top of the pyramid. There's been growth, and plenty of it, but only a relatively small number of people, the richest 1%, have seen much benefits.

As Ezra Klein writes:

In the past, I've called this "The Conehead Economy." Plenty of growth in the economic body, but all of it happening in the top percent. Were that to happen to a person, you'd see six inches of growth in their forehead and doctors everywhere would be puzzling over how to correct the grotesque deformity. As it is, the media trumpets the growth, the politicians backslap over the roaring economy, and everyone wonders why the average American seems so unhappy. [...]

Meanwhile, government policy is explicitly aimed at accelerating the income distortions. [...] But don't object, o' Democrats, lest you be accused of class warfare which, as we know, only happens when the middle class wants their wages to keep up with productivity, as they did in the last generation.

Lest anybody conclude from this that statistics are essentially dishonest, consider this: there are many ways to calculate the average. The method used above, the mean, is just one way of many, and while it has its uses, it is very vulnerable to being distorted by a few very high or very low values. When it comes to income, a better measurement of average is usually the median, which for the above toy economy works out at $30,000 a year, with zero growth on average. Not quite so useful a figure for either the economists or for Mayor Quimby's re-election chances, but it reflects better the actual experience of 4 out of 5 people.

Sunday, July 15, 2007

Telling lies with statistics

    There are lies, damned lies, and statistics.
    -- attributed to Benjamin Disraeli.

I'm a great fan of Darrell Huff's perennially[1] best-selling How To Lie With Statistics. The title is deliberately ironic, as the book is really about how not to be be misled by the misuse of statistics.

One of the most egregious examples of seductively bad reasoning in economics is the Laffer curve, particularly as the central plank of Reagonomics and supply-side voodoonomics. Not surprisingly, the Wall Street Journal loves it (they know their audience...), as PZ Myers describes.

See also Brad DeLong, and Crooked Timber on throwing out outliers like Norway.



[1] Amusingly, this too is an example of lying with statistics. Back

Friday, February 02, 2007

Marriage and journalistic honesty

I came across a particularly interesting (in the sense of ew, what's that freakish thing living under a rock???) case of yellow journalism recently.

First, some background: in January, the New York Times ran a story [story has been archived] about their analysis of the latest census results. They reported that, for possibly the first time in recorded history, a majority of women in the USA are living without a husband.

Take note of the story headline, 51% of women were living without a husband. The story says:

For what experts say is probably the first time, more American women are living without a husband than with one, according to a New York Times analysis of census results.

In 2005, 51 percent of women said they were living without a spouse, up from 35 percent in 1950 and 49 percent in 2000.

Coupled with the fact that in 2005 married couples became a minority of all American households for the first time, the trend could ultimately shape social and workplace policies, including the ways government and employers distribute benefits.

It's quite clear: not "single women", or "unmarried women", but "living without a husband".

In case there is any doubt, the New York Times gives details about what they mean: they tell what counts as "living without a husband", they explain that one reason for the change is that women are getting married later than ever before, and they even admit that the living arrangements are often temporary:

In a relatively small number of cases, the living arrangement is temporary, because the husbands are working out of town, are in the military or are institutionalized. But while most women eventually marry, the larger trend is unmistakable.

"This is yet another of the inexorable signs that there is no going back to a world where we can assume that marriage is the main institution that organizes people's lives," said Prof. Stephanie Coontz, director of public education for the Council on Contemporary Families, a nonprofit research group. "Most of these women will marry, or have married. But on average, Americans now spend half their adult lives outside marriage."

The New York Times couldn't have been more clear and upfront about what the Census Bureau's 2005 American Community survey results show.

But that's not good enough for conservative pundits, like the fine citizen journalists at Lifesite.net. This is what Peter Smith had to say about the story:

New York Times Gets Another Story Very Wrong - This Time it's about Marriage
Accused of "journalistic malpractice" for skewing stats to incorrectly show most women not marrying
By Peter J. Smith

Strong words, but hardly correct. As the earlier quotes show, the NYT did not, by any stretch of imagination, claim that most women are "not marrying". If there is any "journalistic malpractice" here, it is by Smith. What part of "Most of these women will marry" was too hard to understand?

Smith writes:

However, Roberts creates his own analysis by using the Census Bureau's "Living Arrangements of Persons 15 Years Old and Over by Selected Characteristics", by including in his 51% figure of women living without a spouse: unmarried teenage and college girls still living with their parents, women whose husbands work out of town, are institutionalized, or are separated from husbands serving in Afghanistan and Iraq.

I have to wonder why Smith thinks it is appropriate to exclude women who are living without a spouse when talking about women who are living without a spouse.

Naturally, the conservatives questions the NYT for including young adults, 15 years and older, in the survey. But fifty years ago, marriage at 16 was relatively common -- now it is virtually unheard of. That's a significant shift, and one which continues as people choose to remain unmarried for longer periods of their life. That's precisely the point the NYT is making.

Smith criticises the NYT figures for failing to match the the census figures which show that "60.4% of men and 56.9% of women over 18 years old are married." But the last US census was in 2000, and the NYT is explicitly referring to 2005 data. Naturally there is going to be some difference.

Smith is less than transparent with his quoting of statistics:

Among marriageable women over 18 years old, 56.9% of women are married, with 53% having a spouse present, 1.4% with a spouse absent, 9.9% widowed, and 11.5% divorced.


The numbers don't quite add up... 53 + 1.4 + 9.9 + 11.5 = 75.8%. That falls short of 100% by almost a quarter. Who has he left out? But even his figures for married women are off: 53 + 1.4 = 54.4%, shy of the 56.9% in total he quotes. I suppose the difference could be "no answer": those who declined to say whether their husband was absent or not.

Who are the people of LifeSite? Their About Us page gives a couple of clues:

  1. LifeSite emphasizes the social worth of traditional Judeo-Christian principles but is also respectful of all authentic religions and cultures that esteem life, family and universal norms of morality.

  2. LifeSite's writers and founders have come to understand that respect for life and family are endangered by an international conflict. That conflict is between radically opposed views of the worth and dignity of every human life and of family life and community. It has been caused by secularists attempting to eliminate Christian morality and natural law principles which are seen as the primary obstacles to implementing their new world order.

(Emphasis added.)

Yes, you read that right. "Secularists" are against life and family. "New world order" (at least they aren't capitalising it any longer). And as for war, well, the only war worth talking about is the supposed (that is, imaginary) war against Christian morality.

If you think I'm exaggerating, I urge you to spend some time with Google investigating the stories on their site. Google on "site:www.LifeSite.net war" and you will find plenty of references to the "war on families", the "culture war", "war on religion" and even the "war on home-schooling", but very little if anything on any actual war involving soldiers and bombs and death.

It isn't that the people of LifeSite are ignorant of the Iraq invasion and occupation. Oh no, they are happy to tell their readers that media coverage of anti-Iraq war protests are examples of the culture war. But as for the actual war itself, virtually nothing. The NYT reporting on marriage is a Clear and Present Danger, but an actual war with actual deaths started by people like George W. Bush (born-again Christian), Donald Rumsfeld (another fundamentalist Christian), and Dick Chaney (what a surprise, another devout Christian) is barely remarked on.

But I digress... unfortunately it isn't just the axe-grinding fringe upset with the NYT for reporting facts which disturb the conservative blinkered worldview. Even the mainstream Boston Globe (deliberately?) misunderstands the NYT story, writing:

...how does the Times reach a contrary conclusion? By excluding from the category of women with husbands the "relatively small number of cases" -- in fact, it's more than 2 million -- in which "husbands are working out of town, are in the military, or are institutionalized." That startling Page 1 headline is true, in other words, only if the wives of US troops at war are deemed not to have husbands.

Sheer and utter nonsense. The NYT never said they didn't have husbands. It said they were living without a husband. That's a legitimate distinction: a legally separated women still "has" a husband, since she's still legally married, but she's clearly living without a husband. Although the reasons are different, wives whose husbands are in jail, or in Iraq, are likewise living without a husband -- just like the NYT says, in plain and simple English.

The NYT takes care not to confuse "living without a husband" with "unmarried", while the conservative pundits deliberately confuse the two. And let's face it, if they didn't, there would be no story here -- except, naturally, the original story that more than half of all women of marriageable age in the USA are now living without a husband. Whatever the reasons for this social change, whether the true figure is 51% or 50% or 49%, it is a significant social change from the "Good Old Days" when girls married at 16 or 17 and stayed with their husband until one of them died.

The Boston Globe makes a point of noting that the "relatively small number of cases" mentioned by the NYT is in fact two million women. Two million people is a lot of people to invite to a party. But compared to the approximately 120 million women of marriageable age in the USA, "relatively small number of cases" is a completely accurate description.

Yes, that one or two percentage points tips the number of women living with their husband from (approximately) 51% to 49% -- which is the whole point. World War One lasted for only four years, and the social changes set in motion by women going into the workforce are still taking place today. The invasion of Iraq is a few weeks short of four years old, and with no real sign (just a lot of talk) of it coming to an end any time soon. Why is the Boston Globe so resistant to the idea that it too might have social consequences?

Probably because conservatives, by definition, are conservative, and social changes are disturbing and frightening to them.

We can see where the Boston Globe is coming from:

Taken at face value, that's a pretty disquieting statistic. If society is to flourish and perpetuate itself, it must uphold marriage as a social ideal

Oh really? Says who? Does he really think people won't go to work to put bread on the table if they don't have a wedding ring on their finger?

What it looks like to me is that some pundits and journalists, like Peter Smith of Lifesite.net, have found a way of doing investigative journalism that has all the advantages of theft over hard work, with the extra advantage that it isn't illegal.

  1. Start by taking a perfectly responsible, honest, factually-correct newspaper article that discusses facts you disapprove of. Being responsible, it will almost certainly go into details of what facts they are actually talking about: e.g. the NYT story clearly mentions the temporary nature of the separation for women whose husbands are serving in Afghanistan or Iraq, and takes great care not to confuse "living without a husband" from "unmarried".

  2. Accuse the authors of the original story of making extreme claims which they never did. It helps if you attack an article in the NYT or other big commercial newspaper -- your "exposé" of their "malpractice" will still be on the Internet years after the NYT has pulled their article off the free list and made it by purchase only.

  3. Support your claims by tossing the article's own honest disclosures back at them. Take care to imply -- but not actually say -- that the original story never mentioned them.

The more honest the journalist, the more information he gives you to accuse him of dishonesty. Why investigate actual dishonest reporting, when it is so much easier to fling mud at honest reporters?

Thursday, December 21, 2006

Murder in Baghdad

One of the more egregiously stupid Internet memes (in the actual sense, not in the stupid LiveJournal sense of a quiz or survey) going around is the flood of chain letters claiming that Iraq is no more dangerous than Washington DC. If you've got pro-Iraq war friends, you've probably seen some of them. They use statistics to prove that living in Washington DC was -- and maybe even still is -- more dangerous than living in Iraq.

And by prove I mean mislead.

John Rogers of Kung Fu Monkey had something to say about the claims. Actually, he has two things to say: a nice version, and a not-so-nice version.

The nice version looks at the numbers, corrects the misleading use of statistics -- is it an accident that the pro-war zealots are averaging the death rate over chaotic murderous Baghdad and peaceful Kurdistan? yeah, sure it is -- and comes up with some conservative estimates for the yearly death rate in Baghdad:

544 murders per 100,000 in Baghdad versus 80 in Washington at the most lawless. (Today, Washington's murder rate is "merely" 35 per 100,000.)

Comparing apples with apples, Baghdad is at least fifteen times more dangerous than Washington D.C., and that only counts the risk of murder, not of kidnapping, torture or maiming.

One thing which surprised me was the equivalent figure for US troops in Iraq: 602 deaths per 100,000 per year. Given the low profile the media has for casualties, I was shocked that it is as high as that -- but I'm even more shocked that the risk to civilians in Baghdad is merely 10% lower than the risk to hated foreign occupiers.

The not-so-nice version... well, let's just say that Rogers' questions the sanity of anyone who unfavourably compares U.S. cities with a place where car bombs regularly kill hundreds of people, where virtually every single day the corpses of torture victims are found with electric drill holes through their bodies.

Rogers asks:

[...] denying reality Is. Not. Helping. We can't have a conversation about what to do next as long as a chunk of this country keeps clapping its hands and wishing hard. Hey, somebody wants to argue we need to stay for ten years, fine, lay it out. I'm open to the idea (I'm a Powell guy, what can I say). But how am I supposed to take this seriously when some people in the same breath try a bunch of statistical shell-games to show everything's just hunky-dory?

Tuesday, June 06, 2006

Lying with statistics

Mark Chu-Carroll from Good Math, Bad Math writes about Fundamentalists' dishonest use of statistics:

Woodmorappe wants to figure out how much space is needed [on Noah's Ark] by 16000 animals. How does he do that? Easy. He figures out an average amount of space needed by each animal, and multiplies it by the number of animals. Doesn't sound bad as a way of making an estimate, right?

Except... How does he figure out how much space is needed by each animal?

He figures out the median size of the animals in the ark; and determines that the median-sized animal would need a space of .5 by .5 by .3 meters - that is less than 1/10th of a cubic meter per animal. Multiply that by the number of animals, and you come up with 1200 cubic meters. Not terribly much space at all - which would mean that there would be plenty of room for food for the animals.

So what's wrong?

What's wrong is that the median is a dishonestly misleading figure to use.

In general, there are two major figures that can be used as an "average" value: the mean and the median. The median is the "middle" value, and is very good for estimating what a "typical" example of some collection is. For example, if you want a good idea of how much money a firm pays it's employees, the median is the best figure to use.

The mean is the "average" you probably learnt about it school: add up all the figures, and divide by the number of figures.

In this case, using the median to extrapolate to the total space needed on the Ark is dishonestly misleading. Here's a simplified example of why:

Suppose you've got to ship seven animals from one farm to another. How big a truck do you need?

You've got five rabbits, each about 15cm (6") tall, one sheep about 90cm (3') tall, and a cow 1.5m (4'11") high. The median is the middle value of 15, 15, 15, 15, 15, 90, 150 = 15cm, and is much more typical for our five animals than is the mean of 45cm.

To see why Woodmorappe's calculation is wrong, let's extrapolate from the "average" to the total height. That's what Woodmorappe is doing: he needs to know how big the Ark must be to hold all those animals. The right way to do it is to multiply the mean by the number of animals: 45*7 gives us 285cm, which is the total height of our seven animals. The wrong way, which Woodmorappe does, is to multiply the median: 15*7 which gives 105cm. That enormously underestimates the size need, and notice that the total height we calculate using Woodmorappe's method isn't even big enough for the cow.

Woodmorappe radically underestimates the size of the Ark. There are thousands of tiny animals, rat sized or smaller, which outnumber larger beasts like elephants not just in the numbers of individuals but in the number of distinct species (or, as the Fundamentalists prefer to say, "kinds"). Using the median, Woodmorappe extrapolates from the typical animal size (rat-sized) and effectively pretends that larger animals like apes, elephants, cows and lions take up no more room than a rat. Using the median in this way isn't a subtle error -- it is such an obvious error, such a strange thing to do, that it is hard to credit that it could be an innocent mistake.

(Good Math, Bad Math is moving to Science Blogs.)