Looks like April Fools but aren't

Wikipedia's "In The News" for April 1st looks like it's nothing but April Fools pranks, but in fact all the stories are true -- despite Faux News' usual quality reporting:

Every item on the home page of the user-generated site Wikipedia is fake. The featured article is about the "Museum of Bad Art" in Boston. The headlines include such stories as NASA monitoring diamonds falling from the sky and the Irish prime minister streaking in public — both of which barely stretch real recent news events.

In fact every one of those is a legitimate, real news story. The April's Fools prank was to fool people into thinking the stories were pranks:

• Ireland's Taoiseach [President], Brian Cowen... is seen publicly naked in Dublin, following months of economic uncertainty.


• NASA reports a shower of diamonds from the sky.


• German scientists unearth a row of suckers belonging to an ancient order.


• A revolutionary new online tanning service receives one million hits within two months of being established.


• Henry Allingham of the United Kingdom credits cigarettes, whisky and wild, wild women for his seemingly impossible longevity.


• A newspaper discovers that pay-per-view porn is amongst a number of unusual things being purchased by British MPs on their claimed expenses.


• The merging of Hartford and New Orleans is found to have severe environmental consequences.

For those who don't know their US geography, Hartford, Connecticut is about 1424 miles away from New Orleans, Louisiana.

The real stories:

The nekkid Taoiseach: an artist snuck naked portraits of Brian Cowen into two of Ireland's most prestigious art galleries.

The NASA shower of diamonds: a meteor that exploded over the Sudan included nano-diamonds in the fragments remaining.

Row of suckers: the discoveries of three ancient extinct octopus species.

Online tanning service: a viral PR campaign to alert people of the dangerous of tanning salons.

Cigarettes 'n' whisky: Britain's oldest man, and the oldest surviving World War One veteran, really did credit his longevity on cigarettes, whiskey, wild women... and a good sense of humour.

Politician claiming pr0n expenses: Come on now, are you really surprised?

And the merger of Hartford and New Orleans actually refers to the collision of two ships.

Wikipedia's "On this day" for 1st of April are amusing too. Go check them out here.

Wikipedia 1, China 0

China has demanded that Microsoft, Yahoo and Google all censor their web applications or be banned. All three software giants -- reluctantly in the case of Google, seemingly willingly in the other two cases -- complied.

China also demanded that Wikipedia censor itself for Chinese viewers. Wikipedia refused, and the powers-that-be in Beijing responded by blocking access to Wikipedia from within China.

Or rather, they tried to block access. But it seems that China needs Wikipedia more than Wikipedia needs China -- after barely a month, China has stopped blocking Wikipedia, instead concentrating on the cat-and-mouse game of trying to block only certain sensitive (that is, embarassing to the political leaders) pages, like those on Tiananmen Square.

China is desperate to catch up to the West, and that means accessing our knowledge banks, especially the Internet. If China can't afford to block Wikipedia, they certainly can't afford to block Google, Microsoft or Yahoo -- let alone all three. China is bluffing with a pair of twos, and unlike Wikipedia, the three software giants didn't have the cajones to call their bluff.

More Wikipedia bashing

If it wasn't so frustrating, it would be amusing to see Stephen J. Dubner of Freakonomics and his frequent efforts to bag Wikipedia.

He's taken up prefacing every attack on Wikipedia with a explanation of just how much he likes it, no really, he likes it just fine. This time, he gloats that the US Patent Office has removed Wikipedia from their list of approved sources of information when they have to check a patent application. Apparently, the US Patent Office has, up until now, been using Wikipedia as an official source "for years", according to critic Greg Aharonian.

I smell a rat. Wikipedia wasn't even close to ready for prime time as recently as two years ago; it's arguable whether or not it is ready now (although I would argue that, used with care, it is). It certainly wasn't heavily in the public eye two years ago. I find it difficult to credit that an organisation as conservative as the US Patent Office would have been using Wikipedia in 2004.

But what the hey, maybe it was an official source. Stranger things have happened.

Dubner writes:

The argument in defense of Wikipedia that I find most troubling is that it is self-correcting and self-policing, which is to say that, Hey, in the end all the mistakes and vendettas get fixed by caring and level-headed people.

If being self-correcting and self-policing (rather like the free markets Dubner champions as an economist) is a bad thing, what's the alternative? An official government fact-checking agency? We could call it The Department of Homeland Truth, and make sure that only things that get approved by the bureaucrats in the government are printed.

Perhaps not.

Maybe Dubner thinks that the self-correcting model is a bad one, and that the correct model is something like that of the Encyclopaedia Britannica, which is ... self-correcting and self-policing. Hmmm.

So what is his point? As an economist, he surely knows that all of science (including the Dismal Science, economics) is self-correcting and self-policing, and operates by concensus rather than fiat.

Science and knowledge advances according to what is said, not by who says it. Britanica has got a reputation to uphold, and it may be that we can trust that they value their reputation enough that we should treat the facts they publish as trustworthy; but ultimately we have to take their word for it. In other words, we believe Britannica because of who they are. Wikipedia instead exposes the whole messy business of deciding what's true and what isn't, and that is far more valuable, even more valuable than the facts themselves. If Wikipedia says something is a fact, we can (if we take the time and effort) see exactly why we should believe it.

Dubner has made an empirical claim here. He stated:

The problem, of course, is that if someone happens to read or cite a Wikipedia entry at a moment when all those things haven’t been fixed, which is obviously a vast, vast, vast majority of the time, then the mistakes get promulgated as fact.

Dubner claims that, on average, supposed "facts" in Wikipedia are wrong not just the majority of the time, not just the vast majority of the time, but the vast, vast, vast majority of the time.

I'm not sure what the difference between "vast majority" and "vast, vast, vast majority" would be. 51% is clearly "the majority", and it stretches the common meaning of the words "vast majority" to apply it to anything less than a four-fifths majority. Three vasts surely means something like 999,999 in a million, but let's give Dubner the benefit of the doubt and make it a nice low 90%. So Dubner believes that, overall, any supposed fact on Wikipedia will be wrong at least 90% of the time.

That's an empirical claim that can be tested. Choose (say) one thousand random Wikipedia entries, and count the number of supposed facts in those entries. If more than one in ten are correct, then Dubner is full of sh.. mistaken.

Of course, not all Wikipedia entries are of equal quality. I dare say that there are obscure "facts" on pages that have never been checked since they were created that are wrong; no doubt there are entries with all the trustworthiness of gossip magazines. So what? Lazy and incompetent researchers will treat bad sources as if they are good sources, Wikipedia doesn't change anything there. Ten years ago, people used The Little Bumper Book Of Fun Facts For Tiny Tots as their major reference, and now they use the Internet the same way. Bad reseachers will treat anything they read as infallible, not just Wikipedia. If you are doing "real" (i.e. important) research, you should not be relying on any single source of information, whether it is Wikipedia or Britannica, because no single source is error-free. Good researchers will treat Wikipedia as it is meant to be used: look for citations, and follow the link to verify the facts. And unlike most other secondary sources, Wikipedia warns you when caution is needed.

Wikipedia and science

It never ceases to amaze me how people just don't get Wikipedia. Not just any old people, but those who you would expect to grok the concept often have the most difficulty with it.

Case in point: economist Stephen J. Dubner of Freakonomics has written a blog post entitled "More Welcome Ridicule For Wikipedia". A tad hostile, don't you think?

First he damns it with faint praise:

Wikipedia is generally fun, sometimes useful, often entertaining. What it isn’t is very dependable, for the very reason that makes it fun: it is an encyclopedia whose content is generated by random contributors.

Then he links to a "a better job of ridiculing Wikipedia than we could ever dream."

Hardly the first time Dubner has attacked Wikipedia. What's prompted this new attack?

Comedian Stephen Colbert ran amuck on American television, attacking and defacing Wikipedia entries. Hardy har har.

Of course, neither Dubner nor Colbert mention that the system worked: Colbert's vandalism was detected rapidly and reversed.

I'm forced to wonder what it is about Wikipedia that's eating Dubner. The Wikipedia system has striking similarities to the way free markets work. Anyone can go into business as a supplier of information in Wikipedia, just as anyone can go into business as a supplier of goods and services in a free market. In both, these suppliers vary greatly in quality; in both, there is no centralised "expert" who decides what is good and what is bad; in both, there are problems with cheats and vandals. One shouldn't discount the free market and institute central planning because of problems with, say, polluters. One creates systems that control or limit the problems. Free markets have laws which punish polluters, or "sin taxes" which make polluting industries pay for their anti-social activities. Wikipedia has revision control.

As an economist, no doubt Dubner could talk for hours about the emergent properties of free markets, and how the "random contributors" to the market create dependable supply and demand of goods and services without any central planner "expert" who decides what suppliers should supply or where investors should put their money. So why is he so resistant to the idea that "random contributors" to Wikipedia can lead to emergent properties of reliability, accuracy and accountability?

Had Stephen Colbert run a coffee shop and, for a lark, started putting powerful laxatives in the food and drink he served to customers, would Dubner claim this was "welcome ridicule for capitalism"? No of course not -- he'd surely point out that market economies don't collapse because of the actions of a few malicious or stupid sellers, and that buyers would rapidly learn that Colbert was providing tainted goods and they'd avoid his store, sending him broke. The free market's feedback loops are slow: likely it would take weeks or months before Colbert's dodgy store was shut down. Compare that to the Wikipedia system, which locked the vandalised articles within twenty minutes. (Of course, not all vandalism is detected that quickly. The higher the profile of the article, the more likely vandalism will be detected quickly; but the lower the profile, the less likely anyone will bother vandalising it.)

But there is another similarity to Wikipedia's open content that is even stronger than the free market. That is science. Like Wikipedia, science operates by consensus and peer review. There are experts in science, but it your peers' respect that proclaims you "expert", not merely your paper qualifications -- there are many Ph.D.s but they are not all held in the same regard, and some of the greatest scientists never held formal qualifications. Likewise, not all Wikipedia editors are equal. There is no central committee that decides scientific truth, instead there is an open market in ideas, just like in Wikipedia.

Science is transparent: much of the scientific consensus emerges from public debates in journals which are open for anyone to subscribe to. But while science also includes private debates behind the closed doors of offices and laboratories, Wikipedia is completely open: virtually everything is debated in public, with the debates recorded in the Talk pages and the editing history of the articles recorded for anyone to see. Wikipedia is even more transparent than science. For those who wish to take the time to dig under the surface, Wikipedia gives the reader the means to judge the reliability of every sentence.

You can't say the same thing about Encyclopædia Britannica. You're reduced to a take-it-or-leave-it attitude towards the articles: you either trust everything in the encyclopædia, or you trust nothing. You either trust Britannica's status as an expert source, or you don't. You can't easily determine whether this particular editor or that writer has an axe to grind, you simply have to trust that they don't. (Britannica has a good reputation for a reason, so that trust is probably not misplaced -- but you're still taking them on trust.)

Like science, Wikipedia emphasises how do you know rather than who are you -- and I suspect that's what is eating Dubner. The old authoritarian impulse at work perhaps? "Obey my authori-tay!" Both science and Wikipedia demand references, and reliable ones at that, and give no credence to people based only on "qualifications". Both operate through peer review rather than executive fiat. That's hard for some people to deal with, even some scientists.

There's a further similarity between Wikipedia and science that is amusing. In the early days of the Royal Society, there were many amateurs -- actually they were all amateurs -- and meetings often became extremely heated as people passionately argued at each other for their own unsupported ideas. Sounds just like Wikipedia during "edit wars". Nevertheless, from this unpromising start, the scientific demand for evidence rather than "expert authority" won out. In fact, a close look at the history of science shows that periods of reliance on authority have resulted in a reduction in reliability and accuracy, and for much the same reason as centralised planning usually leads to inefficient allocation of resources.

Free markets work because of, not in spite of, lots of ordinary business people. Science works because of, not in spite of, lots of ordinary scientists who aren't neccesarily leading experts in their fields. And likewise, Wikipedia doesn't depend on experts, although experts are always welcome for the depth of their knowledge and their understanding of the need for evidence and references. Wikipedia doesn't need experts because of who they are, but because of what they know. Anybody who knows what an expert knows, even if it is only a tiny little bit of what an expert knows, can contribute and help the incremental emergence of knowledge.

What really makes Wikipedia great

There's been a lot of recent discussion about the Hive Mind and when collectivism is wise. For the Internet-savvy generation, the two best examples of collective wisdom are Google and Wikipedia. I would hope that Google needs no support -- it is by far the best search engine available today. But Wikipedia is frequently criticised for equating the opinions of amateurs -- and not very good amateurs at that -- with the opinions of experts.

But there is a factor which people have not considered. While Google is collectivist, it is opaque -- Google's Page Ranking algorithm is secret. We cannot criticise or test Google's results, only take them as revealed wisdom.

But Wikipedia is the opposite: it is completely transparent. You aren't forced to choose between accepting Wikipedia's articles as revealed wisdom, or rejecting them all together. Instead, you can click on the "discussion" button on each page, and see for yourself the discussions, arguments, disagreements and agreements that led to the article. Where the stakes are high, the wise person will not only read the article, but take into account the article's history and those who wrote it. Was this section written by somebody with an axe to grind? Is this point controversial? Does this authour have the respect of other authors, or is he a loose cannon with an agenda? All that information is publically available in Wikipedia.

In that regard, Wikipedia reminds me of that other great example of collectivism: the scientific consensus. In science and mathematics, it doesn't -- or at least shouldn't -- matter whether you are a self-trained amateur or Mr. Establishment himself: it's what you say, not who you are, that is important. The scientific consensus is completely open and transparent (at least in principle) -- anyone can borrow or subscribe to the appropriate scientific journals and follow the debate between scientists as they nut out a collectivist opinion.

In that regard, I see the greatest feature of Wikipedia is its transparency: unlike other encyclopedias, with Wikipedia you don't have to accept the wisdom of the article on trust, but can see for yourself where it came from.

Previous posts on this subject:
The Hive Mind is stupid and boring -- why do we love it so?
The Hive Mind, part II

The Hive Mind, part II

A week ago, I discussed The Edge's Jaron Lanier and his essay about the Hive Mind.

The Edge has published a set of responses to Lanier's essay here. My favourite comes from Cory Doctorow:

Wikipedia isn't great because it's like the Britannica. The Britannica is great at being authoritative, edited, expensive, and monolithic. Wikipedia is great at being free, brawling, universal, and instantaneous.

See also comments on Boing Boing.

The Hive Mind is stupid and boring -- why do we love it so?

Jaron Lanier, writing for The Edge, writes about our infatuation with the Hive Mind.

As somebody who believes strongly in the advantages of such examples of collectivism as Wikipedia, Google's page ranking, the Open Source philosophy, and the benefits of free and open competition, Lanier's argument appeared provocative (if not downright ignorant) to me -- at first glance. But after getting past the first part of the article, and actually reading it through to the conclusion, I realised that he wasn't arguing against the wisdom of the crowd, but warning that crowds can easily turn into mobs, which are notoriously stupid. (The intelligence of a mob, it has been said, is that of its stupidist member divided by the number of members.)

There are many examples of collectivist behaviour where the crowd is more intelligent than the smartest individual. It is the basis of capitalism: the free market is often better able to allocate scarce resources than the wisest autocratic ruler, let alone some out-of-touch bureaucrats. It explains why the scientific consensus is so often right, when individual scientists so often have feet of clay and irrational foibles.

But likewise, crowds are not always wise, the free market is not always clever (witness Tuplip Mania, the South Sea Bubble, and the Dot Com Boom), and sometimes it takes an individual genius to shift the scientific consensus. The Hive Mind (or the Borg Collective if you prefer) is dumb precisely because it is made up of many individuals with but a single opinion. Crowds are wise when they contain opinions and free competition. Lanier's article, despite its title and provocative opening, is not so much warning that crowd decisions are stupid as reminding us of that sometimes they can be, and that we, as individuals, need the sense to tell when to listen to the crowd and when to seek out individual experts.