Wednesday, May 21, 2008

Summon Monsters? Open The Door? Heal? Or Die?

It's the name of a Murakami book and exhibit, but it's also the question taunting online journalism.

Jason Calcanis thinks newspapers will die; newspapers themselves are trying to heal; and Adrian Holovaty wants to open the door.

But Digg, Reddit, and Hacker News have all chosen summoning monsters - specifically, trolls.

Here's my comment on a Hacker News thread about sites HN users want to see banned from HN:

I wish I was kidding. I'm absolutely serious. My life would absolutely be better if I never saw my own blog on Hacker News. I read Hacker News to discover new stuff I can benefit from reading. Nothing I've already posted is ever going to be a new discovery to me, by definition, and the comments my posts get on Hacker News are distracting and infuriating. Those comments are always new stuff, but they're very rarely new stuff I can benefit from reading. Once in a while I learn something or discover a new perspective, but these comments more usually just send me into episodes of Tourette's Syndrome.

Here's a small thread where I responded to critics on Hacker News:

I didn't really get anything out of posting that, or reading the comments that prompted it. The entire incident had no value at all, and probably shortened my life by at least a few seconds (anger and hostility have profoundly negative health effects). Many people have noticed this sort of thing, and it's pretty much inherent to the structure of blog commenting in general.

A funny thing about these sites is that they know about this problem. Hacker News is very concerned about not turning into the next Reddit; Reddit was created as a better Digg; and Digg's corporate mission statement is "at least we're not Slashdot." None of them seem to realize that the order from least to most horrible is identical to the order from youngest to oldest, or that every one of them was good once and isn't any longer. There's an entirely consistent and predictable theme here: devolution, and people who are surprised when it happens. It should be obvious, when it happens this consistently, that the decay isn't incidental; it's inherent. If it happens every time, for every site of this nature, it isn't coincidence.

Another thing which happens every time for every site of this nature is me telling sloppy thinkers to go to hell. People on Reddit and Hacker News think of me as a jerk, but there's a simple explanation for that: on Reddit and Hacker News, I am a jerk. This doesn't bother me much. There's a homeless couple who think of me as a saint, because I bought them a huge meal at Taco Bell. At that Taco Bell, on that day, for those people, I was a saint. Everyone has the capacity to be a jerk or a saint, so the smart thing to do is find contexts which maximize your saint potential and minimize your jerk potential. Obviously for me, neither Reddit nor Hacker News qualifies as such a context.

Unfortunately I spend more time at Hacker News than I do at Taco Bell. I go back to these sites anyway (albeit reluctantly, and with decreasing frequency), because, if you're willing to filter them dilligently, they provide a valuable and interesting stream of new information - but with terrible side effects. The side effects occur predictably in every variant of this approach to aggregating streams of news, because these Web apps are a terrible, terrible way of aggregating such streams.

Capturing the aggregate information about what a given community considers interesting is very useful; but it's insane to conflate what the community finds useful in aggregate with what each member of the community will personally find useful. Unfortunately, very many sites have made precisely this insane mistake - practically everyone except, in fact. Consequently we have a whole genre of sites with very predictable signal/noise decay.

This happens because the kind of filtering that a social networking approach to news gives you has some serious flaws. I'm mainly ranting about the Hordes Of Moronic Imbeciles Expressing Their Opinions About You Problem, but two other problems to consider are the Tim Bray Problem and the Cory Doctorow Problem. These guys are each more interesting to the developer community than they are to me, which puts them on my radar way more often than they should be, and in either case this causes a problem.

The Tim Bray Problem is that the developer community pays a lot of attention to Tim Bray, but I have never derived any use at all from any information connected to Tim Bray in any way, and after exposure to a lot of such information, I'm becoming very confident that nothing Tim Bray says about anything will ever make any difference to me one way or the other.

This man's hat is literally the only interesting thing about him at all. (It's a fedora.)

As aggravating as the Tim Bray problem is, it's easy to fix. You could solve it with a regular expression.

The Cory Doctorow Problem is related, but much thornier. The Cory Doctorow problem is that the developer community also pays a lot of attention to Cory Doctorow, like Tim Bray, and virtually everything Cory Doctorow says is absolutely useless in my opinion, again like Tim Bray, but unlike Tim Bray, Cory Doctorow occasionally delivers insight that is absolutely crucial and nothing short of brilliant.

Cory Doctorow's a lot like blog comments in general, in that extracting the signal from the noise nearly requires an element of masochism in your personality. His sparkling moments of genius, essential and unmissable though they are, elude you completely unless you're willing, for their sake, to sacrifice a lot of time to filtering out his gigantic standard daily quota of incredible time-wasting silliness.

Unlike Tim Bray, Cory Doctorow often annoys me. The Cory Doctorow Problem is worse for me than the Tim Bray Problem, because Tim Bray is a glass of water when I'm not thirsty, and Cory Doctorow is a winning lottery ticket buried under a hundred thousand yipping chihuahuas who all need to pee. I could really use that lottery ticket, but I really don't want to deal with those chihuahuas.

Both these problems present major obstacles to getting real value out of social networking news sites, but the worst obstacle of all is the system of upmodding and downmodding comments. Your search for news becomes derailed into a video game - anything which involves computers and scoring points ultimately becomes a video game - and as video games go, it isn't a lot of fun. It sure as hell isn't Tetris.

Worse yet, it's almost ubiquitous. If you're highly vulnerable to distraction, this game will steal your time. We should all know that it's an incompetent social networking strategy because it comes from Slashdot originally - a site which specializes in crashing innocent servers so that nerds have an excuse to swear at each other - but despite this horrible pedigree, it's a very popular approach. Every site which uses this approach inevitably ends up hated by former users, so it's kind of weird that the approach is so popular, but there's an easy explanation. The ultimate reason, of course, is a mistaken belief - specifically, the idea that user numbers are a better metric for Web app success than usefulness. This implies that anything which generates more traffic is inherently better. But purely for the sake of argument, let's abandon this 1997 mentality and ask ourselves how useful a site like Reddit or Hacker News can ever really be.

When you build a system where you get points for the number of people who agree with you, you are building a popularity contest for ideas. However, your popularity contest for ideas will not be dominated by the people with the best ideas, but the people with the most time to spend on your web site. Votes appear to be free, like contribution is with Wikipedia, but in reality you have to register to vote, and you have to be there frequently for your votes to make much difference. So the votes aren't really free - they cost time. If you do the math, it's actually quite obvious that if your popularity contest for ideas inherently, by its structure, favors people who waste their own time, then your contest will produce winners which are actually losers. The most popular ideas will not be the best ideas, since the people who have the best ideas, and the ability to recognize them, also have better things to do and better places to be.

Even if you didn't know about the long tail, you'd look for the best ideas on Hacker News (for example) not in its top 10 but in its bottom 1000, because any reasonable person would expect this effect - that people who waste their own time have, in effect, more votes than people who value it - to elevate bad but popular ideas and irretrievably sink independent thinking. And you would be right. TechCrunch is frequently in HN's top ten. Meanwhile, I just decided to test this theory, so I went to the absolute bottom of the Hacker News list of stories, and I found out Malcolm Gladwell's got a new book on the way.

The flawed design assumptions of sites in the Hacker News genre inherently attract trolls and inherently reward those trolls for polluting our minds with their groupthink. This is why racism and sexism do so well on those horrible sites. It's also why I want my blog banned from Hacker News. I'm not even kidding.

It's not coincidence. It's a fundamental design flaw. Building a site like Hacker News and then trying to keep it from being overrun with losers and nimrods is like coating your kitchen floor in powdered sugar and then trying to keep it clear of ants and earwigs. People are asking themselves how to save Hacker News, and they need to realize there is only one technology which can save Hacker News, and that is the time machine.

However, although this obviously aggravates me, I know what Nietzsche said about it: battle not with monsters, lest ye become a monster. I don't always remember this rule, but I observe it when I can. Railing against the madness of it all and getting my blog banned from Hacker News would accomplish a lot less for me than just putting down the Hacker News crackpipe and becoming one of those lovely people with a cool calm head and better things to do.

But there's an important caveat: just because people have things to do doesn't mean they can't learn things from the rest of the world. People with too much self-respect for negativity magnets still need a good way to get their news. I want to throw out the bathwater, not the baby.

I've come up with what I believe is a powerful solution, but unfortunately I don't currently have time to implement it. This may change in future. If so, I'll start it as a business. If not, I'll start it as an open-source project and spend time evangelizing it. One way or another, I'm going to introduce my fix to the world, because people shouldn't have to choose between flame wars and autistic mode.

Autistic mode, of course, comes from Ghost In The Shell, and there's no surer sign of fundamental brokenness in the systems geeks use to share information with each other than the fact that more geeks know about Cory Doctorow's dull, shallow, lazy, unimaginative, self-serving, and unbearably self-absorbed Down And Out In The Magic Kingdom than have seen the anime TV series version of Ghost In The Shell - true-school hard sci-fi cyberpunk which is infinitely superior to both movie versions, and on a level with the original manga.

Cory Doctorow has some great moments, but if you say "science fiction" to somebody, and they think Cory Doctorow before they think Masamune Shirow, that means our information-sharing systems are polluted with bullshit, and I for one am tired of it.


Update 2:

I can't believe I didn't see this coming:

I don't have a problem with Tim Bray - I have a Tim Bray Problem. I don't always choose my words as carefully as I'd like, but as far as these specific words are concerned, I intend them very precisely. Likewise, even though I obviously have a bee in my bonnet about Doctorow's Magic Kingdom book, I expect to like Little Brother when I finally get the time to read it. It's not even slightly personal.

Update 3:

If you've been on the Internet for a while you probably recognize a number of tactics in this post as blatant trolling.

The core argument I'm making here is that fundamental design flaws in sites like Hacker News make them extremely vulnerable to trolling.

Logically, therefore, if my argument is correct, you would expect this piece of trolling to do well on Hacker News.

It was in the top 10 yesterday. It's still in the top 10 today. Quod erat demonstrandum.

My experience with blog posts which address flaws in social software that encourages unhealthy interaction patterns between its users is that these blog posts are often interpreted as personal attacks and rarely as serious analysis. But consider the serious analysis. The flaws are two: one, the voting/registration systems reward trolls and bury great information; two, the commenting systems turn discussion into a video game. If you pay attention to these two flaws, you don't even need me to describe my secret planned technological solution, because it's already obvious.