The spread of link rot

By Felix Salmon
June 28, 2013

When Anil Dash lamented, last December, about the web we lost, he wasn’t speaking literally — he was talking about a culture which got swept away by a tidal wave of Silicon Valley money. But with today’s news that Google seems to be about to vaporize a significant number of the blogs on its Blogger platform, it’s becoming increasingly clear that the problem of link rot isn’t going away — if anything, it’s getting worse.

I’m a great believer that once something is placed on the internet for free, it should continue to stay there, for free, unless there’s an extremely good reason to delete it. Back when hosting websites was difficult and expensive, that was easier said than done. But now web hosting is effectively free, there’s really no excuse — and one might hope that, as a result, we’d see less link rot.

But that’s not what’s happening. For one thing, the institution of the permalink is dying away as we move away from the open web; if you’re not even on the web (if, for instance, your content comes in the form of a show on Netflix), then the very concept makes no sense. What’s more, we’ve moved into a world of streams, where flow is more important than stock, and where the half-life of any given piece of content has never been shorter; that’s not a world which particularly values preserving that content for perpetuity. And of course it has never been easier to simply delete vast amounts of content at a stroke. (For instance: the Kanye West and Alec Baldwin twitter feeds.)

The Wikipedia page on link rot says (at the time of writing) that “permalinking stops broken links by guaranteeing that the content will never move” — but in the real world that’s not much of a protection at all. Content management systems change, and when they do, many publishers don’t bother to ensure that the old links still work. (Which is why, for instance, old links to Gawker tend to die, even though the website is still going strong.) And of course permalinking can’t prevent an entire blog from getting deleted — as Google is now threatening to do with certain adult sites.

Small personal blogs die every day, of course, but it’s no protection being owned by a huge media company, either. My boss, Jim Ledbetter, used to edit a site called The Big Money, which was unceremoniously killed off by the Slate Group, its archives lost to history; more recently, Thomson Reuters did the same thing to one of their sites, News and Insight. (The press release announcing the move was one of its victims; a shadow of it lives on here.) When these decisions are made, the fate of the archives never seems to matter; the result is thousands more dead links scattered across the internet every day, pointing to once-valuable resources which no longer exist.

These mass deletions are huge; they make me feel almost sheepish about the anger with which I greeted, say, Greg Mankiw’s decision, back in 2007, not only to close his blog to comments, but at the same time to delete all the previous comments which had been made, with no warning. All the conversations which had taken place in his comments section, all the smart rebuttals which had been made — all of them just disappeared, overnight. Today, I’d barely blink at such a thing: after all, it happened to me, a couple of years later, when all the comments on my Portfolio blog got deleted.

What I fear is that the entire web is basically becoming a slow-motion Snapchat, where content lives for some unknowable amount of time before it dies, lost forever. Sites like archive.org can’t possibly keep up; and the moguls who own most of the content online are simply not invested in the ideals of the link economy. When even Google is giving bloggers just three days to save their sites or see their content disappeared — three days when many of them are on summer vacations, no less — it’s pretty obvious that there’s no such thing as a truly benign online organization any more. There may or may not be one or two, at the edges; I have a decent amount of faith, for instance, that the BBC is going to honor its permalinks for many years to come*. At the other end of the spectrum, Anil will, as well. But for those of us who make our livings linking to other things on the internet, it’s simply a fact of life that most of our links will die. If, that is, our own permalinks aren’t killed off first.

*Update: Or, maybe not!

More From Felix Salmon
Post Felix
The Piketty pessimist
The most expensive lottery ticket in the world
The problems of HFT, Joe Stiglitz edition
Private equity math, Nuveen edition
Five explanations for Greece’s bond yield
Comments
13 comments so far

Mankiw deleted all the comments on his blog? Too bad we can’t go back in time and delete all the advice he gave W.

Posted by KenG_CA | Report as abusive

You know what’s really outrageous? There are billions of unrecorded conversations every day, and the people having them don’t even have the decency to permalink them.

If the web is more and more a part of the fabric of normal life, why should we expect everything happening to be archivist-friendly?

Posted by PhilipW | Report as abusive

Your optimism about BBC does not extend to major American news organizations. When NBC News discontinued its partnership with Microsoft, all the URLs for its archived Nightly News videostreams at msn.com and at Bing.com became dead links.

Posted by AndrewTyndall | Report as abusive

You’re clearly just getting half the story on what Google is doing with blogger blogs. I’m a little worried about you, because it seems that lately you’ve had your eyes just partially open, as if you’re anti-Google.

What Google actually did was target people who were using Blogger to make money off adult content linking — essentially monetizing their Blogger blogs without adhering to the TOS that directs users to use AdSense.

I understand some people are upset that they cannot make money off porn via Blogger, but all they need to do is move to WordPress or find a host and build their own blog using WordPress templates, Drupal or Joomla, etc. There are a lot of choices out there, and people pretending as if what Google has done is some great evil, is a bit pathetic.

When they killed Reader, that was something worth getting upset over. This — Blogger’s exclusion — is not.

Posted by GRRR | Report as abusive

I have two conflicting notions. First, my personal one, which is: Archive everything. Save everything. PDF everything. I never trust that something’s presence today implies an expected presence tomorrow.

Secondly, on the macro scale, an opposite feeling: The web’s ephemerality both standard now, and inherently useful — when everything is saved for posterity, nothing is saved for posterity (except the enormity of our archival impulse).

I remember a statistic from a year after the Egyptian revolution, that something like 20% of the Tweets on the topic had since disappeared in one way or another. Which is why a paperback edition of Tweets from Tahrir is so valuable a history.

Posted by dmcdougall | Report as abusive

“Google seems to be about to vaporize a significant number of the blogs”

Do you actually have any idea what that number is and how significant it is, or did you just pull that out of your behind?

In academia at least, there’s a movement to using DOIs instead of direct links to maintain pointers to online published material. This prevents link rot (at least as long as the DOI servers themselves are maintained), but doesn’t necessarily guarantee access. In ye olde days of course, if a book went out of print or a magazine died, copies would continue to exist in libraries.

Posted by Moopheus | Report as abusive

Unbelievable. Felix, who already thinks the world should provide him unending music and video for free, now believes it’s someone else’s responsibility to provide him the entire web, 24/7 for all eternity, in CASE one day he happens to think he wants to get some random piece of information from who knows how many years ago.

Felix, it’s never been easier to make PDFs. If you want to save something, save it and stop whining that someone else should do it for you.

Posted by EconomistDuNort | Report as abusive

“These mass deletions are huge; they make me feel almost sheepish about the anger with which I greeted, say, Greg Mankiw’s decision, back in 2007, not only to close his blog to comments, but at the same time to delete all the previous comments which had been made, with no warning”

The differences being that (a) Mankiw did this deliberately, and (b) he did it because he lost so many arguments.

Posted by Barry_D | Report as abusive

Archive.org has approximately all of Mankiw’s comments and will probably eventually have approximately all of the adult content about to be deleted. I have even installed discontinued software from Archive.org before. It would be nice if it were indexed instantaneously, but it accomplishes much of what you are wishing for.

Posted by RHewitt | Report as abusive

Apart from actual deletion, web browsers (especially Firefox) are also dropping support for older web standards, so older websites won’t be displayable on those browsers. Mostly at the moment it just stops them displaying correctly (which can mean text the same colour as background).

Posted by ChrisWheeler | Report as abusive

I thought the NSA was archiving all of our stuff.

Posted by rokid | Report as abusive

This is a test of the emergency re-clueing system. You are about to hear the sound of someone Missing The Point so hard that he can leap over church steeples without discomfort.

“Felix, it’s never been easier to make PDFs. If you want to save something, save it and stop whining that someone else should do it for you.”

This has been a test. Had this been an actual re-clueing, someone with more patience than I have would have gently explained the hard parts to poor Nort.

Posted by FredFnord | Report as abusive

The communistic cloud cannot be avoided. You see that blogger blogs aren’t “safe,” in that they can be deleted any time. It happened to my first blog. Many wonderful years of posts vanished overnight. I did manage to save the first two years using software called HTTracker. Those two years of chronicles now exist only on the harddrive of an old Windows 95 computer. With thousands of posts on my current blogspot blog, which I have kept since 2006, I no longer fear deletion. I just accept the fact that it could happen anytime. I started a new wordpress blog, and I will admit my whole approach to and awareness of blogging has changed dramatically since (1) my blog was deleted (2) all this ‘cloud’ stuff arrived on the scene and (3) I realized and now understand just how powerful entities like Facebook and Google are.

We netizens are like ants. When one of us “dies” there is always a replacement nearby, left to carry on.

Posted by DaveLucas | Report as abusive
Post Your Comment

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/