Algo-powered curation

By Felix Salmon
January 12, 2011
Paul Kedrosky has a great little essay today on the way that curation is set to be the new search, since the utility of the old search is steadily declining.

" data-share-img="" data-share="twitter,facebook,linkedin,reddit,google" data-share-count="true">

Paul Kedrosky has a great little essay today on the way that curation is set to be the new search, since the utility of the old search is steadily declining.

Google’s ranking algorithm, like any trading algorithm, has lost its alpha. It no longer has lists to draw and, on its own, it no longer generates the same outperformance — in part because it is, for practical purposes, reverse-engineered, well-understood and operating in an adaptive content landscape.

Kedrosky rightly points to Twitter as one powerful source of curation, but there are and will be others:

The re-rise of curation is partly about crowd curation — not one people, but lots of people, whether consciously (lists, etc.) or unconsciously (tweets, etc) — and partly about hand curation (JetSetter, etc.). We are going to increasingly see nichey services that sell curation as a primary feature, with the primary advantage of being mostly unsullied by content farms, SEO spam, and nonsensical Q&A sites intended to create low-rent versions of Borges’ Library of Babylon.

I’m enthusiastic about this to the point of being downright conflicted: I’m involved in putting together one such nichey service myself (on which more anon), and can’t wait for a whole ecosystem of such things to arrive.

Up until now, publishers have tended to be suspicious of anything which is built on the idea of sending people away to algorithmically-generated corners of the internet which aren’t all carefully vetted. But if you can get over that conceptual hurdle, there’s a very exciting world out there which has barely been explored at all—a world of vibrant websites whose content is powered by algorithms but still produced, curated, and edited by very smart humans. Techmeme is a good example; there will be many, many more.

More From Felix Salmon
Post Felix
The Piketty pessimist
The most expensive lottery ticket in the world
The problems of HFT, Joe Stiglitz edition
Private equity math, Nuveen edition
Five explanations for Greece’s bond yield
Comments
7 comments so far

I would refer to twitter for curation as often as I turn to a USA Today or Fox news poll to find out how to vote.

Posted by OnTheTimes | Report as abusive

OnTheTimes, so you must not be a) following the right people on Twitter and b) using the right tools to gather the links they’re dropping there. That’s can be a cumbersome project still, admittedly, but in certain areas like finance, media and politics – where there’s an active twitter community linking to the best stuff – it’s by far the best way to keep up with quality work/discussion in real time.

Felix, anxious to see what you have cooking – something along the lines of The Daily Curator, perchance?

Posted by MickWeinstein | Report as abusive

This is interesting. Yahoo started out not as pure search, but as a directory.

In January 1994, Jerry Yang and David Filo were Electrical Engineering graduate students at Stanford University when they created a website named “David and Jerry’s Guide to the World Wide Web”.[4] David and Jerry’s Guide to the World Wide Web was a directory of other web sites, organized in a hierarchy, as opposed to a searchable index of pages. In April 1994, “David and Jerry’s Guide to the World Wide Web” was renamed “Yahoo!”.[5][6] The yahoo.com domain was created on January 18, 1995.[7]

Then search got better and the curated aspect seemed less important. Can yahoo recapture it’s original value by saying it’s no longer a search engine? Is bing going about its advertising wrong by associating itself with search at all?

Posted by Zdneal | Report as abusive

@Mick, “cumbersome project”? Full-time job is more like it.

I think there is some value in twitter, but it takes an enormous amount of filtering to extract it.

Posted by OnTheTimes | Report as abusive

@OnTheTimes, follow some smart people on Twitter who like to drop links and comment on them, then use http://tweetedtimes.com to gather those links and comments – that’s not so cumbersome

Posted by MickWeinstein | Report as abusive

Felix, interesting post. You’re certainly correct in your assessment that the entire curation ecosystem is growing and along with it there has been an will continue to be a proliferation of interesting tools that people can use to help filter, organize, archive discover and share content on virtually any topic.

How useful any one tool might be depends heavily upon the intended use. Some are designed to help curate (or in my own opinion) more artfully filter real time streams – most especially twitter however I think this form of curation is actually closer to aggregation unless those lists of tweets and other content and subsequently perused by a person and further refined. In fact I would go so far as to argue that a purely algorithmic effort at filtering content should really be described as aggregation and not truly curation.

The way I see it you wouldn’t have an algorithm choose the most important 10 pieces of French Impressionist art out of a 10,000 item collection. Could an algorithm really determine not only why they were the most important pieces but also make decisions about how to display them in a way that places them in context with one another as well as within the larger context of the entire period?

Perhaps the day when that will happen isn’t far away but as someone that has followed our progression towards Ray Kurzweil’s “Singularity” it is pretty clear that we’re not there yet.

On the other hand humans are already very skilled curators. In fact, virtually every one of us curates something. Granted, most of the stuff we curate may not be to your liking – you may not be as fond of ceramic dogs as my mom is, for example, but that doesn’t mean her collection is any less well curated for that fact – but when you do seek content on a particular topic it’s highly likely that if you look for material curated by an expert in that area it will deliver far greater value for the time spent to locate it than you are likely to realize using search engines to find material on the same topic.

As an example, take a look at content curated within Pearltrees (disclosure, I am the Chief Evangelist for the company) – here a a couple of links to pearltrees on topics that have received recent attention:

Apple Minus Steve Jobs: http://bit.ly/ecBIed
WikiLeaks: http://bit.ly/gZhgJH
Star Trek: http://bit.ly/eBuIPR

I think you’ll find that humans do a much better and far more context sensitive job of curating content – I think it’s for this reason that Gabe Rivera (techmeme founder) has always hand picked the sites that Techmeme uses for its principal index and it is also why he now retains a team of people to manually curate (at least in part) the content that now appears on the site.

Perhaps one day machines will do this job as well or better than people but for now I think machines are still more aggregators while humans are the true curators on content both physical and digital.

For some additional thoughts on this topic, please check out Tom Foremski’s excellent piece on SiliconValleyWatcher

Posted by ostarr | Report as abusive

Hi Felix,

You’re right algorithmic curation is definitely an interesting area at the moment especially with real time content.

If you’ve not already heard about DataSift it sounds like it may be something of real interest to you, DataSift is a real time content curation engine for the social web. It has multiple input sources and a range of different augmentation options and you have control of how the data is curated through our CSDL (Curated Stream Definition Langage).

If you are interested then do sign up to our Alpha programme. http://datasift.net

Many thanks
Sarah

Posted by DataSift | Report as abusive
Post Your Comment

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/