A few years ago – in 2006, to be exact – I wrote a scraper to crawl Amazon.com’s affinity links for The Economist. Think of affinity links as the basis for Amazon.com’s recommendation engine. They’re the links at the bottom of each page with headings like “People who subscribe to The Economist also subscribe to…” These links give you a recommendation: If you like The Economist, you’re also likely to be interested in, say, Foreign Affairs or The New Yorker rather than Guns & Ammo or Mother Earth News.
I wrote the spider in Perl (though since then I’ve moved on to Python, executing my scrapers on the great ScraperWiki site). Once I had the data, I put it into Pajek – a wonderful network visualization program out of Slovenia’s University of Ljubljana – and gave the resulting diagram to an artist over at The Economist.
The board of directors over at The Economist loved this diagram because it showed their magazine as a bridge among high-end specialist publications. (Just avert your eyes from Wired, which has a similar claim.) It’s exactly what a sophisticated general interest newsweekly should be.
But much more came out of this exercise than a flattering diagram for The Economist. How is Martha Stewart Living connected to Soldier of Fortune? You’ll have to talk to me to find out. Or maybe dig a little through the older posts of this blog.