Comments on: Is the long tail being undermined by an online monoculture that is ‘worse than mass media’?

By: tom s.

tom s. — Fri, 13 Nov 2009 18:42:39 +0000

Thanks for the response to the article. I hope it’s clear that I was trying offer a step up from the all-too-popular anecdotal evidence for recommender systems’ long tail effects, rather than any kind of proof of their inevitable behaviour. There are just too many ways to implement recommender systems for any comprehensive statement of that kind. I suspect the outcome will have more to do with the incentives of whoever is running the system than with its technical details.

That said, as G. Manfredi says, if you look at even the most successful recommenders (the Netflix Prize winners) they have a much easier time with heavily ranked items than with rarely ranked – I have a graph that shows that effect clearly here. Also, an informal experiment I did with a million pings of Amazon’s recommender suggests strongly that the recommender system leans more heavily to the popular items than does Amazon’s overall purchases, so the recommender is probably not helping promote minority interest products much there.

Finally, have you seen this recent paper on using experts for a recommender system? http://www.nuriaoliver.com/RecSys/wisdomFew_sigir09.pdf.

By: G. Manfredi

G. Manfredi — Fri, 13 Nov 2009 02:06:08 +0000

Very interesting analysis. We tend agree at our company — that most recommenders rely on popularity too much, even those that are “personalized”. The reason is that an item with a single “rating” cannot be relied on as a good recommendation… unless of course, that rating person was SPECIAL. In other words, unless that person was a “maven” whose reputation warranted his rating to be used as recommendations.
We’ve tested this approach of identifying “mavens” in order to go deeper into the long tail to identify potentially valuable items to recommend. But mavens can only be used for personalization if they were themselves “ranked” personally for the recipient of the recommendation… iow, each user would need their own network of maven recommenders to truly receive great recommendations.
Which, btw, is what we’ve done and are coming out with. And our experience at Discover My Network to date shows a deeper penetration of the long tail while still retaining quality/engagement of the recipient. Great to see others seeing this issue.