Is the long tail being undermined by an online monoculture that is ‘worse than mass media’?
Whenever I make these claims someone says “Well I use Netflix and it’s shown me all kinds of films I didn’t know about before. It’s broadened my experience, so that’s an increase in diversity.” And someone else points to the latest viral home video on YouTube as evidence of niche success. So this post explains why your gut feel is wrong.
As indicated above, Tom Slee wants to show that the long tail hypothesis is undermined by recommender systems. Even though individuals may feel they have more choice, in actual reality, there is a process of maintreaming choice going on.
To argue the case, Tom refers to the following:
* a paper by Daniel M. Fleder and Kartik Hosanagar called Blockbuster Culture’s Next Rise or Fall: The Impact of Recommender Systems on Sales Diversity. They simulate a number of different kinds of recommender system and look at how these systems affect the diversity of a set of choices. Towards the end of the paper they observe that some of their recommender systems increase the experience of diversity for every individual in the sample and yet decrease the overall diversity of the culture. So I wrote a program that does basically what they do in their paper and tweaked it to highlight this result.
* Tom also undertakes his own simulation exercise, which confirms the above findings.
His rather strong conclusion:
“Online merchants such as Amazon, iTunes and Netflix may stock more items than your local book, CD, or video store, but they are no friend to “niche culture”. Internet sharing mechanisms such as YouTube and Google PageRank, which distil the clicks of millions of people into recommendations, may also be promoting an online monoculture. Even word of mouth recommendations such as blogging links may exert a homogenizing pressure and lead to an online culture that is less democratic and less equitable, than offline culture.
Individual diversity and cultural homogeneity coexisting in what we might call monopoly populism.
But don’t think this is just about automated recommender systems, like the ones that Amazon and Netflix use. The recommender “system” could be anything that tends to build on its own popularity, including word of mouth. A couple of weeks ago someone pointed me to this video of Madin, a six-year-old soccer prodigy from Algeria, and the next day my son, who moves in very different online circles to me, was watching the same one. I know who Jim Cramer is even though we don’t get CNBC in Canada because everyone is talking about him and helping his disembodied head to shoot down Jon Stewart. More people watched Tina Fey being Sarah Palin online than on Saturday Night Live, and Fey is now famous in countries where no one watches the TV show. Clay Shirky writes an essay and I get five different links to it in my Google Reader feed in one morning. Our online experiences are heavily correlated, and we end up with monopoly populism.
A “niche”, remember, is a protected and hidden recess or cranny, not just another row in a big database. Ecological niches need protection from the surrounding harsh environment if they are to thrive. Simply putting lots of music into a single online iTunes store is no recipe for a broad, niche-friendly culture.”
We asked Sam Rose to answer this challenge:
“I don’t think that this article argues that the longtail is an illusion everywhere. I think it shows that the “longest” parts of the “longtail” are the most difficult to see. A one-person niche doesn’t spread around online like a virus. I think the article argues that the “longtail” may be an illusion in places that are trying to foment a monoculture to drive sales (like Amazon). Actually, Amazon tries to tie popular monoculture to “niche” sales, by showing “people who bought this also bought this”.
The article talks about how something that is a niche spreads to become basically more than a “niche”; it becomes “popular”. The model in the article does not account for how the system may open exponentially more niches by way of it’s affordances. I believe that the model is useful for understanding what happens when decision making is affected by recommendation engine feedback loops in networks, but may not be reflective of the greater ecology of systems like Amazon.
It does seem, however, to show that when the primary focus is sales of a product, such as on a system like amazon, that tensions will try to drive towards monoculture (because the rules of the system are pushing things in that direction).
It would be interesting to adjust this model to represent a system of making/using/sharing instead of sales, and see what emerges from that… “
November 13th, 2009 at 4:06 am
Very interesting analysis. We tend agree at our company — that most recommenders rely on popularity too much, even those that are “personalized”. The reason is that an item with a single “rating” cannot be relied on as a good recommendation… unless of course, that rating person was SPECIAL. In other words, unless that person was a “maven” whose reputation warranted his rating to be used as recommendations.
We’ve tested this approach of identifying “mavens” in order to go deeper into the long tail to identify potentially valuable items to recommend. But mavens can only be used for personalization if they were themselves “ranked” personally for the recipient of the recommendation… iow, each user would need their own network of maven recommenders to truly receive great recommendations.
Which, btw, is what we’ve done and are coming out with. And our experience at Discover My Network to date shows a deeper penetration of the long tail while still retaining quality/engagement of the recipient. Great to see others seeing this issue.
November 13th, 2009 at 8:42 pm
Thanks for the response to the article. I hope it’s clear that I was trying offer a step up from the all-too-popular anecdotal evidence for recommender systems’ long tail effects, rather than any kind of proof of their inevitable behaviour. There are just too many ways to implement recommender systems for any comprehensive statement of that kind. I suspect the outcome will have more to do with the incentives of whoever is running the system than with its technical details.
That said, as G. Manfredi says, if you look at even the most successful recommenders (the Netflix Prize winners) they have a much easier time with heavily ranked items than with rarely ranked - I have a graph that shows that effect clearly here. Also, an informal experiment I did with a million pings of Amazon’s recommender suggests strongly that the recommender system leans more heavily to the popular items than does Amazon’s overall purchases, so the recommender is probably not helping promote minority interest products much there.
Finally, have you seen this recent paper on using experts for a recommender system? http://www.nuriaoliver.com/RecSys/wisdomFew_sigir09.pdf.