We continue yesterday’s coverage of the Quaero Forum on the politics of search engines, with a interesting discussion between Erik Borra of Open Search, being challenged by Richard Rogers

Issue 1: Manipulation vs. spam

Rogers:

Normally search engine companies argue that their logics are closed and not transparent, because if they were open, the search would be open to spamming and manipulation. In the presentation, you have that you strive to be open about the logics, but have no answer yet for spam. search engine companies already have an answer. Yours?

Borra: My first answer would be the one from Nutch:

Search engines work hard to construct ranking algorithms that are immune to manipulation. Search engine optimizers still manage to reverse-engineer the ranking algorithms used by search engines, and improve the ranking of their pages. For example, many sites use link farms to manipulate search engines’ link-based ranking algorithms, and search engines retaliate by improving their link-based algorithms to neutralize the effect of link farms.
With an open-source search engine, this will still happen, just out in the open. This is analagous to encryption and virus protection software. In the long term, making such algorithms open source makes them stronger, as more people can examine the source code to find flaws and suggest improvements. Thus we believe that an open source search engine has the potential to better resist manipulation of its rankings.

A second answer would be that because people can devise their own ranking plug-in’s with different ranking schemes, it would be harder to manipulate and spam them.

Issue 2: Personalization vs. tribalism

Rogers: With plug-in’s people can choose their own ranking algorithm, and thus can personalize results. Classically, there is the idea that shared media experiences (people seeing similar things) make society — i.e., commonality in exposure but different views about what it all means (cf. community, which is different). Personalization, however, creates tribes, hate groups privileging other hate sites and their results, dictatorial regimes privileging ‘official’ sites. Search engines claim a kind of egalitarianism (indexing the ‘whole web’), but then ranking according to authority. Yours?

Borra: Search engines might claim a kind of egalitarianism but this is not because of indexing the whole web. Florian Schneider’s presentation for example has questioned the ‘borders’ of the database and index of a search engine. There is also the tendency of search engines to localize and personalize, e.g. iGoogle serving results which are specifically tailered to you (recall also perfect recall from Michael Zimmer’s talk), or a specific country version of a particular search engine. One might also be tempted to talk about an ‘objective’ ordering but there are enough examples that prove the current ranking schemes of the big search engines are not that objective at all.

The shared media experience that makes society is already diffusing by the broading offer of information channels via internet and interactive television. Of course personalization of the current engines also adds to the decline of the shared media experience.

One might think a p2p engine like Open-Search has the danger of tribalism and segregation because your peers might not ‘know’ as much as somebody else’s peers. At the level of the search engine however, there are no peers. To the open-search engine, the p2p layer is an abstraction of a generic storage device, a database. It is assumed that each peer will have access to the entire database at all times, regardless of which peers connect to which peers. We are not using a flooding model, as does e.g. Soulseek.”