Relationships between web documents are historically defined by hyperlinks. Search engines learn about the importance of webpages by analysing the number of incoming hyperlinks, taking into account the existing importance of the linking pages. Webmasters quickly learned that hyperlink are important, that hyperlinks are worth money and
that hyperlinks can be used to manipulate search engines. In response, search engines started to offer webmasters a new tag attribute called "nofollow", to label hyperlinks that should not carry value. More recently, search engines even started to penalize websites on which paid links are not labeled with 'nofollow'.
The idea is simple. With the introduction of the nofollow tag, hyperlinks became black or white. The hyperlink pool got split into a "traffic only" and a "traffic + value" subset. Traffic is something you can buy, and contitutes not by coincidence the business model of the large search engine companies that pushed the "nofollow" adoption. The value is something that a document should deserve. Selling "value-carying" links is now considered manipulation and a black hat technique.
Many high weight user contributed websites consequently started using nofollow by default to avoid link spam. Wikipedia was one of the first, and who knows, DMOZ could be next. If the search engines have to discount hyperlinks from these relatively high quality and widely controlled resources, they are probably throwing away a lot of information. Besides getting rid of manipulative text links, much of the better quality document weighth information will be removed from their ranking algorithms. This may for example have a strong impact on Google page rank distribution. How long ago did Google distribute a visible page rank update?
We can speculate that this may evolve into one or a combination of three scenario's.
- Search engines have to rely even more on other information, such as actual usage data (from toolbars, analytics logs, ...).
- Search engines will not always not follow nofollow. Or: not all nofollow links will be discounted.
- A more nuanced / fine-grained set of attributes will be introduced to describe the value and features of hyperlinks.
None of these scenario's will ever solve the problem of search engine manipulation. They are just part of a process of co-evolution between the actors in the field.