The Future of Search - A Brief History of Search

Traditionally search engines have been part of a simple two way process. A user tells the engine what to search for and the engine finds the most relevant results it can for the keywords provided. Algorithms beavered away, determining what page was most relevant for the keywords, which pages were spamming, how authoritative a page was and a myriad of other factors. Essentially the search engine was a massive database, which the user could retrieve data from based on the quality of their search.

While the Internet was still largely in the domain of non-commercial enterprises, such as technical and university staff this was fine. Search engines were largely used as expert tools and anyone with any skill in using them could find what they wanted fairly quickly. The Internet explosion has changed all of this. In 1998 there were a few dozen results for a search, where as now there might be hundreds of thousands, since almost everyone is using the Internet as a resource and search engines have moved on from being expert tools into the realm of the regular user

The fact that this growth will continue exponentially is unavoidable. As the Internet itself grows, so too will the amount of information and disinformation available on any given subject increase. Unlike traditional forms of archiving, almost everything is saved (with no mind to quality) and as the web grows, the task of searching or of being ranked well for a given search becomes ever harder.

The initial response to this includes improved advanced tools and speciality search engines for the user and improved algorithms for the engines. The largest change was the shift of emphasis from on-page to off-page factors. Any information can be manipulated to give a false impression and thus, where previously META data had served to categorise pages, now the META tags were being spammed. Relevant content (quite rightly) remained 'king', without content a page was unlikely to get indexed for a search term, but this was heavily open to abuse, cloaking became a common method of affecting search ranking, either with hidden text, or underhand redirects.

This all changed with the advent of link-analysis algorithms. Rather than considering each individual web page as an isolated island, link analysis algorithms treat the web as an interconnected "spiders web", with the hyperlinks the strands connecting each page to each other and forming a relationship. The concept of link analysis opened up an entirely new way of determining the relevance and importance of pages, one where the linking habits of other pages could affect the ranking of your own site.

The most famous link analysis algorithm is PageRank. Invented by Google's founders Larry Page and Sergey Brin, it was one of the most important factors in the relevance of Google's results compared to other search engines at the time. This was a major contributing factor to their meteoric rise in market share to become the dominant search engine in most of the world today.

PageRank, like many early link analysis algorithms, paid no attention to the contents of the linking pages. Without context, links became a commodity and link farms and exchanges grew. Buying links or spamming sites that allowed posting became a common way to abuse this system.

Link contextualisation, that is, not just the number of inbound links, but where they came from, the authority and theme of those pages and the content which surrounded the links were introduced to algorithmically measure the popularity of pages. Links from, and information placed on trusted or 'authoritative' sites such as BBC, DMOZ or The Economist)were weighed more highly.

Unfortunately, these new techniques merely provided a short term solution. The financial rewards associated with performing well in the SERPs are high and relevancy began to slip once again.

Link contextualisation immediately became prone to abuse. "Google Bombing" is an excellent example of how this was being manipulated where websites were made to rank highly for unrelated terms. Usage tracking opened up a new form of click fraud. Some Super Affiliates managed to become 'authoritative' and quality resources like DMOZ and Wikipedia were regularly manipulated, to the extent that DMOZ closed its doors to submissions until very recently and Wikipedia have taken the step of adding the rel=nofollow attribute to all their external links (ii) .

Tackling off-site content resulted in spammers setting up several sites with content designed to draw in users and spiders to core sites. This is increasingly becoming a more common practice. The problems the engines have are exactly the problems they had with on-site content, merely distributed over many sites. An entire sub-culture of 'splogs' (Spam Blogs) appeared, where robots automatically produce blogs and comment on other blogs, for the sole purpose of affecting the index.

Search has now begun to become more personalised. The earliest form of search personalisation is perhaps geo-targeting, where the information about the user’s location can be used to deliver more relevant results. Now search engines have access to many other pieces of information about their users, and are beginning to use this to provide more focused results. User behaviour is measured more closely. Some search engines monitor which search results users click on, which is likely to be used to determine future search results. Users are also encouraged to use search engine toolbars, which report back behaviour patterns to the engines and give implicit data on page popularity.

More recently sub-engines, such as Trexy (iii) and younanimous (iv) utilise the intersection of individual search patterns to predict best fit results and Google search has become personalised across the board for all logged in users. The advent of Web 2.0 (v) saw a huge boom in user interaction and associated search repercussions.

To date, the contest between the black hat spammers and search engines has been a continual arms race. For every method introduced by the search engines, the spammers have worked out a way around it. This looks set to continue into the future although following past trends will become increasingly difficult, and above the capability of all but the most technically competent and determined hackers.

With the increasing trends towards personalisation, the signs are that black hat SEO will move increasingly towards malicious controlling of users' computers and website accounts to enable their tactics to work in a world where the user’s themselves are an increasingly important part of the algorithm. Google have already begun working towards helping users fight badware (vi) , presumably with the impact to their relevancy in mind.

Where search goes from here is by no means set in stone. It seems clear that we are in a transitional phase in the development of web usage. While it appears that search is going to play a massive part in the future of the Internet, it is likely that we will see huge paradigm shifts before things settle, as the huge popularity of the social search movement inflicts a heavy toll upon the relevancy of current search methods.

<< previousnext >>

Notes:
ii 20 January 2007
iii Trexy.com
iv younanimous
v "Web 2.0 is the business revolution in the computer industry caused by the move to the internet as platform, and an attempt to understand the rules for success on that new platform. Chief among those rules is this: Build applications that harness network effects to get better the more people use them." Tim O’Reilly
vi Stopbadware.org