Tuesday, December 22nd, 2009

How web search engines work

Evidence that Flickr has changed their search engine

What is a Search-engine?

Search engine is a tool that we use in Internet. Whenever we search for particular topic, we use the search engine. The information searched in the internet is provided by the search engine with a list of websites that could serve our purpose. The search engine depends entirely on the data that is collected and maintained in a database.

Types of search engine

The search engine can be of two types

A. Manually powered search engine:

This is one type of search engine where the database is prepared manually with a link to the webpage along with a short description about that page.

B. Crawling engine:

A crawling engine traverse all the WebPages and prepares a database based on the keywords that is collected from that page. Here the database is prepared automatically by the software.

How it works?

Almost all the search engines depend on software for the purpose of gathering information from a website and in turn index them in database. The software used for this purpose is called spider, just like a spider it crawls every website, grabs keywords from them, and prepares a database along with link about the source.

How a spider crawls?

The spider searches the WebPages in the following way,

1) It first starts with a very popular website or a website that is frequently visited.

2) It gathers the keyword from that webpage and then continues its crawling to the links available in that page.

3) Some portion of the WebPages cannot be crawled by the spider because the webmaster of that page may have instruction for the spider as not to crawl the page.

4) In this way, it achieves success in gathering information about that site.

Next, the spider prepares a database for all the keywords specified in that page with their corresponding links. This is followed by is indexing of the database which is done as follows,


a. The database formed needs to be indexed for effective functioning of the web search engine.

b. One of the popular way of indexing is Hashing where a separate hash table is prepared

c. The hash table contains array of numbers that are linked to a particular keyword.

As the search engine is indexed, whenever a query for a particular word is requested the search engine track the database for that keyword and retrieves the webpage that matches the query.

The query becomes more effective with the use of operators like AND, OR, NOT etc

Optimizing Search Engine

The following are the ways through which a webmaster achieves a top rank in the search engine.

The web masters who are responsible for creating their WebPages have a tag included in their HTML scripts for improving the SEO (search engine optimization). The Meta tag gives an option for the webmasters to include the keywords (relevant to contents of the website) that a user might search in search engine. By having these keywords in the script, the search engine increases the rating of that particular website and shows the link to that website in the first page.

Some webmasters make it a habit to include invisible keywords in their page through which they achieve a top ranking.

Tags: , , , , , , , , , , , ,

Comments are closed.

-->