Robots - an introduction

Summary : Aims of this section

To gain an appreciation of the importance of Robots, and why the Webmaster should be aware of them.

Summary recommendation : Make your web pages 'robot' aware. Most importantly, if you don't want your pages indexed/listed in the main search engines, then you must make your pages tell the robots to do as you require.


What is a web Robot ? Robot Options How to Set your Pages Net links Home


What is a web robot ?

A robot is essentially an automated visitor to web sites. The robot will  trawl through the internet for new or updated WebPages.
Since there are millions of web pages, the days of someone manually documenting and indexing web pages are long past.

So, every minute of every day, millions of computers - as part of the many search engines, use automated programs or 'robots' to search and index web pages.

Robots are an essential part of the net, and they are responsible for almost all web pages being indexed.


Robot Options

The following the table list the four main options that the webmaster can apply to their web pages.


Summary Table of the 4 main Robot rules.

Web Page Aim Meta name Content name Notes
Do not index page, but follow any links ROBOTS NOINDEX This tag best used for pages that have little information, but that which do contain many web links on then - such a net-links section.
Index the page, but do not follow links ROBOTS NOFOLLOW Use for a page which you want listed, but do not want the robots to follow links
Do not index the page, nor follow any links ROBOTS NOINDEX,NOFOLLOW For pages which you never want indexed with search engines.
Index the page and follow any links ROBOTS INDEX,FOLLOW Tag not necessary, since such behaviour is the default attitude of the web robots


How to set your pages

The following mini-guide will show how to set your commands for any robots that visit your web pages.
Screenshots are all from Frontpage 2002, although the same applies to other web creation software.

Stage 1.

You need to call up the 'page properties'. To do this, right click anywhere on the web page you are editing, and select it.

Following steps 1 and 2, a new box should pop up.

3. In the popup box, enter the directives you want for the robots.

Special points of note

'NAME' section  :   ONLY EVER type in 'robots', no other text should be entered.

'Value' section : This is where you tell the robot what to do.

So, You have 4 different variations to enter in the 'Value' Box.

Your 4 options again are ....


* Special note : The ordinary behaviour of the web robot will be the fourth option ' index, follow' , so it is NOT necessary to add this command to your page properties. However, personally, I still do. An invitation to robots to scan your pages and follow your links can not do any harm, and may improve you site ranking.


Net links

The following external links will give you in-depth explanations to the importance of Robots.

Using Robots Meta Tag : A nice guide on robots


Well, that was your summary guide for Web robots.  Your are also suggested to view related issues on Keywords and how to describe your pages.