Robots - an introduction
Summary : Aims of this section
To gain an appreciation of the importance of Robots, and why the Webmaster
should be aware of them.
Summary recommendation : Make your web pages 'robot' aware. Most
importantly, if you don't want your pages indexed/listed in the main search
engines, then you must make your pages tell the robots to do as you require.
What is a web robot ?
A robot is essentially an automated visitor to web sites. The robot will
trawl through the internet for new or updated WebPages.
Since there are millions of web pages, the days of someone manually documenting
and indexing web pages are long past.
So, every minute of every day, millions of computers - as part of the many
search engines, use automated programs or 'robots' to search and index web
pages.
Robots are an essential part of the net, and they are responsible for almost
all web pages being indexed.
Robot Options
The following the table list the four main options that the webmaster can apply
to their web pages.
Summary Table of the 4 main Robot rules.
| Web Page Aim | Meta name | Content name | Notes |
| Do not index page, but follow any links | ROBOTS |
NOINDEX |
This tag best used for pages that have little information, but that which do contain many web links on then - such a net-links section. |
| Index the page, but do not follow links | ROBOTS |
NOFOLLOW |
Use for a page which you want listed, but do not want the robots to follow links |
| Do not index the page, nor follow any links | ROBOTS |
NOINDEX,NOFOLLOW |
For pages which you never want indexed with search engines. |
| Index the page and follow any links | ROBOTS |
INDEX,FOLLOW |
Tag not necessary, since such behaviour is the default attitude of the web robots |
The following mini-guide will show how to set your commands for any robots
that visit your web pages.
Screenshots are all from Frontpage 2002, although the same applies to other web
creation software.
Stage 1.
You need to call up the 'page properties'. To do this, right click anywhere on
the web page you are editing, and select it.
Following steps 1 and 2, a new box should pop up.
3. In the popup box, enter the directives you want for the robots.
Special points of note
'NAME' section : ONLY EVER type in 'robots', no other text
should be entered.
'Value' section : This is where you tell the robot what to do.
So, You have 4 different variations to enter in the 'Value' Box.
Your 4 options again are ....
NOINDEX |
NOFOLLOW |
NOINDEX,NOFOLLOW |
INDEX,FOLLOW |
* Special note : The ordinary behaviour of the web robot will be the fourth
option ' index, follow' , so it is NOT necessary to add this command to your
page properties. However, personally, I still do. An invitation to robots to
scan your pages and follow your links can not do any harm, and may improve you
site ranking.
Net links
The following external links will give you in-depth explanations to the
importance of Robots.
Using Robots Meta
Tag : A nice guide on robots
---
Well, that was your summary guide for Web robots. Your are also suggested
to view related issues on
Keywords and how to describe your pages.