WebsiteDays.com create your web site free, Hosting, Coding, E-commerce, Free Website Builder. Download free website builder to create your website! Personal, and ECommerce website builders...

WebsiteDays.com for russia bryansk

Click on your region below to contact a Websitedays.com designer, or Websitedays.com sales person for info about Online Website builder system.

You can also send an email to

and we can forward your enquiry to respective designers in your area.

Almetyevsk Arkhangelsk Astrakhan
Barnaul Bryansk Chelyabinsk
Dagestanskiye Ogni Dmitrov Dubna
Dzerzhinsk Ekaterinburg Gatchina
Irkutsk Ivanovo Izhevsk
Kaliningrad Kazan Khabarovsk
Krasnoyarsk Kubinka Kursk
Kuznetsk Moscow Murmansk
Nizhni Novgorod Novgorod Novorossiysk
Novosibirsk Obninsk Omsk
Orenburg Penza Perm
Pervouralsk Petrozavodsk Pskov
Rostov-on-Don Samara Saratov
Sarov Sergiev Posad Severomorsk
Shchelkovo Smolensk St Petersburg
Staritsa Stavropol Taganrog
Tomsk Tver Ufa
Ulan-Ude Ulyanovsk Vladivostok
Volgograd Voronezh Yaroslavl
Yegoryevsk Yekaterinburg Yoshkar-Ola
Zelenodolsk Zhukovsky

»   The Proper Way To Use The robot.txt File

Article submitted in category: Web Design And Development
Tags: | disallow | file | user-agent | index | robottxt file | robottxt | spiders | * disallow | user-ag

The Proper Way To Use The robot.txt File by: Jimmy Whisenhunt

When optimizing your web site most webmasters don’t consider using the robot.txt file. This is a very important file for your site. It let the spiders and crawlers know what they can and can not index. This is helpful in keeping them out of folders that you do not want index like the admin or stats folder.

Here is a list of variables that you can include in a robot.txt file and there meaning:

User-agent: In this field you can specify a specific robot to describe access policy for or a “*” for all robots more explained in example.

Disallow: In the field you specify the files and folders not to include in the crawl.

The # is to represent comments

Here are some examples of a robot.txt file

User-agent: *

Disallow:

The above would let all spiders index all content.

Here another

User-agent: *

Disallow: /cgi-bin/

The above would block all spiders from indexing the cgi-bin directory.

User-agent: googlebot

Disallow:

User-agent: *

Disallow: /admin.php

Disallow: /cgi-bin/

Disallow: /admin/

Disallow: /stats/

In the above example googlebot can index everything while all other spiders can not index admin.php, cgi-bin, admin, and stats directory. Notice that you can block single files like admin.php.

About The Author

Jimmy Whisenhunt is the webmaster at VIP Enterprises http://www.vipenterprises.org

vipenter@vipenterprises.org

This article was posted on February 06, 2005