How to stop web crawlers - search engine crawlers

You can make a robots.txt file that you put in your site main directory. If its good web bots like yahoo, google, and aol you will most certainly stop them by doing this.

The robots.txt is a file that you put in place for web bots to read. All web bots are supposed to be setup to read them and follow instructions.

Here is an example of a robots.txt file
====================================

User-agent: *
Disallow: /example1/
Disallow: /example2/
Disallow: /example3/
Disallow: /example4/
Disallow: /example5/
Disallow: /example6/
Disallow: /example7/
Disallow: /example8/
Allow: /

or you can allow specific direcorites with the example below

User-agent: *
Disallow: /example1/
Disallow: /example2/
Disallow: /example3/
Allow: /example1/
Allow: /example2/
Allow: /example3/
Allow: /example4/
Allow: /example5/

You could copy and paste the above information into a new text file and then save it as robots.txt

But before you use it you must read the directions below in order to get an idea of how to use it. Then paste it into your home directory of your website. If you want to block a specific directory, here is an example below.

NOTE: You can also turn off indexing on the directories of your website that you choose. Contact your Web hosting provider for instructions on doing this.

===============

If you want to block:

www.yoursite.com/privatefolder/

You add this to your robots.txt file:
Disallow: /privatefolder/

If you want to allow:

www.yoursite.com/privatefolder/

You add this to your robots.txt file:
Allow: /privatefolder/

===============

MORE EXAMPLES:

===============

If you want to block:
www.yoursite.com/zachs-pictures/

You add this to your robots.txt file:
Disallow: /zachs-pictures/

If you want to allow:
www.yoursite.com/my_videos/

You add this to your robots.txt file:
Allow: /my_videos/