A robots.txt file is a short text file that resides in your home directory. Before search engines spider your site, they look into this file to see which files/file types and/or directories they are not allowed to see.
This tool will make your robots.txt file. As a little 'extra' it also includes a block to many unwanted spiders that only crawl your site in order to collect the email addresses stored on your pages.(SPAM-bots).
[ Robots.txt Generator ]
|
simple is beautiful
The Generator Blog: Robots.txt Generator.
 2 ... 2 ... |
0 comments:
Post a Comment