This type of file allow search engine spider or crawler to access or not access website.
Place this type of files to the root directory of your website.
Example of robots.txt file
User-agent: *
Disallow:/
Above sentence describe that robots will not index any page of your website.
User-agent: Googlebot-Image
Disallow: /
If you dont want Google' Image bot crawler to make your websites images searchable by search engine then just place above lines to your robots.txt file.
No comments:
Post a Comment