When a search engine crawls (visits) your website, the first thing it looks for is your robots.txt file. This file tells search engines what they should and should not index (save and make available as search results to the public). It also may indicate the location of your XML sitemap. The robots.txt file belongs in your "public_html" folder. It should be accessible with a link like http://www.youdomain.com/robots.txt. It is important when you have some pages which you don't want Google to crawl like payment page or login page.
How to create robots.txt file ?
A robot text file must be named "robots.txt." you can simply create plain .text file not .html and do not use Word file. Name the file in all lower case letters and "robots" should be plural. Always put your robots.txt file in your root directory
Where to put robots.txt file ?
The location where you put robots.txt file is most important. You must put it in root directory of your website otherewise search engines user agent will not able to find it. They do not search whole site to find robots.txt file they just look at the main root directory of your website. If you don't put this file in the main directory the user agency think that you don't have robots.txt file on your website.
Example of robots.txt file:
# Googlebot user agent is disallowed to see the following file:
User-agent: Googlebot
Disallow: /login.html/
Disallow: /peyment.html/
Disallow: /temp/
On the above “User-agent” are search engines' crawlers and disallow: indicates the file name which you want to disallow to user agent. You can also put your comment on this file.
No comments:
Post a Comment